Sarina AI Learning Analytics - How users Engage with GenAI

Sarina AI now has learning analytics that are actionable and ready for educators to gain insight into how their users are engaging with the platform.

This update releases the home dashboard, featuring the following analytics captured from a course level. You can also select many courses and see statistics across them all. More dashboards and features are planned. Feel free to email or contact us if you have feedback on these new features.

A part of this update focused on making sure specific users were not identifiable. Privacy is a big focus in AI, as well as ethical use of AI. These statistics do not identify individuals or focus on users, they are  focused on courses, and threads within the courses. This also works with our redaction and de-identification feature which helps make sure there is no identifiable information within Sarina AI's systems.

Aggregated General Statistics 

• Ability to see threads over time, depending on the period selected there will be a small chart showing activity and the number of threads created during the period.
• Messages during the period selected.
• Average messages per user (rounded to 2 decimal places).
• User corrections to Sarina AI which shows that users have read, interpreted and decided that Sarina AI made a mistake and told the AI to correct it.

Aggregated statistics for a single course.

Chat with Analytics

Naturally, you can chat with your data through the "Bot" button. The AI will have access to the same analytics shown to you, as well as definitions of interaction types.

 

Sentiment Scores and Interaction Types

You can also see how student sentiment changes over time, which might help you understand how students are handling different weeks in the courses selected. This data is aggregated for all threads that are within all courses you have selected. Technically, each message has a sentiment score which is averaged across the thread.

There is also an interaction type for each thread, which is a set category determined by AI based on the contents of that thread. These are the interaction types available:

• Creative Idea Generation: Brainstorm and develop innovative ideas for projects or assignments.
• Concept Exploration: Deepen understanding of complex concepts through interactive explanations.
• Drafting and Writing Support: Get assistance with drafting essays and reports, including structure and content.
• Customisable Templates: Create and modify templates to enhance organization and efficiency.
• Instruction Clarification: Interpret and clarify assignment instructions to ensure clear understanding.
• Coding Assistance: Receive help with programming tasks, including debugging and code suggestions.
• Networking Problem Solving: Troubleshoot networking issues and understand IT concepts through practical scenarios.
• General Problem Solving: Tackle a wide range of academic problems to enhance critical thinking.
• Problem Decomposition: Break down complex problems into manageable, step-by-step parts.
• Specification Understanding: Analyse project specifications and requirements for better comprehension.
• Language and Writing Enhancement: Improve spelling, grammar, and writing quality with AI feedback.
• Proofreading and Revision: Using AI for thorough proofreading to enhance the quality of written work.
• Concept Learning with AI: Engage with AI as a teacher for tailored explanations and examples.
• Language Translation Support: Translate texts to support language learning and accessibility.
• Off-topic: Conversations that are not related to academic work, university policies, or other educational purposes.

 

Sentiment scores and interaction types for the period.

Keywords and Word Cloud

In addition to the interaction types and general statistics, you can also see common negative keywords and a word cloud with all word frequencies allowing you to see what students are likely focusing on in their conversations with Sarina AI.

Topic Analysis and Problems

One of the major features of the analytics is topic analysis and modelling using HDBSCAN - you can customise the parameters in the advanced filter options. Each cluster is then aggregated and an AI summary of the top 8 identified clusters is provided. This analysis and clustering is done on the message level, not thread level. You might find the topic analysis/problems identified are not quite perfect, but they'll give rich insight as to how your users are using the tool and the potential problems with your instructions/prompts/assignment specifications.

 

Pain points and overall topics for a course

 

You can also see how users might be focusing on something that they should know already, or what they're actively struggling with, allowing you to adapt content, or mention it in class times.

 

 

Summary

These learning analytics are just some of many that could be displayed, we would prefer to not overcrowd the dashboard and focus on key metrics and information that you as educators can use to your advantage to improve user experience whilst balancing privacy. Feel free to provide feedback through our contact us page.

These analytics do not reveal specific users, and are kept high-level. Identification of users is against our privacy policy. Users should remain anonymous and be able to get help as they need to without feeling as though educators are always looking at their chat logs. The AI does not have access to names, student identifiers or other details.