Diagnosing Learning Difficulty and Engagement
Consultancy project developing a standardised difficulty index for an ed-tech platform, combining error rates, hint usage and give-up rates into a comparable measure across subjects.
Context
Algebrakit builds technology for mathematics education. As a data analytics consultant, I analysed behavioural data from the online learning platform and translated it into something the company could act on.
Problem & research question
The client wanted to understand engagement on the platform — and, in particular, how the difficulty of its subjects could be measured and compared in a way that supports product and content decisions.
The research question: how can behavioural exercise data be transformed into a fair and interpretable measure of subject difficulty?
My role
I developed a standardised difficulty index — combining error rates, hint usage and give-up rates across subjects into a single, comparable measure — and used it to analyse engagement patterns alongside differences in student ability.
Data
Exercise-level behavioural data from the platform, including error rates, hint usage and give-up rates across subjects.
Approach
- Standardised error rates, hint usage and give-up rates into z-scores so that subjects could be compared on a common scale.
- Combined these signals into a subject-level difficulty index, designed and validated for interpretability.
- Compared hint usage against give-up behaviour to surface engagement patterns, accounting for differences in student ability.
Results & insights
The standardised difficulty index let the client compare subjects on a common scale and identify which topics were relatively more difficult.
Comparing hint usage with give-up behaviour also revealed engagement patterns, while highlighting the need to account for differences in student ability when interpreting results.
Challenges & limitations
- Working within the constraints of a real client relationship: scoping questions, respecting data boundaries, and delivering on a deadline.
- Any difficulty index involves judgement calls about what counts as difficulty — those choices had to be explicit and defensible.
What I learned
Consultancy compresses the full analytics workflow into a few weeks: understand the domain, analyse honestly, and communicate clearly. It taught me to treat the client conversation as part of the method, not an afterthought.