Smart Practice Algorithm in Mobile App

As you’re working through one of our courses or paths, you’ll occasionally come across a screen reminding you to practice. We’re not nagging you or trying to interrupt your flow. This is an AI-powered feature called smart practice that essentially helps you hack your study sessions, so you practice the right material at the most optimal times.

Smart practice uses an algorithm to plan customized coding practice sessions for you. “With smart practice, we keep track of what you’re learning, when you learn it, and how well you’ve been doing,” says Dónal Ó Dubhthaigh, Senior Product Manager at Codecademy who worked on the feature. “Based on those variables, we present practice to you according to what you most need to cover, so we prioritize things that you might be forgetting.”

The genius framework behind our smart practice feature is a science-backed concept called “spaced repetition.” Research has shown that you can retain information longer and mitigate your brain’s natural forgetting process if you review material at strategically spaced intervals. For example, if you’re studying with flashcards, it’s better to see cards that you’re not getting right or might’ve forgotten than it is to see cards that you’ve recently reviewed and feel confident about.

Here’s an inside look into how Codecademy’s engineering team created smart practice in our courses and mobile app so you can maintain your momentum, feel prepared to tackle advanced coding concepts, and meet your goals faster.

The project: Make practice easier by strategically surfacing the concepts that learners need to review.

Previously, learners would have to manually choose what they want to practice, which can be time-consuming (and, well, boring). So, about two years ago, Dónal and his team got to work figuring out how to “cook up the algorithm” that could personalize practice for our learners, he says.

Using pedagogical models as a guide, the engineers had to:

• Build an algorithm that crunches learner data and presents appropriate practice material
• Code something that runs (and make sure it’s fast)
• Add Smart Practice to our mobile app, Codecademy Go

Investigation and roadmapping

“Bloom’s Taxonomy is at the heart of a lot of the current projects that we do today and how we teach. We want learners to go up the pyramid of thinking skills: remembering, understanding, applying, analyzing, evaluating, and creating. The reasons why we got into spaced repetition as a system are that: it helped on the ‘remember’ part of the pyramid; it tracked how learners were doing; and it did something with all that data. We wanted to make the platform smarter, so we thought about how we process data and what we could do with it.

There’s a concept called spaced repetition, which basically makes it easier for you to remember things if you reactivate the neural pathways at intervals and do the work to try and remember the thing. We figured that we could do that with the technologies, hence the smart practice algorithm.

We had all these pedagogical theories on forgetting curves — then myself, the engineering manager, and the back-end engineer went through the process of seeing what’s possible. How can we get something decent to run in code? And then how can we make it run fast enough?”

Implementation

“Most of our projects are very focused on the front-end of the learning environment. We prioritized smart practice in large part because we had a back-end engineer on the team who focused on Ruby, Ruby on Rails, algorithm, and databases. It was pretty ambitious for us to do, and we spent a long time getting [the algorithm] to work in the first place.

Our first version took 20 seconds to figure out what you should practice, which shows how complicated the problem was to solve and the power of the system. There are hundreds and hundreds of facts that a learner covers, and then there are all the touchpoints that they have historically with that fact. Like, when did they last see it in the learning environment? When did they last practice, if at all? And then what are their scores? The algorithm will crunch all of this and figure out what’s a priority. Now it runs in under two seconds.”

Troubleshooting

“We made a bunch of headaches for ourselves. Because of the nature of our content, learners will learn something in a lesson, practice it in a quiz, and then apply it in a project. You’re using and seeing the same learning standard in multiple formats over period of a few days, and we didn’t want learners to have to repeat that again. So, we added in a delay to the algorithm: If it’s the first week since you’ve learned the thing in the first place, we’d tell you, there’s no practice for you to do. Eventually we removed the delay because of learner feedback; we changed the system to give learners more choice about practicing instead of having it unavailable. We also wound up simplifying how we communicated the prioritization, which is how the mobile app runs now.

Then when all the engineers on the team were trying to test the algorithm, they didn’t have much that they practiced and we’re bad at — they were too good. We were like, what’s a regular learner going to do? We ended up making our own scripts to make new accounts and invent ‘learners’ who had all the attributes that we wanted to test.”

Ship

“Getting the algorithm to work more and more efficiently was cool. I remember just seeing the time metric get lesser and lesser, to the point where we didn’t even need to have a loading animation anymore. It took a lot of iteration and different technologies to get there, for example, we moved a bunch of stuff to GraphQL.

Our biggest success was when we put out a legitimate upgrade for the mobile app in June with smart practice flashcards. Learners really liked being able to just click a button and practice super quickly. You can do flashcards in just a few minutes, so it enables you to fit practice into your routine and learning process in a lot more straightforward way.”

Retrospective

“In the whole process of putting this feature together, as a team, we learned a lot more about pedagogy. This was a project where we were getting much more into the heart of how our learners should learn and building that learning science into the product.

In total, there were nine people who worked on this in two waves: the web platform, then the mobile app. Software Engineer II Jahaziel Guzman and Software Engineer II Julie J. worked solo for months on the algorithm.”