The Limits of Mathchops

What can an adaptive tool like Mathchops do, and what can’t it do?

In theory, it should have a model of each student’s learning state – a sense for what a student knows and what she doesn’t. It should also have a model of the domain – which questions belong, their relative importance, and how they relate to each other. It should also be able to incorporate best practices based on reputable learning theories, like the 10 I wrote about previously – things like spaced repetition and retrieval practice.

But that’s just the theory. In practice, it’s great at certain things and not so great at others.

To get a better sense for its capabilities, I think it’s helpful to place questions into 5 categories:

1) Known. These are the questions the student has learned previously and can get right 90%+ of the time. The software:

• mostly avoids these questions.

• periodically tests students on them to help with retention.

• alerts the tutor if they aren’t in the “known” category any more.

• uses these known questions to keep confidence high, so that wrong answers aren’t too upsetting.

2) Forgotten or Adjacent. These are questions that a student can easily learn on her own. She might need a slight nudge (in the form of the right answer or a short explanation). She might need to reflect for a brief period, or practice them a few times. But she doesn’t need much tutoring for these. The software:

• finds these questions.

• provides different versions of them to ensure mastery (different numbers, slightly different variations).

• provides adequate explanations.

3) Can Learn With Help. These are related to questions a student understands, but are just a bit too hard for the student to learn on her own, even with explanations and repeated practice. The software:

• confirms that the student is not able to answer these questions.

• brings the questions to the attention of the tutor (and student).

4) Too Hard…for now. These take too much time to explain and too much time to retain. The software purposely keeps these away from students.

5) Irrelevant. The computer keeps students away from questions that will never appear.

How valuable are these functions? That depends on how important you think categories 2 and 3 are. I believe category 2 is especially important because such a large percentage of the material has already been covered in school (at least 90% for most SAT students). So much of test prep is simply finding and retaining questions you can easily master but haven’t yet.

But what is this type of software bad at? There are (at least) three big things and one small thing.

First, it’s not great at getting your student to use the software in the first place. Tutors are extremely good at getting students to commit to consistent work over a long period of time. Software can do more with that time (extend it, make it more efficient)…but only once they’ve started. If a student doesn’t see the value of the work, he’ll never start it in the first place. Sales chops are irreplaceable.

Second, it’s not great at teaching very new and difficult concepts. I probably would not rely on Mathchops to teach SOHCAHTOA, for example.

Third, it’s only adaptive within its domain. This may be obvious, but a math practice tool is only good at math practice, not reading or executive function. A tutor can switch effortlessly between every subject on the test, general test-taking strategies, and sleep routines. A tutor is also better at sensing frustration and keeping students in a “learning state”: not overwhelmed, not distracted, not too excited – just ready to learn.

Another thing it’s not great at is replicating the test. Taking official practice tests in test-like conditions is very important. But that’s not what adaptive software is supposed to do. It’s a tool designed for a very specific goal (getting students to answer lots of good questions). Many features that help it achieve this goal – like individually timed questions, numeric-only answers, short games – make it unlike a real test. But since we already have great tools for achieving the goal of ‘real test replication’ (namely, official practice tests), I don’t think this ‘deficit’ matters too much.

One last note: it’s likely that software will get much better at some of these things it’s currently not good at. I will leave that topic for another post, but I will just say that I’m very skeptical that it will ever fully replace tutors – I think it’s much more likely to enhance what we do.