The Test That Teaches

What the very best tests do

Unfortunately, all tests are vulnerable to gaming. There’s even a “law” about this, often attributed to Charles Goodhart: When a measure becomes a target, it ceases to be a good measure. Let me give you a quick example. Suppose you wanted to assess someone’s reading ability. A vocabulary test might do a pretty good job. Most people don’t use words like inchoate and internecine in everyday speech, so you’re only likely to be familiar with them if you read certain kinds of books¹. A test might do a good job of identifying which students know these words, and we might infer that these students have read the kinds of books that have these words, and then further infer that they are pretty good readers.

But what if your college acceptances depended on this test? And what if you knew that these words (or words like them) were going to be on the test? Then a student might study those words and memorize reasonable definitions. That would, of course, undermine the inferences you’d like to make, and would therefore weaken the value of the test. Instead of indicating what a student had read and understood, it would merely indicate what a student had memorized.

A bad test like this would also incentivize students to spend a lot of time on relatively unimportant skills. There is some value in learning words and figuring out how to memorize material, but I’d much rather have students spend that time trying to understand Shakespeare.

Nevertheless, education needs tests. They diagnose weaknesses and highlight strengths. They rank students. They provide students, parents, teachers, and schools with information that helps them make better decisions about what and where students should study.

And the very best tests do one more thing for us: they teach. Students actually become better learners through the test prep process itself. If that sounds unrealistic, consider a different kind of test. Suppose I wanted my students to be physically fit, and I gave them the following test: Run a 6 minute mile.

Would I worry that students might “game” this test by focusing on running miles? No, please do! I don’t think there’s any way for a 9-minute-miler to become a 6-minute-miler without becoming more physically fit.

A great math and reading test does the same thing. Any attempt to game it that results in large score increases will simply make you better at math and reading. You might try to figure out which questions show up the most, then practice them repeatedly. You might work on certain strategies (like paraphrasing). These activities will work. And they will also make you a better math and reading student.

Interestingly, the act of testing is itself an excellent way to learn. As I mentioned in 10 Useful Learning Theories, when you attempt to remember something and succeed (retrieval), you actually remember it better.

Tutors have known that good tests drive learning for a long time. But it’s gratifying to see this idea enter the mainstream. This month, hundreds of University of California professors signed a letter asking the UC system to bring back tests. If you’re interested in their reasoning, you can read the full letter here. As Professor Svetlana Jitomirskaya points out in The Free Press, the tests don’t just find better students. The tests make students better. Here she is describing one student who was not ready for the rigor of college math:

“Preparing for the test is itself a powerful intervention. If Diego knew that the SAT stood between him and a Berkeley engineering degree, his drive would have led him to use free, high-quality resources away from rote memorization and toward real mathematical reasoning. The preparation itself would have rewired his foundation. We failed Diego once by not providing him a decent math education. We should not fail students like him again by removing the incentive to build one themselves.”

Can these tests still be gamed in some ways? Of course. In my 6-minute-mile example, the run time is a little less impressive if you exclusively trained for that, rather than achieving it in the process of many other athletic pursuits. But the process still helped your physical fitness, and would likely help you across many sports, even if you did not directly practice them. Similarly, when preparing for standardized tests, you can narrowly focus on the problem types that show up repeatedly. You might focus on reading particular subjects and prioritize skills that are useful for short passages. The scores will very likely be a bit inflated as a result, but that inflation will be dwarfed by your intellectual growth.

1. I have to admit, I’m not 100% sure how to pronounce these words