Monday, September 24, 2012

The Difference Between Polling, Moneyball, and Teacher and School Evaluation

Nate Silver:

But before we get lost in the weeds, let’s consider a more basic question. What did the polling look like at this stage in past elections, and how did it compare against the actual results?

Our polling database contains surveys going back to 1936. The data is quite thin (essentially just the Gallup national poll and nothing else) through about 1968, but it’s nevertheless worth a look.

This is a basic part of modeling at FiveThirtyEight or Football Outsiders; run historical data through your algorithm and see if it correctly predicted future performance.

You can't do this in education because you don't objectively know who the eventual winners are. You don't even know the object of the game, to be honest.

Even so, there seems to be little interest in this sort of historical sanity check in the world of data-driven ed reform. RIDE came up with a system for calculating student growth for teachers and students based on current test data -- that is, they could run the numbers six years back. They claim to have not done that; they certainly haven't released it. This is the foundation of their whole Race to the Top edifice, but they don't care to look very closely at it, and they don't want you to.

No comments: