Wednesday, September 25, 2013

When I Hear "Months of Learning" I Reach for My Revolver

Gary Rubenstein:

The difference between a group scoring in the 27th percentile and the 30th percentile is very small, .07 standard deviations. To give you an idea of how small this is, a 27th percentile on the SAT math section is a score of a 430 while a 30th percentile is a score of 440, which is a difference of one question out of about 60. So how does this get converted to 2.6 months? Well, in a 2007 report by Hill, Bloom, Black, and Lipsey called Empirical Benchmarks for Interpreting Effect Sizes in Research, they estimated how much students in different grades generally learn in a year, in terms of standard deviations, where one standard deviation is roughly around an increase by 30 percentile points. They estimated that in earlier grades students progress more than they do in later grades. So if you were to give first graders a pretest at the beginning of the year and a post test at the end, they would, on average, go up by .97 standard deviations which is about 30 percentile points. But this report says that secondary math students are only expected, on average, to gain about .27 standard deviations, which equates to around an increase of 8%. So for secondary math, the .07, or 3% advantage that the TFA teachers got was equivalent to about 26% of .27 which meant they learned 26% more than the average secondary math student learns in a year, and since the school year is 10 months, that is 2.6 months.

Don't underestimate how crucial this "months of learning" rhetoric is to reformers, or how flimsy it is.

No comments: