Monday, May 11, 2009

Watching the Regent, vol. 1

A while ago I noted Passing Notes, a neo-education reform blog whose assertions tend to run far in advance of the facts, but I could never seem to successfully register and comment on the blog, and I figured I had bigger fish to fry and focused my ire elsewhere.

I finally realized that its primary author is, in fact a big fish, a member of good standing in the new education establishment, you might say: Angus Davis of the Rhode Island Board of Regents for primary and secondary education.

So... let the fact-checking commence!

In a post from last weekend he writes:

...the nearest KIPP school is in Lynn, MA, where more kids are on grade level in reading and math than schools in Marblehead (could you imagine a school in Central Falls where the students outperform the kids in East Greenwich?!?).

OK, let's see, he references a Boston Globe article from last year (bad sign) that says:

Now, KIPP students' scores surpass those of Lynn district schools and state MCAS averages, and its 2007 English and math scores bested Swampscott and Marblehead, respectively.

Ah... so only the math scores are higher than Marblehead. And anytime someone's comparing their English scores to one district and their math scores to another, there's some cherry picking going on.

So let's go to the state website. Since Davis uses the present tense, I'm going to look at the 2008 data, which is also the only data which includes 8th graders at KIPP. Links: Marblehead, KIPP, Lynn district. I made graphs! But I got a C in statistics, so they just show percentage of students proficient or above on the MCAS. Note that this is not longitudinal data -- it shows the scores in each grade for this year, not what the 8th grade cohort scored in previous years.

Let's start with Language Arts:

This is the prototype sales pitch for KIPP -- impressive improvement in Language Arts achievement, but short of Davis's claim. Also, note that Lynn shows comparable growth every year except seventh grade, where things apparently go pear-shaped.

Math:

First off, yes, KIPP seventh grade math tops Marblehead thanks to a dip in its scores -- before swapping places. So Davis's claim has a kernel of truth, but it is certainly not true that "more kids are on grade level in reading and math than schools in Marblehead."

Beyond that, this is a weird graph. It has KIPP virtually closing the achievement gap in 5th grade and ending up with a widening gap in 8th. I added a separate (green) line for just the scores of low-income KIPPsters, which accentuates the downward trend (overall removing the low income scores from Marblehead and non-low income scores from KIPP tends to move them apart a consistent point or two). Again, Lynn looks pretty good in 6th and 8th grade, not so much in 5th and 7th. If you can close the elementary math achievement gap in math in a single year, it calls a lot of the framing of these issues into question (look a second grader in the eye and tell them if they’re on track to get into a good college or not, etc.).

Finally, as a cautionary point about data-driven schools and the achievement gap, here's the scores for the Science and Technology test, given in 5th and 8th grade only:

What is measured gets taught and otherwise...

It is a shame that the rhetoric runs ahead of the facts, and advocates feel it is necessary to make unsubstantiated claims about what I'm sure are genuinely good schools in many ways. And, as Aaron Pallas's analysis of Roland Fryer's claims about the Harlem Children's Zone illustrate, this is not an isolated incident. It is what happens when you write the marketing copy first and wait for numbers that fit. You stretch and pull until you find something "close enough."

As Mike Klonsky put it today:

I'm sure that a small school with a good group of teachers and lots of resources can produce a jump in test scores. But let's cool it on the miracle talk please.

No comments: