Tuesday, October 05, 2010

Do We Have the Stomach for this Fight?

Here's the thing: at the same time Waiting for Superman, NBC's Education Nation, and Oprah launched an education reform onslaught with Geoffrey Canada and his Harlem Children's Zone at its center, New York City was releasing its school progress reports for last year.

Check out the Harlem Children's Zone Promise Academy Charter School's rankings for student progress (i.e., "value added") in English Language Arts. I'm not sure which of these stats makes the punchiest soundbite or exactly how to word them:

  • The percentile rank of the median ELA growth HCZPA's lowest third of students, relative to similar NYC neighborhood and charter schools: 0%
  • The above compared to the "city horizon:" -10.6% (how you generate a percentage rank that's negative, I don't know)
  • Percentile rank of median ELA growth for all HCZPA children compared to similar NYC neighborhood and charter schools: 4.2%
  • Above compared to the "city horizon:" -1.4%

Put another way: HCZPA had the least growth in ELA among the lowest third of their students among NYC neighborhood and charter schools schools with a similar population.

This isn't some random school I cherry-picked, it is the flagship school of Geoffrey Canada's Harlem Children's Zone. It's so extreme that your first reaction is probably "Wow, something must be wrong with those numbers." Which is fine by me. That's the beauty here: you either have to give up some of your faith in the miracle of the high performing charter school as the solution to our problems, or some of your faith in the reliability of narrow measures based on test scores and, in particular, the efficacy of more complex value-added measures.

The thing is, I haven't seen anyone else mention this rather breathtaking statistic, but I can't be the only one to have wondered how the HCZPA did on the new report card. This is news, and a wonderful way to turn the discussion in a different direction, but even opponents of business model reforms haven't brought it up yet. And to be honest, even doing so makes me feel a little guilty somehow.

Later... the explanation for the negative growth percentile (you can find the guide for teachers in the link above):

For each element in the Progress Report, the peer range is the range of scores earned by peer schools in the 2008-10 period excluding “outlier” scores that deviate so dramatically from the other scores that it is not reasonable to use them as reference points. An “outlier” score is defined as one that is more than two standard deviations away from the mean. The peer range “minimum” is the lowest non-outlier score and the peer range “maximum” is the highest non-outlier score.

10 comments:

Bill Kerr said...

Direct link to HCZ Promise Academy pdf

They take the median scores as zero and so your result is negative if you are below the median. The data is presented diplomatically, ie. they don't show the negatives as a graph but just show the percentage.

You have cherry picked the data since they are going ok in maths and school environment. The data indicates overall that progress is being made and since the environment is positive further progress could well be made in the future.

How useful the data is to broader educational issues is the more important issue IMO. eg. I don't like the "peer school" concept since schools differ so much anyway and the whole idea conceals more inequality.

Tom Hoffman said...

Nah... if you dig into the teachers guide on the page I linked to they explain that the median is 50% and the range is basically two standard deviations above and below. Anything beyond that is an "outlier."

Jason said...

So the charter school using the "Broader, Bolder Approach" and provides students with significant wraparound social services is doing no better than typical relative to other TPS and charters in the area (0 = median/typical).

Meanwhile, there are other schools, both charters and TPS's, that do not use some massive web of coordinated services for its students and do a better job.

To me, it sounds like a further extension of Roland Freyer's work which found little evidence that any of the extra, non-school based interventions at HCZ were having an effect.

Does this mean we can't tackle education through neighborhood social services? Certainly not but you're presenting this in a very twisted way. This is a potentially greater indictment of full-service community schools with wraparound services as a school reform technique than this is a comment for or against charters.

Tom Hoffman said...

0 is not median! It is a percentile ranking!

What it means is that all this is descending into gibberish. I agree that Fryer's study actually purported to show that the non-school interventions in HCZ had little effect. Yet! that's not how it was presented by him or the people he used to promote the work. Yet! it was certainly his argument that the HCZ school was a nearly singularly impressive gap closing school. Yet! we all knew it wasn't really different in performance than other high performing charters. Yet! now we see it appears to be performing at a *much* lower level than other charters and neighborhood schools.

So... what? This is a giant mess.

Jason said...

Ah, I see the numbers.

Actually this report card is quite unclear on the growth numbers.

The ELA growth for all students at HCZ was 54 and the median growth in the bottom third of their students was 58.

The range of their "peer horizon" is from 52.9 to 79.3 for all students, and 58.0 to 88.7 for the bottom third.

The part that makes very little sense is that the city horizon (which should be a city wide comparison) in both cases does not include these scores at their bottom range. My only guess is that this is due to their dropping scores-- so basically both of these growth scores in ELA are slightly below 2SD below the median of the city-wide numbers.

The math numbers are pretty stellar, but they're really going nowhere with ELA, at least looking at growth.

That being said, I'm still a bit lost on what your wider point is-- are you claiming this is an indictment of 1) HCZ 2) Charters 3) Schools which provide wraparound social services as a means for school reform or 4) Some combination?

Tom Hoffman said...

My point is that given the role HCZ plays in Waiting for Superman, this should be part of the discussion. Geoffrey Canada and Joel Klein should be forced to explain what it means.

Bill Kerr said...

Canada's approach of early intervention and teaching parents how to parent would have to be the correct one. It works for the middle class. If the evidence suggests that it doesn't work then the only answer within the system would be to do more of it.

In my search to find out who Canada is I found this video interview with Charlie Rose.

Jason said...

http://d-edreckoning.blogspot.com/2010/10/replication-but-no-verification.html

Tom Hoffman said...

The entire discourse around HCZ is a hopeless mess. One thing De Rosa has wrong is that the goal of the project was to raise test scores.

I think we can all agree that the best way to raise test scores in the short term is to put kids in a small, well run school with a singular focus on raising test scores. Particularly if the school has a low population of the most demanding students.

Canada himself at least used to be outspoken about the fact that every initiative need not be justified by test scores.

And while HCZ is ambitious by philanthropic standards, it is still limited compared to what government is capable of. Who gets more comprehensive social services, a kid in HZC, or any kid in Canada?

Way to much is being piled on this project.

Having said that, given how G. Canada seems to have taken sides in the ed reform debate, as far as I'm concerned, it is open season on him.

Bill Kerr said...

I tried to reply but it became too long for blogger's annoying word length restriction, so it's here