Monday, February 27, 2012

Comparing PPSD Schools & RI Charters Based on Free Lunch Rates

Gordon MacInnes and Bruce Baker have written convincingly about the problems with using free and reduced lunch eligibility to compare urban schools. Families below 130% of the federal poverty rate are eligible for free lunch; under 180% for reduced price lunch. These are almost always lumped together in publicly reported statistics, including those offered by RIDE.

However, the achievement gap between free and reduced lunch eligible students is almost as large as that between reduced lunch eligible and non-eligible students, as MacInnes points out:

NAEP reports results every two years by various subgroups, including one for students eligible for free lunch and another for those eligible for reduced lunch. In 2009, free lunch students scored a full 28 points behind the national average (204 vs. 232) on fourth grade reading. Reduced lunchers were 16 points back.

This is a very significant difference. To put it in perspective, New Jersey students were second highest on the 4th grade reading test, 12 points ahead of Tennessee, which was 47th. Moreover, gaps of the same magnitude are found on eighth grade reading and both fourth and eighth grade math tests. The gaps have remained stubbornly in place for twenty years. To dismiss such differences as "meaningless" is at least dubious, if not flat-out wrong.

Baker has more examples.

So... how does this play out in Rhode Island? You can get school by school numbers split out between free and reduced from the federal Common Core of Data (with 2009 - 2010 data most recent), and I've started working on a data graphic (which is by no means done), but the initial results are, well, a lot more lucid than I would have thought. It is all so tidy that I feel like I should chalk it up at least in part to coincidence, and while what follows is not a rigorous or authoritative statistical analysis, it should make a lot of sense to people with some familiarity with these schools. I certainly feel like I have a much clearer and more coherent mental model of what is going on now with urban schools in RI, charter and otherwise, than I did a couple days ago. So, with that preamble/disclaimer out of the way...

As you follow my chart showing free lunch rates per school, declining from left to right, you start with the Department of Children, Youth and Families Alternative Education Program, then second is the Segue Institute Charter in its first year with under 70 students, followed by 55 district schools, 42 in Providence, before you get to the next charter. In fact, the highest free lunch eligibility school in Central Falls is at number 48, believe it or not. That the poorest schools are in Providence is not surprising, but what is interesting is the way the remaining Providence schools and the urban charters tidily cluster together in three groups.

We're going to look at these groupings and take a very impressionistic look at overall test scores, using the current (2011-2012) NECAP scores, averaged over the whole school. There is no point in getting too dainty about this -- the schools cover different grade spans, the poverty rates might have changed, but I don't feel like digging through old data, etc. We are not trying to draw any fine distinctions here. I'd note that doing this more precisely would take a little work since the feds and the state don't code the schools the same way in their data dumps (of course). The scores will be in the format reading%, math%.

Group 1, .709 - .686 free lunch:

  • 57. Reservoir Avenue School (K-5 PPSD): 60%, 50%
  • 58. The Learning Community (K-8 charter): 73%, 60%
  • 60. Textron/Chamber of Commerce Academy (charter HS): 81%, 0%
  • 63. Nathaniel Greene Middle (PPSD): 71%, 51%

Notes: Reservoir and The Learning Community basically switched scores since 2010-2011 (Reservoir had 75%, 56%; LC, 60%, 50%). Greene includes the district gifted program. Also, wtf 11th grade math?

Group 2, .634 - .615 free lunch:

  • 70. MLK Elementary (PPSD): 59%, 43%
  • 71. Paul Cuffee Charter (K-10): 73%, 66%
  • 72. RFK Elementary (PPSD): 69%, 53%
  • 74. Times2 Academy (K-12 charter): 67%, 51%
  • 75. Nathan Bishop Middle School (PPSD): 54%, 44%

MLK's scores are down as the result of an influx of students from closed schools -- which might knock them out of this demographic slice now. Their 2010-2011 scores were 70%, 47%.

Group 3, .466 - .439 free lunch:

  • 98. Classical High School (PPSD) 98%, 48%
  • 99. Blackstone Valley Prep (test scores for 5-6): 68%, 72%
  • 100. Vartan Gregorian Elementary (PPSD): 61%, 66%
  • 101. Highlander Charter (K-8): 68%, 55%
  • 102. International Charter (K-5): 65%, 61%

Notes: Classical is the exam high school for Providence; BVP had only one year's worth of students and has added a couple schools since this demographic data came in.

Of course, the above is more meaningful if you know the context of the above schools. In particular, Gregorian and Classical are the cornerstones of the public education options for East Side white people (and white people citywide) and major demographic outliers compared to the rest of the city. So it is a bit shocking to see that Central Falls charters have almost exactly the same number of students eligible for free lunch, especially when charter school principals are quoted in national publications saying that compared to schools in the same area they have " a higher percentage of students living in poverty."

My takeaway here is that the "good" PPSD schools and urban charters are just a lot more similar than other analyses would lead you to believe, and their scores are mostly determined by demographics. From a parents point of view these are all good schools. But demographically, they aren't like the bulk of the PPSD at all. There were 29 PPSD schools with over 80% free lunch.

Finally, while I know how difficult it is for those very high poverty schools to achieve high test scores, I also know that it is possible, because I've seen it done. The double whammy is that those schools are too weak politically to sustain their gains, and that's the real reason that economic desegregation is necessary (not sufficient) to truly improving opportunity for all children in Providence and RI.


Sean said...

What is the percentage of free lunch range for schools 1-10, 11-20, 21-30, etc.? In other words, is there a large leap from, say, school 25 to school 58? I'm using Safari, which is resistant to D3.

Tom Hoffman said...

I'll look into the Safari issue and generally improve the data graphic as I learn more about D3, in the meantime, here's a quick summary:

1. 97%
2. 95%
3. 89%
then it is very gradual
10. 87%
20. 85%
30. 83%
40. 79%
50. 75%
60. 70%
70. 63%
80. 58%
90. 51%
100. 45%

Sean said...

lHmm. Still seems like there's a non-trivial difference between PPSD high-poverty and the PR machine charters.

That said, I'd love to take this further than the crude measures currently available (free lunch and reduced lunch). Say, average income within free lunch and average income within reduced lunch. So that we could make a statement like, "increasing average income by $1000 is associated with an average test score increase of .05 SD" or whatever.

Awesome, though. Disproving Faulkner's "Fiction is often the best fact." Here, fact is the best fact.

Tom Hoffman said...

For me, the real confirmation of the value of this is Reservoir. When I visited Reservoir and three other kindergartens in the neighborhood, it seemed clear that the main difference was the students. The principal and staff seemed good, but not that different than the rest of the schools, and they aren't allowed to be very different in terms of curriculum and instruction.

The free and reduced numbers say their demographics are the same as the other schools, but your eyes tell you they aren't. When you just look at free lunch, it makes sense.

That isn't a slam on Reservoir, it is just the reality.

Emlyn said...

I'm thoroughly fascinated by this piece, on several levels. I'm the parent of a 4-year-old, who will enter the system next year, and so I've recently done a lot of reading & research into the local system. (We're East side "white folks".)

So what is the conclusion about this data? In your estimation, are free lunch figures a good representation of academic performance? Why?

More to the point: can charter schools in any way claim to produce better results than PPSD ones, given that only certain types of parents (informed ones?) would even know to enter the lottery in the first place?

My neighbor happens to be the director of school support and accountability for RIMA, while my other neighbor is a teacher at the MET school. The MET school teacher & I both have our suspicions about the ultimate efficacy of the charter model, and whether or not it does indeed take away funds/resources/teachers form the PPSD.

We've had several strong debates, but our chief (unanswered) criticism of charter schools is that they don't appear to be scalable. It's one thing to throw a whole lot of money, teachers and attention at a handful of kids in a small, focused setting; some kind of best-case-scenario educational utopia--you're almost guaranteed to produce some positives. But it's quite another to scale up that [successful? do we even know?] model to serve every student in the state, rich, poor or otherwise.

Care to comment? I'd be very interested in your opinion.

Tom Hoffman said...

Hi Emlyn,

Sorry the original chart I refer to in the post is lost.

The Providence charter/PPSD comparison is unusual insofar as a lot of the leadership in PVD charters spent time in PPSD. So a lot of what is implemented in our charters was also done -- earlier -- to some extent in the district.

There has been an incredible amount of political and administrative sabotage in the PPSD in the past decade. This has nothing to do with the union, except maybe that politicians are destroying the PPSD as collateral damage in an attempt to destroy the union.

So you kind of have to decide if you're talking about a comparison between charters and the PPSD as it is or a counterfactual scenario where PPSD administration and the mayor's office hasn't been dismantling successful district schools and programs at the same time the mayor and RIDE are pushing charters.

Of course, for the parent of a four year old, this hypothetical isn't so helpful.

In practical terms, there is a huge range in the student bodies of PPSD elementary schools, even outside the East Side. I believe all last year in my daughter's kindergarten class at Reservoir, one student left and one arrived. At the same time, a teacher friend at another nearby school reports 20-30% of classes changing every year.

The real issue is concentrations of the neediest kids. There are schools in PVD where half, two-thirds or more have some aspect of their life story would blow your mind. Escaping political violence in another country, experiencing violence here, incarcerated parents, and just crazy complicated stuff, like on parent deceased, one institutionalized and the student has some kind of serious medical condition.

And yes, these are the kids who are going to tend to not end up in charters or magnets.

The thing is, that would not be such a problem if we weren't investing so much in the idea of accountability and competition, and if we were willing to accept that the schools with the most difficult students do in fact need a lot more resources.

But in the end, functional school districts make more sense. To be honest, the best thing for Rhode Island, if you had to pick one thing, would be to consolidate the whole state into one or two disricts, and to manage the charter/magnet/etc. schools as part of an overall coherent strategy with a major focus on economic desegregation.

To pull this back to the original post a bit, people who haven't spent much time in these schools (and those who have who don't look at demographic data) don't realize how much of a difference moving a school from 80-90% high-needs kids to, say, 50% or 60%. At 80-90%, the school just collapses as an institution, whether internally or because it doesn't have enough clout to protect itself from politicans.

I don't know if that rant was helpful. ;-)

Emlyn said...

Very interesting.

So when you say "moving a school from 80-90% high-needs kids to, say, 50% or 60%" how is this "move" achieved? Do you mean moving them from the PPSD to independent/charter schools (which are by lottery anyway, right?) or moving them to another PPSD school that can take the overburden?

But I guess this produces 4 questions in my mind:

1. There's the question of politics and school closings (PPSD and charter?) when the numbers aren't good or, as you suggested, when the mayor/politicians are playing politics. Did those schools actually merit closure..? Are RI politics actually partially to blame for RI's educational landscape?

2. There's the question of whether PPSD schools (since charter school enrollment is so low) can handle the influx of students, both from closed schools and regular incoming students. And, since PPSD schools are funded by local taxes, in what zones are these remaining schools located? ie: Do those local taxpayers end up subsidizing students from non-local areas?

3. Does the new RIDE formula take taxpayer funds (or is it federal funds) and allocate these proportionately to both PPSD and charter schools? Any idea how this is decided? Given how little enrollment there is--and non-guaranteed--diverting some funds to charter schools is effectively throwing up our hands at the PPSD schools and leaving them to plod along at half-mast.

4. Is, ultimately, the charter model SCALABLE? Can whatever [defensible] performance data, produced by these schools, actually be replicated--not just district-wide, not just city-wide, but STATE-WIDE, so that whatever is spent, learned, invested or otherwise produced from these schools necessarily finds its way toward eventually serving ALL future kids and not just a lucky few who might benefit from an [as-yet temporary] "alternative" educational opportunity? So far I've not come across a satisfactory answer to this question which, in my view, is the most crucial one. For if charter schools are just some philanthropist's stomping ground, or educator's lab experiment, and it can't be scaled up to serve all, then it's really all just academic...and probably a waste of money and resources, at that.

Any perspectives on these?


Tom Hoffman said...

Hi Emlyn,

On the first point, I mean first getting rid of district lines, which then gives you more leeway to shuffle kids around and balance out the school populations. Move a few kids from Cranston into Providence, vice versa, etc. There are quite a few districts (or county-based systems) in the country as big as Rhode Island.

Of course, this is "impossible" politically, although we'll never know if, say, a $75 million grant from the feds would have greased the skids a bit.

But unlike things which are "impossible" because they are expensive, or "impossible" because they just don't work (e.g., merit pay), reorganizing the state for educational equity is just something we don't want to do. But we could do it, and it would work as long as we stuck with it.

re: 1. I don't think there is any reasonable doubt at this point -- when we're suddenly talking about OPENING a middle school a couple years after closing one -- that the process of opening and closing schools has been deeply corrupted over the past five years, by all kinds of political motivations and ambitions. To argue otherwise flies in the face of common sense.

I'm not sure what you mean by 2.

re:3. The funding formula allocates the student's share of all money -- federal, state and local -- to the charter school or district they attend. It isn't quite that cut and dried because the district is still on the hook for transportation to charters and a few other things.

But yes, once you start pulling more and more money and kids out of a district, it is a death spiral. You can't neatly scale down a school, and the social cost of closing one is usually tremendous.

re:4 One thing that is telling is that in a recent state by state study of charter performance (I think by CREDO), RI scored very high. You won't hear RI's charter policies praised by national advocates though, because the obvious conclusion would be that strictly controlling the size and rate of growth of the charter sector is important for quality. You could say the same thing for Mass.

On the other hand, schools with wide-open charter expansion, like Ohio, tend to have lousy performance.

I think that answers the scalability question.

Emlyn said...

Apparently the Credo study has been widely criticized, for several reasons. But, more problematic, is that the New Orleans school districts were almost totally wiped out by hurricane Katrina, along with most of the city's infrastructure. With a massive subsequent influx of funds, resources and attention, their educational system got more than just a shot in the arm but a complete overhaul, literally from the ground up. When you have nothing to lose--and no existing system competing with it--pretty much *anything* is going to be an improvement.

So, for that reason, I don't find the Louisiana example credible or representative as the conditions there were unique, in the extreme. And hardly replicable.

By #2 I was referring to your mention of MLK and how it was forced to absorb kids from other closed schools. If part of MLK's funding, as with all RI schools, comes from that local tax base, are those taxpayers not subsidizing non-local students? It's unclear to me how the funding is reconciled.

But, even more importantly, is the strain on MLK's resources impacting its academic performance? This would seem to be a serious concern if it's operating at/above intended capacity.

Anyway, as you've no doubt deduced, I'm very skeptical of the charter model. Its aims, at least on the surface, are noble, but it doesn't take a rocket scientist to conclude that it ultimately leaves public schools with less of everything...and that's the last thing they need. I'll have to check out the Ohio example; that sounds like a more real world one.

The question is: how could these experiments in education have been done *within* the existing public system? I've heard it said that this would have been "impossible", given the structure/history/politics of old institutions. But if the greatest criticism of charter schools is that they're detracting from the public ones, then perhaps the way forward, one that satisfies all, is to [somehow] create this experimental environment inside the existing schools where real world forces/influences, student demographics and resources are in play.

Thanks again for your excellent perspectives here; it's refreshing to see a candid assessment of all this. I come from the [old] South African school system--essentially the British system--and seeing this bizarre disarray here is fairly shocking.

Tom Hoffman said...

Ah... I don't know if you noticed but I'm actually in Scotland for the year, so I'm learning a bit about British education.

In terms of, say, MLK, I don't think the funding follows the student to the district school in any specific way. The funding follows the student to the district as a whole. Within Providence, it is just one pool of money. Neighborhood money has no relationship to neighborhood schools (well, aside from fund-raising).

I don't know if there is or will be a long term effect on MLK, but there certainly was a short term one, particularly since the NECAP is given in October. I've got a post somewhere looking at that. They clearly just got a lump of low scoring kids at upper grades when the other schools closed, and, of course, there is no way to factor that into the various accountability systems.

Long run, I don't know.

When I got to Providence in 1999, there was a well established system of "site-based management" for many schools in the district (including Gregorian), and the expectation was that all new schools would be site-based. I helped re-design Feinstein High School as a site-based school, and we were encouraged to "break the mold" (or whatever). We took a lot from The MET, a lot from the Coalition of Essential Schools model originally developed at Brown. We had a huge amount of freedom up to and including defining standards-based graduation requirements with no credit hour (or test score) requirements, and coming up with contract variances through a formal process involving the school, administration and the union.

Site-based management and most of the schools that practiced it were systematically destroyed over the past 10 years, mostly by management (the union was always a bit ambivalent).

This was all mostly the result of NCLB and test score pressure, pressure to standardize, etc.

It is a political problem, but it is not because of "old institutions." It is more because of the new ones, and the lack of political power in an overwhelmingly low income school community that doesn't have a PR budget and powerful (enough) friends.

And the things that are taken for granted now were just unthinkable a few years ago. Spending money to get more students to apply to an already oversubscribed school? Why on earth would we do that? Public relations? Lobbying? We're A SCHOOL.

Emlyn said...

Actually yes, I did notice...I thought maybe you were an English transplant in RI. Or are you from here?

"Neighborhood money has no relationship to neighborhood schools (well, aside from fund-raising)."

I thought this was, largely, how funding was calculated for local schools? Why is it, then, that Barrington has always had such desirable schools? People move there for the schools.

Site-based management does seem to be the most sensible approach for different schools with different socioeconomic-political challenges. Had that model been working?

Yet I come at this, from my own experience, believing that some form of standardization--even just an agreed-to measure of what an "education" is--is crucial in nurturing the next generation of citizens. Otherwise, don't we risk essentially "profiling" students in certain areas to be "less likely" to under-perform? I guess that goes deep...

Yes...PR for schools...what is THAT all about. No wonder Cranston residents protested so vociferously; the stench of [potentially?] for-profit maneuvering is unmistakable.

I don't honestly know what's going on politically in this state. But someone pointed this out to me:

* Blackstone Academy just got a $2.2 million dollar grant to expand, and the group that gave it to them is a foundation associated with the Broad Foundation...which is the think tank that trained Commissioner Gist.

* They went on to explain that when Achievement First Lost their bid to start 2 charter schools in Providence, Gist apparently gave the job of running the charter schools to one of its administrators, who is now at RIMA (if I'm understanding him correctly).

* BUT, Achievement First are heavily influenced by the Broad Foundation, if not linked at the hip. So now we have Blackstone, Gist's clear favorite, getting 2.2 million from the foundation that trained her, which was already linked to Achievement First, and is now overseen by this Gist-appointed RIMA person...who worked for Achievement First.

Needless to say, this does strike one as being a bit fishy (but not at all surprising in the Land of Opportunity, right?). I'm not sure what to make of it.

Tom Hoffman said...

I'm from Pennsylvania, in RI for 15 years now (time flies).

I think we might be talking about different things, but Barrington is a different district (at this point). If you moved to Barrington, your kid's share of state and federal money would go there. If you moved to the other side of Providence and switched PPSD schools, the money is all still going to PPSD (And distributed however they do it. PPSD schools don't have their own personnel budgets, for example).

The reformers are a big keiretsu.