Friday, December 16, 2016

RIDE's AF data makes no sense and is bullshit

I revised my earlier analysis of RIDE and Brown's Achievement First fiscal impact memo as a post for RI Future:

Even if one accepts all of RIDE and RIIPL’s method and rationale for its fiscal analysis, the bottom line is that that all their calculations are derived from a single, early, clearly idiosyncratic study that overstates the positive achievement effects of AF compared to any other study I have subsequently found.

Friday, December 09, 2016

A Closer Look at Brown's Achievement First Fiscal Impact Memo

I've been sneaking peeks at the memo written for RIDE concerning the fiscal impact of the proposed expansion of the Achievement First network in Providence by 2,200 students while wrapping up Fall Forum and getting the next issue of Common Ground together. The memo was prepared by the Rhode Island Innovative Policy Lab (RIIPL) at Brown, which was started with a $4.3 million dollar grant from the Arnold Foundation, so any ed policy nerd can easily guess in what direction this story is going. The memo (that's what they call it, and that's what it is, not a study) is by Noah Kessler, Margarita Machelett, Miraj Shah, Jun Shepard, Justine Hastings, and included in the packet presented to the state Board of Education by the Commissioner last week.

Thanks to the ProJo for presenting some critical responses to the memo today. I was already working on an analysis of the calculations in the memo, and if this issue is hot today, I should probably get a quick (ha!) working version out ASAP. Thus this post. Consider it a draft.

The RIIPL memo is rather obviously a teetering tower of conjecture held up with a few high tension strands of research. While we can and should attack the foundation of this kind of thinking -- that the value of primary and secondary education can be measured in monetary gain or even college attendance -- as well as some of the basic suppositions -- do we really think that it is possible that every teacher in a system of seven schools will perform at one standard deviation above the mean every year? -- what really drives me around the bend is the mistakes and fudges that inevitably turn up in this genre of work with a little digging.

The basic chain of reasoning in the memo is: look at the performance of Achievement First elsewhere, extrapolate those gains over 13 years, run the performance numbers through Chetty, Friedman, & Rockoff's (2014) (CFR) findings to make predictions about future income and college completion, and finally compare that to other alternatives.

Or, as I like to picture it:

Test score growth -> The Chetty-izer -> predicted income growth.

The key piece of evidence here is really Justine S. Hastings (also an author of the AF memo), Christopher A. Neilson and Seth D. Zimmerman's 2012 paper on The Effect of School Choice on Intrinsic Motivation and Academic Outcomes. This is a study of choice in New Haven, with Achievement First's five flagship schools (and only these schools, apparently) representing charter school performance. This is a huge issue in itself, as the flagship schools aren't necessarily representative of the network as a whole across several states. In the district, students are apparently guaranteed a spot in a their neighborhood school (unlike Providence), with the option to apply for a large number of magnet schools:

Currently, half of the schools in the district (20) are magnet schools, and over a third of the students in the district (7,000) apply to them each year through a centralized lottery system. Of nine district high schools, eight are magnet schools open to choice, implying that almost all high school students either participate in school choice themselves or attend school with students who did.

In Providence, we don't really have magnets, other than Classical, but you can choose any district school through a lottery system. I would argue that in Providence's context any over-enrolled district school that you have to win a lottery to attend (Vartan Gregorian, Reservoir Elementary, etc... if anyone has a list, send me a copy) is functionally equivalent to a magnet school as defined in New Haven (remember, it is half the district there, not super exclusive). This is important because it affects the district baseline performance.

OK, let's step back to Chetty, Friedman, & Rockoff for a moment. This has become a very influential report because after some genuinely impressive number-crunching, they were able to show a correlation between, among other things, test scores and future income. Let's just say not everyone is a fan, and it is just one study. Nonetheless, this one of the key threads holding up the RIIPL memo, so it had better be used precisely. So, what data did CFR use in their original study?

We address these two questions using information from two administrative databases. The first is a dataset on test scores and classroom and teacher assignments in grades 3-8 from a large urban school district in the U.S. These data cover more than 2.5 million students and 18 million tests for math and English (reading) spanning 1989-2009. The second is selected data from United States tax records spanning 1996-2010. (page 3, emphasis mine)

CFR only looked at math and reading. We can only draw conclusions from CFR regarding math and reading. Keep that in mind...

The all important math and reading data that the RIIPL memo is going to plug into the Chetty-izer to get future income data comes from the New Haven paper:

We use estimates of AF value-added impact from Hastings, Neilson, and Zimmerman (2012) (HNZ). They find that test scores of children winning the lottery to attend Achievement First increase by 0.346 student level standard deviations.

Feed 0.346 into the Chetty-izer, assume every student at AF will see this effect every year K-12, and they get an increased income at age 28 for each former student of $10,954 - $13,451, extrapolated over a lifetime, that's $268,477 - $330,578.

OK, so, let's look at where this "0.346" number comes from in the New Haven paper. For this sort of thing I use the advanced research technique of searching for the "0.346" in the pdf. Here's where it comes from -- Table 7:

If you look under the third column, "Charter," the "Combined Z-Score" is 0.346. For an extra dollop of confusion, this is also the Z-Score for reading, but that is just a coincidence. Their combined Z-score is the mean of reading, writing and math. The problem is, again, that CFR didn't have writing scores (the word "writing" does not appear in the text at all), and thus we cannot make any CFR-based predictions using writing scores. We're feeding data into the Chetty-izer that it doesn't know how to process.

This is particularly important if you look more closely at the math and writing Z-scores the New Haven study found for AF. Math seems anomalously low and writing seems unreasonably high. It doesn't make any sense and my spidey statistical-noise sense is tingling.

The bottom line is that there is no justification for including writing scores in a CFR-based analysis, and without writing, the combined Z-score for AF goes from 0.346 to 0.127. As a quick recalculation, we can multiply all the subsequent estimates of income gain in the RIIPL by 0.367.

So... let's look back at those projected earnings. For our back of the envelope calculations here, if we assume the income increase goes down linearly with a reduced Z-score, instead of looking at $10,954 - $13,451 at age 28 for a student who had attended AF for 13 years, it would be more like $4,020 - $4,936. That is the difference between including or not including writing scores. Whichever number you believe, if either, there is no way those two numbers should be that far apart.

Another issue illustrated in table 7 is that in math and reading alone, according to Hastings, Neilson and Zimmerman, magnet schools outperform charters (AF). If we assume that over-enrolled Providence district schools are equivalent to New Haven's many magnets, the Chetty-izer would predict that students in the best Providence District Schools are getting a similar lifetime earnings benefit to their projections for Achievement First. This would need to be part of a serious analysis of impact. Not all children in Providence Public Schools are in the lowest performing schools.

There is an additional straightforward mistake in the calculation of the total cohort gains of an additional 2,200 students attending AF schools in Providence. They project the impact of 2,200 students attending AF from K-12, but since AF only proposes a single high school for the three K-8 schools, only 160 of each K-8 cohort of 276 students will get the full "treatment." By my calculations, to correct this their full cohort outcomes should be multiplied by 0.87.

So... my back of the envelope corrections would take the predicted increased lifetime earnings resulting from adding 2,200 seats to Achievement First Providence from RIIPL's $590.6 - $727.3 million down to $188.6 - $232.2 million.

One of the things which is particularly annoying about this economist-written memo about the financial return of a charter school is that they do not do the obvious thing of comparing their projected financial return to the student from this educational improvement to simply chucking money into an index fund. Let's say instead of having a lottery for additional seats at Achievement First, we had a lottery where the winners had $4,347 a year (the amount Providence has to send to AF with each child) deposited in an index fund with a 5% annual return. After 13 years, the student would have about $77,500. Let that sit for 10 years gathering interest (to $126,200), then at 28 start paying out a 40 year annuity of about $7,300. That's a total payout of almost $294,000 (well, if this calculator is accurate). That's nearly in the middle of RIIPL's predicted lifetime benefit of attending AF for 13 years, and basically double my corrected Chetty-ization.

The point is, in financial terms, this is not some kind of extraordinary payout. It is quite ordinary. All the numbers RIIPL throws around -- following the example of CFR's paper -- look quite impressive, but when you start talking about the full cost of educating a child, or a group of children, or cumulative earnings over a lifetime, all the numbers get bigger than you expect very quickly. Also, compound interest is a heck of a thing.

Finally, finally, Commissioner Ken Wagner wrapped up his ProJo op ed supporting the AF expansion by asking:

If one is opposed to the Achievement First proposal, what is the alternate plan for the children of Providence? And how is this plan different - truly different - from what we have already tried over the past 25 years?

I keep this link on auto-dial for occasions like this:

To the current reformers, integration is at best an irrelevance and at worst an excuse to shift attention away from shoddy teaching. But a spate of research says otherwise. The experience of an integrated education made all the difference in the lives of black children — and in the lives of their children as well. These economists’ studies consistently conclude that African-American students who attended integrated schools fared better academically than those left behind in segregated schools. They were more likely to graduate from high school and attend and graduate from college; and, the longer they spent attending integrated schools, the better they did. What’s more, the fear that white children would suffer, voiced by opponents of integration, proved groundless. Between 1970 and 1990, the black-white gap in educational attainment shrank — not because white youngsters did worse but because black youngsters did better.

Not only were they more successful in school, they were more successful in life as well. A 2011 study by the Berkeley public policy professor Rucker C. Johnson concludes that black youths who spent five years in desegregated schools have earned 25 percent more than those who never had that opportunity. Now in their 30s and 40s, they’re also healthier — the equivalent of being seven years younger.

According to the Chetty-izer an education in an integrated school for five years is equivalent to going to Achievement First for twenty-five years, and it is a lot cheaper. All we would need is the will to try something different - truly different - than we've been doing for the last 25 250 years.

Tuesday, December 06, 2016

Dropping the Curtain on the Coalition of Essential Schools

For the past few months I've been working part time with Jill Davidson to put on the final Coalition of Essential Schools Fall Forum -- their annual national conference. It is also the end of CES as an independent organization, more or less. It was Jill's conference; I was just assisting her vision. There are a bunch of reasons to gracefully wind up the organization at this point: structurally it has always been too loose and decentralized for the current funding climate, and the expanded reach of many of its principles has come through new organizations like Expeditionary Learning, Big Picture Learning, High Tech High, and Deeper Learning.

So we held the conference last week in Providence at the Omni to give everyone a chance for one last "conversation among friends." I suppose it is not my place to say so, but it went really well. We had between 400 and 500 people, which was enough for our financial requirements and enough to make the venue feel full -- but still small enough for Jill and I and a few other volunteers to give everyone quick and personal service. I could easily keep an eye on the 15 or so session rooms myself. The Omni staff was great; food was tasty; A/V support way beyond what teachers are used to.

The goal was to still have the same working conference for classroom teachers from Coalition schools that Fall Forum has always been, with a strong thread of reflection, nostalgia, and opportunity for closure as a community.

The emotional center of the conference -- at this point you might consider how many conferences you've attended that had an emotional center -- was its grand-matriarchs, Deborah Meier and Nancy Faust Sizer. Debbie's eyesight is failing so I (and I am sure many others) made it my mission to scurry around making her path as smooth as possible. Debbie and Nancy were in the middle of everything all weekend. Nancy Sizer's closing words were emotionally raw but perfect for the moment, and wrapping everything up with some second-line tunes from the Extraordinary Rendition Band worked better than I could have hoped.

From my point of view, even the screw-ups were kind of amazing. I'd sent out invitations for participating authors to do book signings, but the actual implementation was a bit ad hoc. I ducked into the main ballroom to finally get a bite to eat on the first full day of the conference and when I came out, Linda Darling-Hammond, Dennis Littky (of Big Picture), Debbie Meier, and George Wood (president of CES) were all sitting around a little table wedged in between the bookseller and another vendor table, having a grand chat. The only problem is that without a proper sign, etc., even if other people had realized it was supposed to be a book signing, it was a rather intimidating conversation to interrupt for a signature. It was a satisfying moment to stand back and observe, and maybe grab a young person and whisper, "Do you know who those people are!?!".

CES is kind of like an American labor union -- full participation is mostly based on where you happen to work -- and since I've never worked at a CES school, I've never been an active member. But CES and Ted Sizer's work at Brown is what brought me to Providence in the first place, and I have always thought of my educational philosophy first and foremost as a "CES-style progressive," so it was an honor to contribute to its legacy and do something for those who built it over the past three decades.