I've been sneaking peeks at the memo written for RIDE concerning the fiscal impact of the proposed expansion of the Achievement First network in Providence by 2,200 students while wrapping up Fall Forum and getting the next issue of Common Ground together. The memo was prepared by the Rhode Island Innovative Policy Lab (RIIPL) at Brown, which was started with a $4.3 million dollar grant from the Arnold Foundation, so any ed policy nerd can easily guess in what direction this story is going. The memo (that's what they call it, and that's what it is, not a study) is by Noah Kessler, Margarita Machelett, Miraj Shah, Jun Shepard, Justine Hastings, and included in the packet presented to the state Board of Education by the Commissioner last week.
Thanks to the ProJo for presenting some critical responses to the memo today. I was already working on an analysis of the calculations in the memo, and if this issue is hot today, I should probably get a quick (ha!) working version out ASAP. Thus this post. Consider it a draft.
The RIIPL memo is rather obviously a teetering tower of conjecture held up with a few high tension strands of research. While we can and should attack the foundation of this kind of thinking -- that the value of primary and secondary education can be measured in monetary gain or even college attendance -- as well as some of the basic suppositions -- do we really think that it is possible that every teacher in a system of seven schools will perform at one standard deviation above the mean every year? -- what really drives me around the bend is the mistakes and fudges that inevitably turn up in this genre of work with a little digging.
The basic chain of reasoning in the memo is: look at the performance of Achievement First elsewhere, extrapolate those gains over 13 years, run the performance numbers through Chetty, Friedman, & Rockoff's (2014) (CFR) findings to make predictions about future income and college completion, and finally compare that to other alternatives.
Or, as I like to picture it:
Test score growth -> The Chetty-izer -> predicted income growth.
The key piece of evidence here is really Justine S. Hastings (also an author of the AF memo), Christopher A. Neilson and Seth D. Zimmerman's 2012 paper on The Effect of School Choice on Intrinsic Motivation and Academic Outcomes. This is a study of choice in New Haven, with Achievement First's five flagship schools (and only these schools, apparently) representing charter school performance. This is a huge issue in itself, as the flagship schools aren't necessarily representative of the network as a whole across several states. In the district, students are apparently guaranteed a spot in a their neighborhood school (unlike Providence), with the option to apply for a large number of magnet schools:
Currently, half of the schools in the district (20) are magnet schools, and over a third of the students in the district (7,000) apply to them each year through a centralized lottery system. Of nine district high schools, eight are magnet schools open to choice, implying that almost all high school students either participate in school choice themselves or attend school with students who did.
In Providence, we don't really have magnets, other than Classical, but you can choose any district school through a lottery system. I would argue that in Providence's context any over-enrolled district school that you have to win a lottery to attend (Vartan Gregorian, Reservoir Elementary, etc... if anyone has a list, send me a copy) is functionally equivalent to a magnet school as defined in New Haven (remember, it is half the district there, not super exclusive). This is important because it affects the district baseline performance.
OK, let's step back to Chetty, Friedman, & Rockoff for a moment. This has become a very influential report because after some genuinely impressive number-crunching, they were able to show a correlation between, among other things, test scores and future income. Let's just say not everyone is a fan, and it is just one study. Nonetheless, this one of the key threads holding up the RIIPL memo, so it had better be used precisely. So, what data did CFR use in their original study?
We address these two questions using information from two administrative databases. The first is a dataset on test scores and classroom and teacher assignments in grades 3-8 from a large urban school district in the U.S. These data cover more than 2.5 million students and 18 million tests for math and English (reading) spanning 1989-2009. The second is selected data from United States tax records spanning 1996-2010. (page 3, emphasis mine)
CFR only looked at math and reading. We can only draw conclusions from CFR regarding math and reading. Keep that in mind...
The all important math and reading data that the RIIPL memo is going to plug into the Chetty-izer to get future income data comes from the New Haven paper:
We use estimates of AF value-added impact from Hastings, Neilson, and Zimmerman (2012) (HNZ). They find that test scores of children winning the lottery to attend Achievement First increase by 0.346 student level standard deviations.
Feed 0.346 into the Chetty-izer, assume every student at AF will see this effect every year K-12, and they get an increased income at age 28 for each former student of $10,954 - $13,451, extrapolated over a lifetime, that's $268,477 - $330,578.
OK, so, let's look at where this "0.346" number comes from in the New Haven paper. For this sort of thing I use the advanced research technique of searching for the "0.346" in the pdf. Here's where it comes from -- Table 7:
If you look under the third column, "Charter," the "Combined Z-Score" is 0.346. For an extra dollop of confusion, this is also the Z-Score for reading, but that is just a coincidence. Their combined Z-score is the mean of reading, writing and math. The problem is, again, that CFR didn't have writing scores (the word "writing" does not appear in the text at all), and thus we cannot make any CFR-based predictions using writing scores. We're feeding data into the Chetty-izer that it doesn't know how to process.
This is particularly important if you look more closely at the math and writing Z-scores the New Haven study found for AF. Math seems anomalously low and writing seems unreasonably high. It doesn't make any sense and my spidey statistical-noise sense is tingling.
The bottom line is that there is no justification for including writing scores in a CFR-based analysis, and without writing, the combined Z-score for AF goes from 0.346 to 0.127. As a quick recalculation, we can multiply all the subsequent estimates of income gain in the RIIPL by 0.367.
So... let's look back at those projected earnings. For our back of the envelope calculations here, if we assume the income increase goes down linearly with a reduced Z-score, instead of looking at $10,954 - $13,451 at age 28 for a student who had attended AF for 13 years, it would be more like $4,020 - $4,936. That is the difference between including or not including writing scores. Whichever number you believe, if either, there is no way those two numbers should be that far apart.
Another issue illustrated in table 7 is that in math and reading alone, according to Hastings, Neilson and Zimmerman, magnet schools outperform charters (AF). If we assume that over-enrolled Providence district schools are equivalent to New Haven's many magnets, the Chetty-izer would predict that students in the best Providence District Schools are getting a similar lifetime earnings benefit to their projections for Achievement First. This would need to be part of a serious analysis of impact. Not all children in Providence Public Schools are in the lowest performing schools.
There is an additional straightforward mistake in the calculation of the total cohort gains of an additional 2,200 students attending AF schools in Providence. They project the impact of 2,200 students attending AF from K-12, but since AF only proposes a single high school for the three K-8 schools, only 160 of each K-8 cohort of 276 students will get the full "treatment." By my calculations, to correct this their full cohort outcomes should be multiplied by 0.87.
So... my back of the envelope corrections would take the predicted increased lifetime earnings resulting from adding 2,200 seats to Achievement First Providence from RIIPL's $590.6 - $727.3 million down to $188.6 - $232.2 million.
One of the things which is particularly annoying about this economist-written memo about the financial return of a charter school is that they do not do the obvious thing of comparing their projected financial return to the student from this educational improvement to simply chucking money into an index fund. Let's say instead of having a lottery for additional seats at Achievement First, we had a lottery where the winners had $4,347 a year (the amount Providence has to send to AF with each child) deposited in an index fund with a 5% annual return. After 13 years, the student would have about $77,500. Let that sit for 10 years gathering interest (to $126,200), then at 28 start paying out a 40 year annuity of about $7,300. That's a total payout of almost $294,000 (well, if this calculator is accurate). That's nearly in the middle of RIIPL's predicted lifetime benefit of attending AF for 13 years, and basically double my corrected Chetty-ization.
The point is, in financial terms, this is not some kind of extraordinary payout. It is quite ordinary. All the numbers RIIPL throws around -- following the example of CFR's paper -- look quite impressive, but when you start talking about the full cost of educating a child, or a group of children, or cumulative earnings over a lifetime, all the numbers get bigger than you expect very quickly. Also, compound interest is a heck of a thing.
Finally, finally, Commissioner Ken Wagner wrapped up his ProJo op ed supporting the AF expansion by asking:
If one is opposed to the Achievement First proposal, what is the alternate plan for the children of Providence? And how is this plan different - truly different - from what we have already tried over the past 25 years?
I keep this link on auto-dial for occasions like this:
To the current reformers, integration is at best an irrelevance and at worst an excuse to shift attention away from shoddy teaching. But a spate of research says otherwise. The experience of an integrated education made all the difference in the lives of black children — and in the lives of their children as well. These economists’ studies consistently conclude that African-American students who attended integrated schools fared better academically than those left behind in segregated schools. They were more likely to graduate from high school and attend and graduate from college; and, the longer they spent attending integrated schools, the better they did. What’s more, the fear that white children would suffer, voiced by opponents of integration, proved groundless. Between 1970 and 1990, the black-white gap in educational attainment shrank — not because white youngsters did worse but because black youngsters did better.
Not only were they more successful in school, they were more successful in life as well. A 2011 study by the Berkeley public policy professor Rucker C. Johnson concludes that black youths who spent five years in desegregated schools have earned 25 percent more than those who never had that opportunity. Now in their 30s and 40s, they’re also healthier — the equivalent of being seven years younger.
According to the Chetty-izer an education in an integrated school for five years is equivalent to going to Achievement First for twenty-five years, and it is a lot cheaper. All we would need is the will to try something different - truly different - than we've been doing for the last 25 250 years.
No comments:
Post a Comment