Friday, April 13, 2012

Getting Sankey

In this age of data-driven analysis, we're hitting a serious logjam around understanding the flows of students through and among schools. That is, we're getting increasingly detailed snapshots of student performance (on tests at least), tracking individual students longitudinally, and looking at school-level aggregated performance data, but there's not much public data on how students move between schools, within grade levels in schools, and between district and charter schools.

See, for example, Ursula Casanova's piece on grad rates in The Answer Sheet today.

Or KIPP's announcement that 89% of their students now "returned to their KIPP school or completed the highest grade offered at their school." That's nice, but whether it is good or bad is anyone's guess. Which kids are leaving and why? Are the leavers' test scores low, high, or average? What about retention rates?

Walter M. Haney's suggestion from 2006 is important:

States and LEAs should be required to report not just on test scores but also on grade progression ratios. As I have argued here, rates of student progress through the grades are a more robust measure of educational quality than are test scores. Also, as I have demonstrated, such data are vital in order to interpret test results. The apparent dramatic improvement in 2005 grade 4 NAEP scores in Florida are illusory. Not only is Florida not reducing the race gap, but data on grade transition rates for that state reveal that with 3-4 times as many minority than White students being flunked to repeat grade 3. Florida’s policies are helping to cement educational inequalities in place for years to come. Mssrs. Bush and Bloomberg are simply myopic and misguided. NAEP may provide some useful information on states’ educational progress, but as I have shown, if used in isolation as an “official benchmark for evaluating states' standards,” NAEP results may mislead more than inform.

Aggregate mobility data seems difficult to reconcile with direct experience. What does a 28% mobility rate (in PPSD 2007) really mean at the school level? Does that measure the same thing as KIPP's 89%? ProvPlan tells us that in 2008 over 10% of PPSD students left the RI public school system. What grade levels leave? To where? How many of those were considered drop outs?

States and districts should have all this data by now, what with the millions and millions of dollars spent on such things. The raw numbers would be difficult to process though -- this definitely calls for some data graphics. Something like a Sankey diagram (from Wikipedia):

Or perhaps like this one.

Here's a crude mockup:

In the chart above, the columns are grade levels in a middle school, the rows are school years, the arrows represent the size of flows of students between grades and in and out of the school, and the colors represent test scores of the whole school at the given grade level and how they are distributed to different incoming and outgoing flows.

Something like this would make it a heck of a lot easier to figure out what is going on with test scores.

No comments: