Wednesday, January 26, 2011

Unfortunately, Our (Low) Collective Math Proficiency Prevents Us from Correctly Interpreting Our Math Proficiency Scores

It may be impossible to understate the negative effect that the design of the NECAP math exam has had on the recent education reform discourse and policy in Rhode Island. It is not because NECAP is a bad test. It looks pretty good to me, but I'm not qualified to say.

Actually, the way it is scored and scaled is probably ok as well, aside from the fact that almost nobody, including RIDE, politicians at every level, the media, and of course the general public, is able to understand the rigor of the test and thus interpret the various proficiency levels -- and their legitimate uses -- correctly. This can be done by simply comparing the proficiency rates on various assessments used across multiple states.

First we will look at setting NECAP math scores as targets for school improvement, in particular with the new turnaround goals for the Juanita Sanchez Complex (i.e., PAIS/Cooley) in Providence. Their 2012 goals for NECAP proficiency are:

Reading: 70%, Writing: 60%, Math: 50%.

Now, one measure of a successful turnaround is to exceed one's state average for students with similar demographics -- just looking at poverty is most convenient. And that's enough really, over time, if you can pull it off. It is the kind of line business-model reformers love: "If every year we can turn 5% of our lowest achieving schools into above average performers, in 10 years we'll be able to invade Singapore with our minds." Anyhow, here's the 2009 RI averages for "economically disadvantaged students:"

Reading: 61%, Writing: 43, Math: 12%.

Which one of these is not like the other? The reading and writing gains required by Juanita Sanchez Complex are ambitious but doable. Another open-enrollment PPSD high school in the neighborhood exceeded them in 2009 (before being named "persistently low-performing" and closed).

Why is the math expectation four times the state average for low-SES?

Here's where having a multi-state test is helpful. Lets look at the overall (all-SES) NECAP math proficiency rates for RI, New Hampshire and Vermont (2007, 2008, 2009):

Vermont: 30%, 35%, 35%.

New Hampshire: 28%, 32%, 33%.

Rhode Island: 22%, 27%, 27%.

Juanita Sanchez 2012 Target: 50%

One thing that is particularly worth noting is that there is little of the steady upward trend typical of new standardized test scores in this data (including at the school and district level data I've looked at). NECAP reading proficiency rates have gone up almost across the board. Math has barely budged in fits and starts, so there is no evidence to demonstrate a route to rapid gains.

Getting back to cross-state comparisons, it is common knowledge that Rhode Island is an educational underperformer. What about Vermont and New Hampshire? Let's look at the "Nation's Report Card," the NAEP proficiency rates and ranks in math. NAEP only goes up to grade 8, so that will have to suffice.

Massachusetts: 52%, #1.

New Hampshire: 43%, #4.

Vermont: 43%, #4.

Rhode Island: 28%, #37.

From this data you see that Vermont and New Hampshire are high performers nationwide, and that the raw proficiency rate for 8th grade NAEP is in the ballpark of 11th grade NECAP rates.

Finally, let's pull all the way back to an international comparison. While it is difficult to believe that every child in Singapore or Finland would not be proficient with distinction on the NECAP, this analysis by Richard Rothstein et al provides some context:

We can compare performance in top-scoring countries with NAEP’s proficiency standard. Comparisons are inexact—all tests don’t cover identical curricula, define grades exactly the same, or have easily equated scales. But rough comparisons can serve policy purposes.

On a 1991 international math exam, Taiwan scored highest. But if Taiwanese students had taken the NAEP math exam, 60 percent would have scored below proficient, and 22 percent below basic. On the 2003 Trends in International Mathematics and Science Study, 25 percent of students in top-scoring Singapore were below NAEP proficiency in math, and 49 percent were below proficiency in science.

What does all this mean? It is reasonable to conclude that RIDE's statewide goal of 47% proficiency on NECAP math by 2012 would entail RI effectively jumping from around number 37 today to number 1 in high school math by the time test administration takes place in October 2012. Effectively, there is a year and a half of instruction left to make that goal. It is not going to happen.

Even less plausible is the goal of jumping from 3.2% proficiency to 50% in a high poverty school in that same time period. To even set that goal demonstrates profound innumeracy and incomprehension of basic educational statistics. This is not analysis, or leadership. It is magical thinking. Or it is simply setting up schools to fail.

In fact, moving the Juanita Sanchez Complex's NECAP math proficiency to 15% by 2012 would be a big victory. It would, literally, indicate that the school was in the process of turning around. Getting over 20% by 2015 would probably mean outstripping the math achievement among low-income students in Vermont (18%), New Hampshire (17%), or any other state in the union.

In part two, I'll continue the analysis to using NECAP math as a graduation requirement.

No comments: