Thursday, January 27, 2011

Unfortunately, Our (Low) Collective Math Proficiency Prevents Us from Correctly Interpreting Our Math Proficiency Scores (Part II)

Following up yesterday's overly verbose look at NECAP math scores and school improvement targets, today we'll take a comparative look at NECAP math used as a graduation requirement, a proposal currently under consideration by the RI Board of Regents. Happily, I came up with some rather straightforward graphs this time.

A few preliminary notes. The MCAS, NECAP and NAEP all use a four point scale where 3 is proficient. 2 is something below proficient, and 1 is the lowest score (4 exceeds standards, ofc). In the case of the MCAS and proposed use of NECAP for graduation, you only need a "2" to graduate, so I focus on those levels, rather than proficiency numbers as I did previously. All the percentages below are % at "2" or above. Also, I apologize for the lack of links and references for the numbers. Digging this stuff up is a nightmare of PDF's, and I just couldn't keep track. Everything is as up to date and accurate as I could make it.

The first graph shows 8th grade NAEP math scores in RI, NH, VT and MA. This is the best cross-state data we have, and it establishes a clear pattern: MA is tops (2nd in the US in this case), NH and VT close behind (ranked 6 & 7), and RI clearly lagging (38th).

At the right of the graph, we've got the "2" or higher rate for the MCAS (for MA) and NECAP (for RI, NH, VT). Now, obviously these are different tests. The point of this graph is not to compare achievement between the states, but simply to note that the pass rates diverge, with the NECAP states all going down while maintaining their relationship to each other.

One explanation for this might be that MA has had more time to adjust to their test than the NECAP states. Indeed, scores have gone up considerably and consistently over the past decade on the MCAS, including, for example a 20 point jump in the fourth year of the test's administration. So in the second graph we look at 8th grade MCAS/NECAP scores to see if they diverge like the high school scores due to 10 years of MCAS practice. They don't. They're consistent with the NEAP scores, except perhaps the 8th grade MCAS is a little tougher relatively (data note, this is aggregated grade 3-8 math for VT, because it is all I could find).

Perhaps high school math instruction does fall off a cliff, at an equal pace, in NH, VT and RI, compared to MA. The only objective indicator here I could think of was math SAT. This is noisy of course due to participation rates, test design, etc., but the basic pattern holds. MA top, VT and NH close behind, RI lagging. Note that VT had a four point drop here from the previous year, when they were just one point behind NH. So this suggests that high school math achievement in does not wildly diverge between MA and the rest.

I don't know what to conclude from the above other than the high school math NECAP is quite a bit more difficult to pass as a graduation requirement than the MCAS. Maybe this is because the NECAP covers more difficult math in some absolute sense, maybe it is some feature of the design of the tests, maybe the NECAP just diverges from the existing curriculum in VT, NH, and RI in a way that is difficult to adapt to, maybe it is just harder to prep for.

In a sense, it doesn't matter. From the point of view of a kid, teacher or school, the 11th grade NECAP is harder to pass.

This goes against what seems to be the conventional wisdom among Rhode Islanders -- that we went with NECAP as an easier alternative to MCAS. The data does not back up that view. I see no reason to not believe that if Massachusetts switched to NECAP, the percentage of students eligible for graduation would drop by about 25%, from 93% to upper 60's.

I don't necessarily disapprove of the difficulty of the math NECAP, but it renders it inappropriate for use as a graduation requirement.

No comments: