Tuesday, May 07, 2013

How difficult is the 11th grade NECAP math test, especially compared to the MCAS?

The following is based on my testimony to the RI Senate Education Committee a couple weeks ago.  I've been working on getting it all cleaned up and laid out on two sides of a sheet of paper, and I'm almost done with that layout challenge, but in the meantime, here's the blog post version.

How difficult is the 11th grade NECAP math test, especially compared to the MCAS?

This analysis is simply based on outcomes, not item analysis, test design, curriculum alignment or other technical features.

The question is: from the point of view of past students, how difficult is it to get a passing grade?

First, note the importance of specifying the grade level.  RI (and NH/VT) students have consistently scored lower on the 11th grade math test than the grades 3-8 tests.  The pattern is easily discernible from last year's results for RI, in figure 1.

This difference in difficulty was understood and intended by the test designers (see page 308 of the 2007-2008 NECAP Technical Report).  It does not necessarily indicate a difference in teaching and learning performance between 8th and 11th grade.

It is important to acknowledge that if you look at the totality of math testing data for Rhode Island, we do underperform somewhat, even controlling for demographics.  There is no good national test for comparing high school mathematics, so we have to use some proxies.

The “gold standard” for cross-state comparisons is the federal NAEP test.  Unfortunately, 8th grade is the highest level at which it is regularly administered.  As you can see in figure 2, Massachusetts is #1 in the nation, but Minnesota, Vermont and New Hampshire are close to each other and not far behind MA.  Rhode Island lags MA by 10-15% across the achievement levels.

You can also see a similar pattern in SAT scores in figure 3. Minnesota is an outlier because of a low participation rate, but otherwise Massachusetts has the highest mean score (523) with Vermont (522) and New Hampshire (516) close behind, and Rhode Island trailing at 480.

Massachusetts and New Hampshire both particpated in a 2009 pilot of a a 12th grade math assessment. They ranked first and second respectively out of 11 particpating states.

Massachusetts and Minnesota participated as states in the international 2011 TIMMS exam, ranking sixth and seventh in the world.

To summarize: Massachusetts is generally recognized as the highest performing US state for mathematics through high school and a world class performer, but Minnesota, Vermont and New Hampshire are all follow closely behind the Bay State.

The next step is to compare, in figure 4, the scores of each of these states and Rhode Island on their respective high school math assessments. That’s the MCAS for Massachusetts, MCA-II for Minnesota, and NECAP for Vermont, New Hampshire and Rhode Island.

What we see is that students in Massachusetts score much better on the MCAS than students do on the NECAP or MCA-II. The difference is far more dramatic than any of the differences in performance indicated by same-test comparisons.

The straightforward conclusion is that despite the MCAS’s reputation, the NECAP (and the MCA-II) is much more difficult for students to pass.

One argument often raised when comparing NECAP to MCAS performance is that the difference is explained by student motivation, as the MCAS has been an established graduation requirement in Massachusetts for years, but the NECAP has not been until this year in RI. This may play some part, but if it explained most of the difference a similar distribution of scores would been seen in reading, and it is not, as can be seen in figure 5.

MCAS scores, and other test scores for Massachusetts, did increase significantly over time reflecting a coordinated reform program started in the late 1990’s. In the first year that 10th grade students were required to pass the MCAS in order to graduate, the percentage of students scoring the minimum “2” or more jumped 20 points.

As can be seen in figure 6, NECAP math scores simply have not increased the same way when added as a graduation requirement in Rhode Island, nor have they increased much in Vermont or New Hampshire. I included RI NECAP reading scores as a purple dotted line to underscore the difference in difficulty between the reading and math tests.

Of course, we do not know the future. Next year might be the year that RI NECAP scores will jump 20% if we just keep the faith and hold the line. On the other hand, the evidence suggests that, like NH and VT, we could jump to the top rank of states in math and still have over a third of our students scoring "Substantially Below Proficient" on the NECAP.

Data references:

No comments: