Friday, March 05, 2010

About that 11th Grade NECAP Math Test...

Central Falls' 7% proficiency rate on the NECAP math test has now been widely quoted in the national media. What hasn't also been mentioned is how poorly low-income students do on the math portion of the NECAP overall. Here are some proficiency rates for economically disadvantaged students:

  • Vermont: 18%
  • New Hampshire: 17%
  • RI: 12%
  • Providence: 9%

Classical boosts Providence's overall score. Central Falls actually outscores all of PPSD's open enrollment high schools.

Vermont and New Hampshire both do well on 8th grade NAEP math, including relatively well among low-income students. They're doing better than RI, but two out of ten is not that much better than one out of ten.

Perhaps most striking to me is that the Times2 Academy, a math/science oriented charter in Providence, which has had many if not most of its 35 11th graders since kindergarten, and sports a 94% pass rate in reading, only got 28% of its juniors over the bar.

So anyhow, I finally looked at the latest released items from the test, which is administered to 11th graders in October, and, um... it is pretty hard. I should just spare you the cliches, point out some more relevant info, and suggest that if my more math-oriented readers want to take a look and let me know what they think, I'm certainly curious at this point.

The one thing I was thinking was perhaps the curriculum is so mis-aligned that kids taking the test just haven't even been exposed to enough geometry. That's certainly possible, but the poor scores seem to be uniform across all topics, so it isn't only that.

19 comments:

Chris Lehmann said...

I was wondering how those insanely low numbers for both Central Falls and MET. I go back to my favorite idea -- no legislature should be allowed to pass a law mandating a high-stakes test without taking the test themselves and having *their* scores published in the paper, like ours are.

And hey, easiest way to denigrate schools -- make the test really hard and then declare that anyone who didn't pass it isn't proficient.

Is this really where we are in our public debate?

din819go said...

I have no problem with hard tests. What I have a problem with is the curriculum not being aligned to the test the kids have to take at the end of the year. If the test is hard and GOOD what is wrong with aligning a curriculum to the test? Isn't the purpose of the test to determine if the kids have MASTERED the subject matter? What am I missing? Thanks!

jd2718 said...

I like this test, but not for high stakes. It's roughly equivalent to our Algebra 2 exam, though without the trig, with fewer topics, but with trickier arithmetic, algebra, graphs...

Every single question, as written, is too hard for the NY State Algebra (algebra 1) exam, the one required for graduation.

I'll write it up, and provide a pointer. I assume most will agree that 7% is not at all shocking from a high poverty school.

John said...

So, as we play out the chess game, someone in authority (to borrow for Gist) can have a feeling that the union proposals are bad, stop talking to the union, and choose a school according to his or her own calculations (even if that school is doing better by the chosen metrics) and fire everyone before a deadline. And that is not supposed to have a chilling effect? The politician is free to select his or her own scapegoat to demonize for political effect, thus creating examples for other unions who don't accept the mandated terms. And the government will further incentivize non-union shops (charters)?

Isn't that what caused the urban educational crisis? In the 70s, corporations rejected union offers to modernize and governors got into a bidding war to attract rust belt industries. Then Reagan made an example of an unpopular union (one that had supported him in 1980)and Supply Side economics gave incentives to close Brown fields factories that weren't profitable enough and more incentives to move to the Green Fields of the exurbs and the couuntryside, where they didn't have to pay for the social costs of history.

Drive through the great swatch of Blue states from the Applachians to the Ozarks to the Piney woods of Texas where Obama lost and you'll find empty factory after empty factory. The union-busters, having driven down wages in America (while boosting wages some in the Sunbelt) then moved the jobs south of the border then to China and then to Southeast Asia.

How is that divide and conquer playbook different than the RttT-inspired closings of today? When Reagan selected his scapegoat, the entire progressive community was appalled, even if we didn't care for that union.

I wonder if "reformers" have even read this history, much less seen how it worked out in our neighborhoods and schools.

Someone needs to tell "reformers" who loved the Wire that it was not just a movie. They've adopted the same playbook. The New Yorker article on Jackson Whites should be a reminder to liberal reformers who love Bruce Springsteen. Johnny 99, which started with "They closed down the auto plant in Mahwah late last night, went looking for job but couldn't find none ..." wasn't a defense of violent crime, but it concludes "I got debts that no honest man can pay, the banks holding the mortgage and taking my house away, and it was more than all this that put that gun in my hand."

Somehow, reformers think they can take the schemes that deindustrialized America, apply them to schools, and not worsen generational poverty.

Kate Nowak said...

I don't think the test is very hard. The questions are fair and test things students should have learned by the time they complete Algebra 1 and Geometry, which most students do by tenth grade. There isn't any Algebra 2 or Trigonometry content - unlike say the SAT's.

Maybe New England should just do what NY does, and only require a 34% to pass. No, really. That's what we do. So much easier than writing decent curricula for teachers and helping them improve instruction, and the governor and mayors can still go on TV and tout gains in their passing rate.

Tom Hoffman said...

It does seem like the questions are well designed and require a real conceptual understanding. I don't know how unusual the construction of these problem is, how varied their design is over time, and how susceptible to drill they'd be -- I'd guess not very, which would mean you'd have few tricks for jacking up the scores.

Actually, one thing I was wondering was whether some of this stuff is supposed to be covered in Algebra I or Algebra II. I don't know the difference at this point.

Anyhow, I'd suspect some kind of covert or overt adjustments in the difficulty of passing this at some point...

Kate Nowak said...

I'd say the algebra is all pretty solidly Algebra 1: Graphing lines, quadratics, a pretty tricky probability question, unit conversion, scientific notation, operations on real numbers, basic function vocabulary, writing an equation to fit a pattern or a verbal description.

Algebra 2 would be like exponential functions, logarithms, polynomial and rational functions, complex numbers, transformations of functions.

Kate Nowak said...

Really, JD? I wouldn't have been shocked to see any of these on a Math A. But they'd be the harder Math A questions. There aren't any giveaways on this, like there usually are on a NY Regents. Unless they only included harder questions in the samples they released.

Tom Hoffman said...

din819go,

What I really have a problem with is politicians throwing around these numbers (7%!!!) in isolation as proof that a school is "failing," especially when a comparatively good proficiency rate, say 25% sounds no better, and for that matter, a lot of people would say 50% is low enough to call the school "failing."

And throwing around the math number as damning when by most other measures, a school may be doing fine, including college admissions, college retention, etc.

jd2718 said...

Kate,

we are going to disagree, at least in part, here.

Questions 6, 7 and 15 involve the concept of function, not Algebra 1 (in NY)

You mentioned geometry questions (2, 5, 14, 19) - but these are also above Algebra I. The point value for 19 on our geometry regents would be high.

#s 1, 9, and 13 are numeric, but far harder than anything that would show on our state exam.

3, 4, and 8 we might have. 12 we could have, but not as multiple choice, and probably with a higher point value.

If 11 appeared on the NY Algebra Regents, there would be stories in the papers about a "trick" question.

12 is harder probability than we will ever see on the algebra regents.

The graph reading skills in 16 and 17 are not in our curriculum. (In 18, probably yes)

21, 22, 23 could be our questions.

But 20, I'm going to blog 20. I wonder how most Algebra II kids would fare. That is hard.

- - -- --- ----- --- -- - -

In urban areas in New York, most students do not complete geometry until the end of 11th grade. Many not until they are finishing high school.

I agree, that this is not a hard test, but for those barely getting through, this would be nearly impossible.

On the other hand, if I had to teach to a test, I'd far prefer this than any of our Regents.

Jonathan

Tom Hoffman said...

I noticed the lack of giveaways in the released items. I would think that pattern would tend to lead to the *very* low pass rates, compared to a test where at least 10% to 15% of kids in any school could get within striking distance on easier items and then get lucky on just enough of the harder ones.

Also, I don't know if the ordering is the same as the actual test, but I thought at least a few of the written responses would have been easier to make some headway on. My perception is that kids who are marginal to the whole process (for a variety of reasons) are most sensitive to ordering -- start out with five hard problems and they might not even look at the rest, so depending on the results you want, you can probably get quite different spreads depending on how you manipulate that aspect.

Kate Nowak said...

Function is in Integrated Algebra now. Notation, domain, range, which of these graphs/set of ordered pairs is/is not a function.

I didn't say it was nothing beyond Algebra 1. I said "Math A", which was Algebra 1 + Geometry without proofs.

Also - huge distinction - they have a no-calculator part.

But I guess I'm in agreement with you big-picture wise - many of these sample items seem too difficult for a minimum-competency must-pass-to-graduate test, especially if they maintain a reasonable cutoff for passing.

Tom Hoffman said...

Currently, passing this test isn't required for graduation in RI. I think there may be some kind of long term plans to that effect, but I'm not sure. The results are, as we've seen, high stakes for schools however.

Kate Nowak said...

So, it's high stakes for the schools, but not the kids? Do kids have any reason to take it seriously, I wonder? Is it counted as a final exam for a class, or something? Because, this seems like a test you'd want to prepare for, and have to give your best effort.

John said...

Your discussions of test questions are invaluable from an educational perspective. From a legal perspective they are just as valuable. If political leaders, using flawed data, can select whatever school they want to target, they are not required to do so in a dishonest or destructive way. But when you look at Central Falls, it is clear that those leaders are being empowered to place whoever they want on a hit list for whatever reason they want, and then still be able to use that flawed data that to find “a rationale that makes sense” (to use Gist’s description of her decision-making process).

But Secretary Gist recently described the decision to fire all teachers saying,“What was happening before the selection was made was not a negotiation.” and “Whether they (the teachers) say they were supporting the transformation model or not, they say they are willing or not, that part does not factor in.”

But The National Labor Act requires good faith negotiations. Whether its applicable in this case, I can’t say. But focusing only on the spirit of labor law what are we to make of Gist’s comment that if “there is something that the superintendent feels is a message to her” that indicates that something is wrong with the union’s models, she (Gist) will back the superintendent.

So three weeks before the deadline, management can just reject unions proposals and fire everyone, as if that does not send a chilling message to other workers, as well as teachers?

I hope my historical ramblings are seen in that context. When Reagan busted the air traffic controllers union and passed Supply Side economics, corporations were not required to take the incentives to close profitable factories and reopen them in the suburbs or overseas. They were not required to accelerate the de-industrialization of America, breaking families and creating the despair that still cripples so many neighborhood schools.
When Reagan allowed banks and savings and loans to use financial engineering to produce a bubble and a bust, not all bankers did so, but anyone who wanted to use statistical engineering to get rich by devastating society was free to do so. (In “reform” circles they speak of “expectations” along with data as if it was VooDoo. In addition to “VooDoo” economics, in Oklahoma bankers were allowed to use the theory that infinite gas reserves might exist to justify the financial engineering that wiped out almost every good-paying factory job in only 15 years) Bush and Greenspan did not require speculators to drive our economy in the ditch, just like NCLB, the RttT, and other reforms don’t require destructive policies.

I just use Oklahoma as an example because I saw these tragedies unfold, and daily I deal with their legacies. But we are risking a tragedy of national proportions.

And where are the protections against the abuses that you are documenting? Duncan’s words are always balanced. But how many hundreds or thousands of educational leaders are spread across this diverse country who are willing to use those words for scorched earth politics?

And as you write, this is just the beginning of the RttT battles. Numerous RttT proposals require multiple measurements for teachers seeking to keep their jobs, but single measurements are enough to end those teachers careers.

This is contrary to both Duncan’s and Obama’s statements.

If they learn the specific lessons of the abuses you are describing, and think more broadly, Duncan should see the need to provide written protections for teachers and unions. He should repudiate RttT like D.C.’s or Tennessee’s where their words and actions have been unnecessarily inflammatory.

If they do not heed your warnings, Round #2 will be worst. If they do listen to your excleent reporting, I hope you get plenty of credit.

Regardless, I hope every education reporter has bookmarked your blog.

Jason said...

This exam is all Math A (at least when I took it in 2003). I would be concerned if the scores on NECAP were being converted into a 100 point scale which makes up 20% of a student's grade like the Regents in NY. However, this exam is largely used as a "thumbs up" or "thumbs down", will not be high stakes for kids until 2012 (as far as I can tell).

The trouble in districts like Providence is WORSE than the 9% shows. The reason is because of the HUGE number of students who have a "1" on this exam. IIRC, something like a 15% earns a "2" on this exam. In Providence, nearly all of the students are not capable of earning a 15%.

Taken in the context of an "exit exam" that is meant to identify the minimum ability required for success in college or an increasingly technology/knowledge driven economy, I have to say this is pretty close to what I'd hope to get all of our kids to be able to achieve.

Is it easy? No, I don't think this test is something to blow off. Is it beyond what our expectations should be and within the realm of possibility? I would say so.

I'm actually impressed with the conceptual nature of some of these questions and I might even use this test as an example to demonstrate that multiple choice does not necessarily mean "no critical thinking".

Ellen said...

There is much discussion here about what Algebra 1 is and isn't. In RI, and other NECAP states, the content in 9th and 10th grades is dictated by the Grade Span Expectations (GSE's) which are the state standards for core content areas. So rather than comparing to what people think it may or may not be in NY, you could go to the RIDE website and find the GSE's.
ride.ri.gov
search on their site for GSE's or state curriculum. It's not an easy site to navigate.

abellia said...

What score do you have to get in order to be "proficient"?

Tom Hoffman said...

I don't know how they determine the cut scores.