Monday, October 19, 2009

Test What Was Taught, Not What Students Should Be Able to Do!

Brookings Institution scholar Tom Loveless, a member of the U.S. advisory board to PISA, offered this sample question for 15-year-olds from the mathematics literacy section of the exam:

For a rock concert a rectangular field of size 100m by 50m was reserved for the audience. The concert was completely sold out and the field was full with all the fans standing. Which one of the following is likely to be the best estimate of the total number of people attending the concert?

A. 2000

B. 5000

C. 20000

D. 50000

E. 100000

I think this is a bad question and not just because I got it wrong. I said 5,000. The answer is 20,000. I don't see why deciding four people, not one, would fit better in a square meter is a sign of math literacy.

This might fly for the lede in a column attacking the validity of standardized testing in general, e.g., "See how this discriminates against students from small remote towns without television reception, the Amish, and agoraphobics," but Mathews and Loveless's argument that this is an example of the specific flaws in PIAS is ridiculous. Understanding how big a square meter is is important and non-trivial. It is sufficiently non-trivial that Mathews doesn't even know he doesn't understand it.

If Loveless wants to argue it is an ill-chosen question because "I think it would throw kids off," then can we apply that criterion to every distractor in every other multiple choice test ever written?

Anyhow, if anyone can find a picture of a sold out outdoor rock concert with people spaced standing one per square meter, I'd like to see it.

They continue:

A key failing of PISA, Loveless said, is that "it does not measure what kids have learned in school." Why not? Because PISA exams are written by the losing side in a century-old debate over how to teach math. For convenience, call the pro-PISA people progressives and the anti-PISA people traditionalists. Loveless is a traditionalist but appreciates the arguments on both sides. The progressives want to make math instruction more relevant to the real world and emphasize mathematical reasoning more than calculation.

"The losing side?" According to whom? But beyond that, the argument against "progressive" math isn't that it is differently successful, but that it is a failure. Why shouldn't "traditionally" taught students know how big a square meter is?

And this is just a non-sequitur:

"(Traditionalists) dislike their approach being dismissed as "shopkeeper math," Loveless said, "like it was old-fashioned to try to compute anything."

First off, according to Google, progressives don't actually throw around the term "shopkeeper math" very much, and even then use it descriptively, not dismissively. More importantly, nobody believes teaching "shopkeeper math" to fifteen year olds is sufficient. We teach algebra today people! College or bust! What planet did Loveless beam down from?

Finally, they pull out this hobby horse:

Conservative politicians who decry how bad the test makes us look should be careful, as the attitudinal parts of PISA lean left. On PISA's student questionnaire, those who support statements such as "I am in favor of having laws that regulate factory emissions even if this would increase the price of products" are deemed to be environmentally responsible. Students who disagree are not.

OK, what is the argument that no regulation of factory emissions is environmentally responsible? It seems to me that there is a clear global consensus based on science and history that some regulation of pollution is responsible and necessary. If you read that and think "I am not in favor of more regulation." Or "I am in favor of less regulation," and then you chose "no," you didn't read the question carefully enough. You only answer "no" if you are against all regulation of factory emission, a position which is manifestly environmentally irresponsible. If someone wants to make the argument that eliminating all regulation of air, water, and other effluent pollution from factories is an environmentally responsible position, I'd be happy to hear it.

Now, you can argue that the plain reading of that question is not obvious to you. Fine. Can we apply the same jaundiced lens the the standardized tests you prefer?

All I know is this, someday my daughters will be standing in a field, watching a sold out rock concert, standing at a much smaller density than one person per square meter, and at that time I hope they are not standing on a toxic waste dump, breathing lead and mercury laden particles in a 110 degree summer day in New England.

Claus von Zastrow said...

Good point re: the question on regulation of emissions. I hadn't thought of it that way.

Regarding the rock concert example.... I would have answered it incorrectly when I was 15, because i had never been to that kind of rock concert. (Pathetic, I know....) I had been to outdoor concerts where people spread out on picnic blankets--fewer that one person per meter.

Still, is this a criticism of PISA in particular or a reminder of how big an impact CONTEXT and assumptions of prior knowledge can have on students' performance.

Graham Wegner said...

I liked this selection from the article:
"Loveless prefers the other major international test, Trends in International Mathematics and Science Study, which aligns more closely with the way U.S. students are taught. He is the U.S. representative to the General Assembly of the International Association for the Evaluation of Educational Assessment, a 60-nation body that oversees TIMSS."
So, you can only trust a test that shows your nation in a good light? Intersetingly, no mention of Finland either ... but maybe that's because they do well on the "other" test.

Anonymous said...

Interesting point about people who have not been in a large crowd.

But I do not believe that there is a progressive/traditional divide here. It's a reasonable applied question, no matter which side we're looking from.

I don't know about the whole exam.

Jonathan