Brookings Institution scholar Tom Loveless, a member of the U.S. advisory board to PISA, offered this sample question for 15-year-olds from the mathematics literacy section of the exam:
For a rock concert a rectangular field of size 100m by 50m was reserved for the audience. The concert was completely sold out and the field was full with all the fans standing. Which one of the following is likely to be the best estimate of the total number of people attending the concert?
I think this is a bad question and not just because I got it wrong. I said 5,000. The answer is 20,000. I don't see why deciding four people, not one, would fit better in a square meter is a sign of math literacy.
This might fly for the lede in a column attacking the validity of standardized testing in general, e.g., "See how this discriminates against students from small remote towns without television reception, the Amish, and agoraphobics," but Mathews and Loveless's argument that this is an example of the specific flaws in PIAS is ridiculous. Understanding how big a square meter is is important and non-trivial. It is sufficiently non-trivial that Mathews doesn't even know he doesn't understand it.
If Loveless wants to argue it is an ill-chosen question because "I think it would throw kids off," then can we apply that criterion to every distractor in every other multiple choice test ever written?
Anyhow, if anyone can find a picture of a sold out outdoor rock concert with people spaced standing one per square meter, I'd like to see it.
A key failing of PISA, Loveless said, is that "it does not measure what kids have learned in school." Why not? Because PISA exams are written by the losing side in a century-old debate over how to teach math. For convenience, call the pro-PISA people progressives and the anti-PISA people traditionalists. Loveless is a traditionalist but appreciates the arguments on both sides. The progressives want to make math instruction more relevant to the real world and emphasize mathematical reasoning more than calculation.
"The losing side?" According to whom? But beyond that, the argument against "progressive" math isn't that it is differently successful, but that it is a failure. Why shouldn't "traditionally" taught students know how big a square meter is?
And this is just a non-sequitur:
"(Traditionalists) dislike their approach being dismissed as "shopkeeper math," Loveless said, "like it was old-fashioned to try to compute anything."
First off, according to Google, progressives don't actually throw around the term "shopkeeper math" very much, and even then use it descriptively, not dismissively. More importantly, nobody believes teaching "shopkeeper math" to fifteen year olds is sufficient. We teach algebra today people! College or bust! What planet did Loveless beam down from?
Finally, they pull out this hobby horse:
Conservative politicians who decry how bad the test makes us look should be careful, as the attitudinal parts of PISA lean left. On PISA's student questionnaire, those who support statements such as "I am in favor of having laws that regulate factory emissions even if this would increase the price of products" are deemed to be environmentally responsible. Students who disagree are not.
OK, what is the argument that no regulation of factory emissions is environmentally responsible? It seems to me that there is a clear global consensus based on science and history that some regulation of pollution is responsible and necessary. If you read that and think "I am not in favor of more regulation." Or "I am in favor of less regulation," and then you chose "no," you didn't read the question carefully enough. You only answer "no" if you are against all regulation of factory emission, a position which is manifestly environmentally irresponsible. If someone wants to make the argument that eliminating all regulation of air, water, and other effluent pollution from factories is an environmentally responsible position, I'd be happy to hear it.
Now, you can argue that the plain reading of that question is not obvious to you. Fine. Can we apply the same jaundiced lens the the standardized tests you prefer?
All I know is this, someday my daughters will be standing in a field, watching a sold out rock concert, standing at a much smaller density than one person per square meter, and at that time I hope they are not standing on a toxic waste dump, breathing lead and mercury laden particles in a 110 degree summer day in New England.