Tuesday, March 03, 2009

Brookings Goes a Millimeter Deep

As Checker Finn said last week, after you go "a millimeter below the surface" on national standards boosterism, you confront the grim reality that people don't agree on what kids should know and be able to do. One nice example/preview comes from this bit from Brookings about what they perceive as bias in the Programme for International Student Assessment (PISA). Specifically (p. 15):

PISA asks students if they agree or disagree with the following statements:

  • Industries should be required to prove that they safely dispose of dangerous waste materials.
  • I am in favor of having laws that protect the habitats of endangered species.
  • It is important to carry out regular checks on the emissions from cars as a condition of their use.
  • To reduce waste, the use of plastic packaging should be kept to a minimum.
  • Electricity should be produced from renewable sources as much as possible, even if this increases the cost.
  • It disturbs me when energy is wasted through the unnecessary use of electrical appliances.
  • I am in favor of having laws that regulate factory emissions even if this would increase the price of products.

The four-point scale consists of strongly agree, agree, disagree, and strongly disagree. PISA assumes through its scoring rubric that students agreeing or strongly agreeing with these seven policies possess “a sense of responsibility for sustainable development.” Students who do not agree or strongly disagree with these policies are considered lacking such a sense of responsibility.

One of the policy statements is innocuous—being disturbed by the waste of energy—but responses to the others are reflective of political judgment. There are arguments on both sides of public policies and responsible citizens can, armed with facts and reason, come down on one side or the other...

Consider the policy statement, “Industries should be required to prove that they safely dispose of dangerous waste materials.” What does it mean to be required to prove? What kind of proof? Is the standard of proof reasonable? Are penalties involved if industries do not meet such a standard? What is safe disposal? How is safe disposal defined? How are dangerous waste materials defined?

OK, fair enough, but by this standard, you couldn't ask any questions about policy. In the abstract, none of these statements is actually controversial. There is no scientific argument that endangered species should not protected. Nor, for that matter, does anyone seriously make the argument that despeciation is not a problem. They do argue that, "Perhaps this particular little fish is not as important as the dam I'm trying to build," or that "Polar bears are doing quite well already, thank you very much."

But anyway, lets assume that we don't want to put any of these statements on a standardized test, and we also won't put "Property rights should be protected by the force of law," or "Thou shalt not steal." on the test either. Fine. But where does that leave us in an environment where people are increasingly hesitant to adopt or create academic standards that cannot be objectively tested? Are we going to write national standards that exclude all controversy, all moral and ethical education? In what way is that progress? How does it prepare our students to make informed policy decisions?


Bill Kerr said...

I think the Brooking study is valuable, thanks for the link

It's a case of one religion (standardised tests) being used to promote another religion (sustainability)

Tom Hoffman said...

I saw that comment coming from half-way around the world.