Wednesday, May 09, 2012

Don't Be Snowed on the Complexity of Unwritten Common Core Assessments

I'm glad Dana Goldstein is taking on the subject of computer scoring of essays and the Common Core, but I'm afraid she has been led somewhat astray by her sources, particularly the Common Core's Appendix B.

But of the eight state standardized-test writing prompts Shermis looked at in his study, none required students to demonstrate knowledge beyond what could be gleaned from a specific text, and four required “relatively content-free” responses. The Common Core, meanwhile, has much higher ambitions for student writing. Here is an example of a Common Core essay prompt—the kind students across the country should be encountering over the next five years:

Compare and contrast the themes and argument found in the Declaration of Independence to those of other U.S. documents of historical and literary significance, such as the Olive Branch Petition.

Brown University computer scientist Eugene Charniak, an expert in artificial intelligence, says it could take another century for computer software to accurately score an essay written in response to a prompt like this one, because it is so difficult for computers to assess whether a piece of writing demonstrates real knowledge across a subject as broad as American history.

Dana cites the second half of an example question meant to demonstrate mastery of Reading Standard 9 for Informational Texts at the 11th/12th grade level:

Analyze seventeenth-, eighteenth-, and nineteenth-century foundational U.S. documents of historical and literary significance (including The Declaration of Independence, the Preamble to the Constitution, the Bill of Rights, and Lincoln’s Second Inaugural Address) for their themes, purposes, and rhetorical features.

Or, more succinctly:

Produce an analysis of the themes, purposes and rhetorical features of a text.

Since there are no standards for what is required for this sort of analysis beyond clarity and logical argument, and those are covered by a different standard, something like this would be perfectly adequate to accurately and completely satisfy the standard:

Based only on a reading of the Declaration of Independence and citing only evidence from the text, answer the following:

  1. What is the theme of the text?
  2. What is the purpose of the text?
  3. How does the listing of grievances (highlighted in yellow in the text) support the purpose of the text?

The standard does not require you to write an essay -- it is a reading standard, not a writing standard. Doesn't require compare and contrast between two documents. It doesn't require any content knowledge outside the text, although it would probably be very helpful. It does not require you to be familiar with those texts beforehand, although it would help, and it certainly doesn't require prior knowledge of something as obscure as the Olive Branch Petition. It does not permit evidence from outside the text, which is completely consistent with the approach and philosophy of the standards as a whole.

This prompt is much more computer-scorable. It is also easier, while still being more closely aligned to the text of the standard. If your state uses my prompt, the tests will be cheaper, the results returned more quickly, probably with more reliability and validity, and your scores will be higher. I don't see any reason why these couldn't be multiple choice questions, actually.

Which approach do you think will win out?

No comments: