And then there were the more serious transgressions, the ones that had us worried about the actual integrity of the test grades. If, for example, you were about to give a student a 2 on one section of the test and happened to notice that another grader had given the same student a 4 on another section (as 4 is the highest grade you can receive, this seems like a substantial discrepancy), and you voiced your concern, you were told to (exact words) "MYOB." (Listen, lady, I'm not being a nosy parker here, this is my first time grading a very important state exam and I just want to make sure everything is copacetic.) If you and all the other graders at your table happened to notice that the essay appeared to be written in two very different handwritings, as if it sure looked like the teacher had made a few changes, and you voiced your concerns, your objections were dismissed. I don't know what teachers at other grading sites experienced, but I have to say that I was treated with less respect than I typically try to treat my second graders, and that had me worried for the validity of the scoring.
If you do it right, having teachers score some state tests can be excellent professional development. I learned a ton scoring the old RI writing assessment through a process which was very well established and executed. Sounds like New York is flailing a bit.
What you have to remember is that well designed, administered and scored tests are the foundation of the entire edifice of data-driven reform. If this isn't done right, all your "scientifically-based" research is invalidated, you're hiring and firing and giving bonuses to the wrong people, opening and closing the wrong schools, giving cash bonuses to the wrong kids, etc., etc.
And remember, we haven't just started giving kids exams. We've been doing it a long time. If we aren't doing it right now, it isn't because we just haven't had a chance to figure out the processes.
Garbage in, garbage out.