Saturday, June 18, 2011

I Don't Think Carleton Jones Knows the Technical Definition of "Broken" or "Screwed Up"

Carleton Jones, Chief Operating Officer of Providence Public Schools:

Dear Teachers Participating in Match:

First, please accept my sincere apologies for not communicating to all of you sooner. I understand this has been a stressful period of time with high-stakes outcomes. While I’m sure you’ve heard many stories about what is and is not happening, please allow me to fill you in with the current situation and next steps.

Current Status

First, the Match system is not “broken” and the data are not “screwed up.” However, it is true that many of you experienced not finding all of your rankings on the lists of Match positions. Principals experienced the same phenomena – rankings they expected to see did not appear. The reason: several hundreds of the rankings submitted did not pass the Employee Id and Last Four SSN validation step. Due to this discrepancy in Employee ID and Last Four SSN, they did not pass this security step and therefore these ranks were not included. Because we are committed to making the best matches possible, we took the time to closely inspect all of the ranks that were initially thrown out. In doing this we found that most of the failed validations appear to have been the result of simple human error on the part of the person submitting the rankings.

Fortunately, in order to ensure as many submitted teacher rankings as possible are considered for Match, we were able to resolve the errors rather than penalize teachers by not counting those rankings. As you can imagine, the task to correct the mis-matched IDs was a tedious and time consuming task. However, we feel this was time well-spent as the result is that more teachers will receive a job through the Match than would have if we hadn’t restored the rankings that were originally deemed invalid. That task is complete and we do not need to ask most of you for additional rankings or any of you to re-validate any of your rankings. Unfortunately, due to the time-intensive nature of carefully validating the previously invalid data, that process required us to extend the notification date from yesterday, June 15th until Monday or Tuesday of next week. (emphasis in original)

I've written mission-critical web applications used by teachers at the school level in Providence, and managed the development of data systems used in schools around the world, including in the developing world, where only the most minimal level of technical experience can be assumed.

If you've deployed an application where "several hundred" out of about 400 teachers cannot validly complete a seemingly straightforward task, you've got a problem with your application, not with your users. By definition, if your users cannot use your application, the problem is you, not them.

The specific problem with the MATCH application seems to stem from not being connected to the rest of the PPSD's data systems, at all. So teachers could not log into the MATCH system. To authenticate themselves the had to select, from drop downs, their employee ID and the last four digits of their social security number. It doesn't appear that the MATCH system confirmed that the person was authenticated at the time they entered this data. If the system couldn't do it, that's bad, but if could have cross-checked the ID and SSN at the time of entry, it is even worse, since then a simple confirmation step (like every web app you've ever logged into) prior to entering one's MATCH choices could have prevented the entire problem.

It is Carleton Jones, and more directly PPSD Broad Resident Spencer Dickinson's, job to recognize that this system was inadequate, and worse than simply using paper. At this point, teachers have no way to confirm that their selections were entered at all, or correctly. They received no confirmation by email or hardcopy. The only thing we know about the system at this point is that its public facing features were incompetently coded and negligently deployed, so there is no reason to trust the rest of the process or the holy algorithm.

Basically, what needs to happen at this point is that the mostly elementary school teachers affected by this process have to lose it. A sit-in in human resources on Monday; picketing; at least badgering the school board and Mayor over this circus. Everyone (and/or the union) should at least get hardcopy confirmation of their choices -- and there are anecdotal reports of mis-ranked schools -- prior to the final selection process.

And Mr. Jones and Mr. Dickinson need to stop acting like such condescending pricks when they're clearly the ones incapable of doing their jobs.

No comments: