Thursday, November 01, 2007

Utecht et al on SIS's

Jeff Utecht has a long post on the strategic importance of student information systems, and his hopes for their evolution in the Web 2.0 world. There have been a flurry of other SIS posts and stories this week, and I'll tie a few of those in here as well.

This is, of course, something I've got a lot to say about, since this has been my primary work focus for a few years and the reason I'm sitting in Vilnius right now. On the other hand, I have been pretty quiet about SchoolTool lately and have kept a fairly low profile for the project in general. Basically, a half-done student information system is useless. "Almost works" is fine for at least playing with some kinds of software, but not a SIS. Even basic testing of an SIS takes a significant investment of time, and I don't want to cause people to waste their time.

That said, we've made significant progress this year and next year will see a sequence of increasingly finished and public releases.

Anyhow... with that disclaimer let's move on to Jeff's post. Jeff writes:

First I believe that the system must have an open database that allows a school to incorporate other programs into it to allow for a seamless system. A program built in a database like mySQL would allow easy integration into the educational programs we talked about in Part 1.

First off, absolutely. A public institution should not store its data someplace where it cannot use and access it any way it sees fit. The fact that this has been accepted in the recent past is just an illustration of how immature this whole field is.

Unfortunately, there is a lot more to enterprise integration than just having an "open database." First off, "open database" could mean a few things. It is probably reasonable to say any database that implements the Open Database Connectivity protocol, or even handles standard SQL queries is an "open database." By that standard the vast majority of commercial database engines are "open."

However, you can still run into problems like Chris Dawson describes:

We were just walking away from a wildly convoluted data system that, while running on SQL server, was nearly indecipherable.

So you not only need a database that is open, but that is well designed and documented, if you expect to re-use it with other applications.

Even then, you're talking about having one application stick its grubby little fingers directly into another application's database. This is common, and it works in many cases, but it is inherently tricky, because you're bypassing all the application logic that the various applications have built around that database. If you have, say, your CMS peering directly into your SIS's tables for account and personal information, you're probably bypassing the SIS's security infrastructure. Also, your CMS may not expect records for new users to just suddenly appear in its database without having created the user itself. Also the CMS may be unpleasantly surprised that it doesn't have write access to parts of its (that is, the SIS's) database. This is not to say that any of these problems can't be hacked around by local admins and developers, but you're looking at a growing set of workarounds. This type of system is also based on the assumption that the SIS's database tables are not going to change, which is almost never a safe assumption, so you may be fixing breakage every time you upgrade the SIS, and certainly major changes if you switch SIS's.

What you really want are API's that allow the two applications to talk to each other as peer applications, respecting each application's programming and business logic, rather than plunging their dirty fingers into each other's databases without asking. You want your SIS to notify your CMS when a new user is created and for the CMS to create the corresponding user in its system. Or at least for the CMS to be able to ask the SIS itself for a list of users, rather than just peering into the database. In 2007, in particular you want a Web Service API.

API's require a bootstrap effect, however. An API that is used by only one application is much less useful than one that is used by many. Otherwise, you're still doing a lot of re-coding if you have to switch applications. If you're dependent on an API which can be changed at the whim of a single company, they can and will break or hide parts of the API if it works to their advantage to do so.

So... what Web Services are widely (at all) used in K-12 enterprise apps, in the US, at least? Ug... SIF. To say I have a love/hate relationship with SIF would be an overstatement. Grudging acceptance/hate is more accurate, but it is nearly the only game in town. Unfortunately, the incumbent players in the SIF polity have decided that dividing a larger chunk of a medium sized market is better than a unknown slice of a vastly larger pie, and they've effectively throttled SIF's growth for the sake of keeping control. It is not so much their greed that bugs me as their lack of vision.

Probably the sanest alternative is the e-Framework approach of leveraging broader industry standards, but it is still a hodgepodge of standards which, even in relatively simple cases often are implemented inconsistently, and the cases often aren't so simple.

Another approach to extendability is what Jim Hirsch seems to be talking about here. Basically using standards for extension established within one programming environment, in this case Java J2EE. This is again, better than nothing, but limited to a given language or framework.

If you've got a good XML Web Services approach, what kind(s) of client access (web, desktop app, phone, etc.) is used is less of an issue, because you should be able to create new clients and mashups relatively easily. This is the "The Machine Is Us/Ing Us" scenario (just use XSLT!) which in fact barely exists today.

So, this sounds pretty grim. What's the answer? Ideally, a little of everything, judiciously applied. What we really need is an open source SIS built on a modern extensible web framework, so that each of these strategies, and hopefully better ones coming in the future, can be pursued in a wide variety of schools, with improvements and enhancements rolled back into the main distribution. That's what SchoolTool aims to be, and part of the reason it is taking so long, and why it may ultimately be so valuable, is that its mission is not to simply be an SIS, but the foundation of the kind of system Jeff describes.


Chris Lehmann said...

And this post is exactly why you all remain my big bet.

Bill Fitzgerald said...

Hello, Tom,

I agree with Chris -- if you weren't building SchoolTool, we'd probably be looking seriously into building out more of this functionality within Drupal.

While we've had this conversation before, it bears repeating here again: SIF is a dead end. The folks behind SIF are more interested in a business strategy than a standard that supportls learning, and they have nearly a decade of non-adoption to show for their efforts. Nice work, guys.

Keep it lightweight. Decide what information *must* be moved between apps, and use the existing standards that already exist.

You are also right on wrt bypassing apis and doing direct database lookups -- that is a one way path to big problems, and a hack that should be punished by having to read through the Moodle API documentation :)



Jeff said...

Thanks Tom for taking my thoughts one step further. Obviously I don't understand the depth of these programs like you do. All that I know is that something has to change. As schools push more and more content online and become more and more connected we need to make sure all these programs work together as seamless as possible. For the most part students don't mind having 4-6 usernames and passwords, but teachers can't stand it and I'm finding it a barrier to getting teachers online. Of course you shouldn't need 4-6 usernames and passwords...we should be able to get this to work....we need to get this to work. Keep fighting the good fight and I too look forward to School Tools!

Rafael Ferreira said...

Bill, I agree with you that "SIF is a dead end" is increasingly the feeling towards SIF (specially in the open source community).

We are hoping to change that mindset and only time will tell whether we will be successful or not. For the sake of the US educational environment as a whole, I hope we are.