Dreaming In Code -- The Myth of the Magic Closet

Scott Rosenberg's new book Dreaming In Code is a great read for anyone involved in software design over the last decade or two. I'm only on Chapter 4, but it's shaping up well. The book is about a failed project, Chandler, to build a "cross-platform, open-source, PIM in the spirit of Lotus Agenda". The words "in the spirit of" are the big red flag, because it revealed a massive hole in the product spec. As Joel's review of Dreaming in Code says, "Whenever the spec describes the product in terms of adjectives (“it will be extremely cool”) rather than specifics (“it will have brushed-aluminum title bars and all the icons will be reflected a little bit, as if placed on a grand piano”) you know you’re in trouble"

Chandler's failure is important because it was being built by top talent; famous names from Apple, Netscape, and other illustrious software projects. That a software project failed is hardly new; but that _these_ guys failed reveals a lot about the state of software development. As Rosenberg says, bridges are built pretty much on time and on budget. Why can't software, after fifty years, be the same?

In defence of the Chandler team, they were tackling a known holy grail. I call it The Magic Closet. Imagine people with messy houses being able to toss all their clutter into a closet and the closet would magically sort and arrange it. Later you could reach in for your swim mask, your high school yearbook, or that little hex screwdriver for your glasses and it would be right there. Chandler applied this idea people's data. Computers have massive storage and fabulous ability to processs information. Surely we can build something where you dump all your contacts, e-mails, photos, pictures and assorted notes. It would magically arrange and cross-reference the data into useful information.

Magic Closets have been tried before. Most recently, Windows WinFS (the Cairo project) was an attempt to replace the file system with a database and gobs of tags and meta-tags. Failed. Lets look at some reasons why Magic Closet projects fail. Of course hindsight is 20-20 and it's easy to criticize. But a repeated failure is not just a fluke. It must reveal something important about our conception of software.

Reasons for failure:
* lack of a data model
* if you need an AI, say so
* too many smart people in a room
* leader is not a programmer

Lack of a Data Model.

Most software can be described in Model-View-Controller terms. The Model is a media-independent representation of the data that the app is about. For a word processor it's text. For a video editor it's audio and video clips. For a PIM it's people, contacts, appointments and messages. Usually software is written by writing the model first, then adding views and controllers. Not only is it important to define the model early (because changing it later is painful), but a working model allows development of views and controllers to proceed in parallel. Doing things in a different order is wierd.

Chandler wanted to be revolutionary. A simple PIM would have a Person record with a PhoneNumber field. Answering the question "What's Joe's phone number?" is easy. But a single field is restrictive. People have home phones, mobile phones, fax lines, and work numbers. People have multiple homes (summer and winter). Offices have multiple phone numbers (switchboard and private direct access). A fancier PIM would support multiple phone numbers per person, but using a fixed schema. Basically, each person would have a primary phone number and a bunch of extra phone numbers. So for many purposes the software can still act as if Joe has a single phone number. But Chandler didn't like fixed schema; they wanted an open-ended system of tuples. Not only could a person have unlimited phone numbers, but there wouldn't even be a primary phone number. There might be rules to determine (based on, say, time-of-day) what the primary number is. Or consider your doctor (Joe) who ends up being also the father of your daughter's boyfriend. Now he has two roles. You wouldn't phone his home number to renew a prescription; and you wouldn't phone his clinic to invite him to a BBQ. Chandler wanted to capture all these "use cases". Except now answering the question "What is Joe's phone number?" is hard. It depends. A skeptic might suggest that the user could simply specify "phone Joe at home". But that defeats the spirit of Lotus Agenda that Chandler wanted to have. The software should know what's going on and (magically) call Joe's cottage when that is the appropriate choice.

Perhaps a data model for this sort of thing is possible. But it's wildly complicated. And users won't use something they can't understand.

If you need an AI, say so

Chandler was supposed to parse useful information out of e-mails and other documents. This requires artificial intelligence. If that's what they really wanted, the Chandler team should have hired dozens of PHDs for a ten-year project to (first) research and then develop such an AI. Otherwise the software is just guessing. And as anyone who's used speech recognition software can tell you, being right 90% of the time is not good enough. Having to review and correct your input is just too painful.

Too Many Smart People In a Room

TMSPIAR. An excess of talent is a big problem. Too many cooks... as they say. I've been in meetings like that and they are highly problematic. Talented people want big scope and big challenges. People at these meetings are either cowed by the high-wattage talent, or get into pissing matches. When a project is open-ended (a code word for vague), then just about any design choice can be justified as a requirement.

Surely all software projects can have a single architect. There might be a few exceptions to this, but a PIM is hardly an F-16. A single mind can always produce a better and more consistent design than a committee.

Leader is Not a Programmer

I don't know if this is true. Mitch Kapor wrote Lotus 1-2-3. But he has a PHD in Psychology. I'm guessing he is non-technical in a way that only a technical person can understand. Non-technical leaders can challenge people in good ways, but it's a risk factor for software projects.

That's it for now; I'll review the rest of the book later.