This morning a friend made a comment about how frustrated he was with the way his company handles design documents and changes. He said he started the day being told “we forgot to include this feature. It’s on the 3rd page of the 4th brief.” The frustration here is that there are at least four briefs, and it appears that they are continual amendments to a core document.

This is a totally unacceptable way to work. The idea of a “living document” is an easy one to understand but I’ve never, ever seen anyone get them right. When the process fails, it seems to die not with a bang but with a whimper as individuals slowly stop making edits to some insanely complicated Microsoft Word file and it withers on the vine. Oh well, at least it wasn’t loud.

Only thing: there will be a bang. The bang will be the sound of engineering departments getting thrown under a bus because features get missed and clients get pissed.

So, why do we have living documents? Because a bunch of people like using words like “agile development” or “aggressive schedule” or “plug-n-play featureset”. So what you’re working on, as a team (or even as a single engineer) often changes not only from week to week but day to day. That, combined with the fact that there are usually 3 or more people who will be authors of a document, shows a clear need.

But why do they fail? What causes the wax to melt on the wings?

Well. Lots of things, but they can be broken down into three broad categories: cultural, workflow, and technical. We’ll bullet point, because that’s organization!

Choice of Document Format.

This is a big one, because it can cause all sorts of problems. The most popular choice for living documents is going to be Microsoft Word, hands down – though I’ve been places that use Excel for everything. Plain text is also popular, as is HTML (I personally will use HTML because I can do formatting in it and I loathe Word).

The first problem we have here is a cultural one.

Program managers, project managers, and suits like Word. They know it, they’re familiar with it. The machines they work with are likely to be laptops running a kind of Windows (they spend a lot of time in meetings and have to be mobile). Since they use Word constantly for other purposes, it’s a natural choice for them.

Engineers, however, don’t think that way. I know a lot of engineers, and only a scant handful know how to use Word and are frankly baffled by it’s startlingly bad usability and overwhelming featureset. When I first started playing with Word for a game I wrote, I made a total mess of the file trying to use formatting rules and display modes and so on and so forth. I’m not a stupid guy, but the word processing program got in the way of me processing words. Word documents are binary files. They can’t be grepped or diffed. They can’t be parsed in perl, they can’t be smartly integrated into version control.

Further, many engineers will be working from machines that are not Windows. They’ll have Linux boxes or (rarely now) Solaris. And they don’t have Microsoft Word – only a shitty “Open Office” version (and if you’ve ever shared a Word doc between a Windows box and Open Office, you’ll know how badly it goofs up formatting).

Clearly the choice of Word is a poor choice for engineers.

Excel spreadsheets have nearly the exact same problems as Word, so we’ll skip that (aside from sorting abilities, they don’t bring a lot to the table).

Plain text and HTML versions of documents bring a similar set of problems to the mix, only from the opposite side. Engineers love plain text, but it suffers in that it is unable to support nifty things like auto-generated tables of context, text formatting, or, well, anything except words. HTML solves a lot of these problems, but you fall into the trap where by non-engineers won’t touch it: it’s difficult to edit if you don’t know what you’re doing, too easy to screw up even if you do, and the document becomes unweildy after a certain size.

There is a sort of half-way solution here, though, and that is to use Google Docs. But I’ll get to that in more detail in a bit.

Version Control

Issues with document version control are the second biggest. This is also a workflow problem. Here is a common scenario:

The PM writes a Word document, Tech Spec v. 1.0.doc. Sends it out for review. The archictect makes some changes and mails out Tech Spec v. 1.1.doc. At the same time, the interaction designer makes some changes and sends out a document also titled Tech Spec v. 1.1.doc, which, of course, does not have the architect’s changes in it. Bam! Version drift.

So the solution then is that the PM will painstakingly try to figure out what the changes are (he can’t run ‘diff’ on the files, mind you). Then he sends out Tech Spec v. 1.1-real.doc. Over the next week, we’ll get 12 versions of the document, an none of them will say what has been changed.

Since reading a 20 page technical specification 4 times a day is a waste of my time, I’ll go on the most recent one I have open, while the latest one may have changed requirements on work I have already completed and thus don’t re-read.

Part of this is workflow. It can be offset by having a single point of contact for the document author (which slows things down), and a summary page of “version changes”. But that’s not a great solution.

Further, we’re passing information around via EMAIL. Ugh. This is the fastest way to get me to ignore a document: I get hundreds and hundreds of mails a day and everything gets lost in a morass of garbage.

Solution! Post the documents to Sharepoint or something similar. Okay, great! But, you know, again, if I’m going to get notifications that it has changed, I’m still getting spammed with mail that I will ignore. It’s nice that there’s a single central place for the most recent document, but we’re also dealing with Sharepoint or some similar product, and they all suck, by and large.

Okay, so let’s check it into a revision control system (CVS, Perforce, whatever). Now everyone has to know how to use an RCS (hah!). Plus, documents should then be in text formats for best results (so that versions can be merged and we can have multiple concurrent editors). We’re back to cultural problems here.

My Solution

Wikis. Plain and simple. Wiki software (like MediaWiki, which Wikipedia runs on) brings the following to the table out of the box:

* Free
* 2 Hour set up time (less if you know what you’re doing)
* Easy to use/learn text formatting system that supports all manner of object embedding (diagrams, pictures, etc.)
* Multiple, concurrent authors can work on the same document – or even shards of a document – at the same time.
* Provides instant access to what exactly changed in the document (this is my favorite thing, ever).
* Revision control
* Access control (read/edit)
* Document Search and Indexing
* Auto-generated goodies (Tables of Contents, Categories, etc.)
* End-User machine agnostic (runs in a browser)

Now, Google docs brings a lot of that to the table but it has what I consider to be two glaring flaws:

* It is externally hosted. Good luck convincing the security officers of a large company to host your top-secret project design with a third party. You can’t lock it down behind the firewall, safe as houses in your intranet.
* It requires that you make a Google account. Again, this is a security thing but it’s also a pain in the ass thing. I like to keep my work and personal stuff separate. I have a Google account but I don’t use it (and don’t really want to). A company I used to work for did all their stuff on Google docs. It was a pain in the ass for me to have to constantly remember this weird password that I put in there just to update my estimated hours on a stinkin’ spreadsheet.