The problem I was trying to solve was this: I had about 2,000 albums worth of music, and I wanted something that would stream music to my internal stereo. I wanted it to be able to perform the following functions:

1) Allow me (or others) to add single songs or whole albums to the music queue;
2) Track which songs, albums, or artists were the most popular;
3) Have a “smart” Artificial Intelligence disk jockey that could spin music on its own.

The program I wrote succeeded in this regard in all aspects. Sure, sure, the code itself was ugly as shit (and written in Perl). Were I to re-write it today, I would do a ton of things differently. But I had about 8 years less experience writing stuff, so I forgive myself.

At any rate, Seraphim itself became an example of emergent behavior on several levels. Not just in the AI of the DJ, but also in the way that people used the system.

This was in the early days of internet music. Before the dark times. Before the MPAA. I made this radio station and a way to interact with it, and then we plugged it into AOL’s network (I won’t discuss how that happened because there may be legal ramifications for the people involved). Suffice to say, suddenly a computer in my house was streaming music to hundreds (possibly thousands) of people.

If you had an account on the system, you could do any of the following:

1) Upload your own library of mp3s so that they were available for play
2) Modify the metadata for any song, artist, or album (genres, etc.)
3) Add songs to the system’s “queue” – the music it was playing.

(If you were a super-user, you could kill songs from the queue or mark them as “never play”).

People would listen to the station rather than their own personal libraries because there was a significant degree of fun involved in being a disk jockey. Perhaps my most favorite emergent user behavior was when someone would start a musical “theme” and the various DJs would try to one-up each other following said theme.

For example, someone might say “the theme is fire“. Then, we’d see a bunch of “fire” related songs show up (“Burning Down the House”, etc.). There was a game made of the music. It was a glorious amount of fun.

Each song, artist, and album had a “karma” score. The more often it was requested, the higher the karma. Picking a single song gave a +1 to each song, artist, and album. That way the system understood popularity (though the scales were different for each [song, artist, album]).

However, the most interesting part of the system (to me) was that if no one put anything into the queue, Seraphim would “auto dj”. And, having lived with it for a year or so, it became . . . exceptionally creepy in how smart of a disk jockey it became.

I wrote the artificial intelligence routines as a lark, to be honest. But this is an example of awesome emergent behavior.

The first thing I did in the system was to “fix” a weakness in the MP3 file format. MP3s have a “genre” tag but that’s very limited. It doesn’t say a lot; it’s a single dimension. So I wrote a large matrix called “Genre Brethren”.

For example, “Rap” is a genre brother to “Gangster Rap” and to “Hip-Hop.” “Speed Metal” is brother to “Death Metal” and “Heavy Metal”. (The system was far more complex, usually seeing 3-6 brethren). Albums, artists, and songs could be tagged with multiple genres.

When in “Auto DJ” mode, Seraphim would start with the most current song and then make choices. Did it stay in the current genre? This was maybe 50/50. If it decided to change genres, it would only move to one of the brethren genres (thus, we don’t move from Slayer to Michael Bolton). We keep a continuity of musical style.

Once it picked a genre, it had to choose a song. But that’s a trick, right? Obviously, we don’t want to pick songs that suck. And that’s where I wrote this thing that worked and worked well. To this day, though, I’m not sure how I arrived at the system.

Cheaply, you can just choose the song in the genre with the most karma. That works once. Ideally, though, you’ll spread out. So I wrote this complicated system whereby it would pick songs. If I recall correctly (and I could pull up the source to see, but fuck that), it went like this:

1) Choose between Song, Artist, or Album in genre.
2) Within that subset, take the top 50 karma values as a grouping.
3) Within that grouping, weigh each one. Those within the top 5 get +5 within 6-10 get +4, within 11-20 get +3, within 21-30 get +2, everyone else +1.
4) Select within that set based on weight.
5) If “songs”, play that song. Done.
6) If “albums”, repeat step 3 based on songs in album. Pick song; play; done.
7) If “artists”, repeat step 3 based on albums, then go to step 6.

I injected a degree of “fuzziness” into the AI routine, too. Without the fuzziness, it might play the same shit over and over again (like, all of Nevermind on repeat). While the ideal was the highest karma value in a given set, there was logic to ignore that aspect and just pull from lower in the stack (there was a routine to drop out of “standard top 50 of type” mode and pull from wherever, or to overweigh to the bottom of the stack).

As it played songs, it marked when things were last played. Thus, no repeats within 6 hours or so. It also did crazy-ass shit like “look for songs that have a positive karma value that haven’t been played in 5 days” and then give those songs extra weight.

What happened was this: I ended up with a disturbing, creepily good disk jockey. My ex-wife and I had multiple conversations about this. We’d be listening to it all day and there would be strange stretches of excellent music choices. So we’d go look and see who had been programming it, and it nearly always turned out to be the machine itself.

I write this only because I’m thinking about writing artificial intelligence routines and Seraphim was one of my first attempts at doing “smart” AI.

This is a totally unacceptable way to work. The idea of a “living document” is an easy one to understand but I’ve never, ever seen anyone get them right. When the process fails, it seems to die not with a bang but with a whimper as individuals slowly stop making edits to some insanely complicated Microsoft Word file and it withers on the vine. Oh well, at least it wasn’t loud.

Only thing: there will be a bang. The bang will be the sound of engineering departments getting thrown under a bus because features get missed and clients get pissed.

So, why do we have living documents? Because a bunch of people like using words like “agile development” or “aggressive schedule” or “plug-n-play featureset”. So what you’re working on, as a team (or even as a single engineer) often changes not only from week to week but day to day. That, combined with the fact that there are usually 3 or more people who will be authors of a document, shows a clear need.

But why do they fail? What causes the wax to melt on the wings?

Well. Lots of things, but they can be broken down into three broad categories: cultural, workflow, and technical. We’ll bullet point, because that’s organization!

Choice of Document Format.

This is a big one, because it can cause all sorts of problems. The most popular choice for living documents is going to be Microsoft Word, hands down – though I’ve been places that use Excel for everything. Plain text is also popular, as is HTML (I personally will use HTML because I can do formatting in it and I loathe Word).

The first problem we have here is a cultural one.

Program managers, project managers, and suits like Word. They know it, they’re familiar with it. The machines they work with are likely to be laptops running a kind of Windows (they spend a lot of time in meetings and have to be mobile). Since they use Word constantly for other purposes, it’s a natural choice for them.

Engineers, however, don’t think that way. I know a lot of engineers, and only a scant handful know how to use Word and are frankly baffled by it’s startlingly bad usability and overwhelming featureset. When I first started playing with Word for a game I wrote, I made a total mess of the file trying to use formatting rules and display modes and so on and so forth. I’m not a stupid guy, but the word processing program got in the way of me processing words. Word documents are binary files. They can’t be grepped or diffed. They can’t be parsed in perl, they can’t be smartly integrated into version control.

Further, many engineers will be working from machines that are not Windows. They’ll have Linux boxes or (rarely now) Solaris. And they don’t have Microsoft Word – only a shitty “Open Office” version (and if you’ve ever shared a Word doc between a Windows box and Open Office, you’ll know how badly it goofs up formatting).

Clearly the choice of Word is a poor choice for engineers.

Excel spreadsheets have nearly the exact same problems as Word, so we’ll skip that (aside from sorting abilities, they don’t bring a lot to the table).

Plain text and HTML versions of documents bring a similar set of problems to the mix, only from the opposite side. Engineers love plain text, but it suffers in that it is unable to support nifty things like auto-generated tables of context, text formatting, or, well, anything except words. HTML solves a lot of these problems, but you fall into the trap where by non-engineers won’t touch it: it’s difficult to edit if you don’t know what you’re doing, too easy to screw up even if you do, and the document becomes unweildy after a certain size.

There is a sort of half-way solution here, though, and that is to use Google Docs. But I’ll get to that in more detail in a bit.

Version Control

Issues with document version control are the second biggest. This is also a workflow problem. Here is a common scenario:

The PM writes a Word document, Tech Spec v. 1.0.doc. Sends it out for review. The archictect makes some changes and mails out Tech Spec v. 1.1.doc. At the same time, the interaction designer makes some changes and sends out a document also titled Tech Spec v. 1.1.doc, which, of course, does not have the architect’s changes in it. Bam! Version drift.

So the solution then is that the PM will painstakingly try to figure out what the changes are (he can’t run ‘diff’ on the files, mind you). Then he sends out Tech Spec v. 1.1-real.doc. Over the next week, we’ll get 12 versions of the document, an none of them will say what has been changed.

Since reading a 20 page technical specification 4 times a day is a waste of my time, I’ll go on the most recent one I have open, while the latest one may have changed requirements on work I have already completed and thus don’t re-read.

Part of this is workflow. It can be offset by having a single point of contact for the document author (which slows things down), and a summary page of “version changes”. But that’s not a great solution.

Further, we’re passing information around via EMAIL. Ugh. This is the fastest way to get me to ignore a document: I get hundreds and hundreds of mails a day and everything gets lost in a morass of garbage.

Solution! Post the documents to Sharepoint or something similar. Okay, great! But, you know, again, if I’m going to get notifications that it has changed, I’m still getting spammed with mail that I will ignore. It’s nice that there’s a single central place for the most recent document, but we’re also dealing with Sharepoint or some similar product, and they all suck, by and large.

Okay, so let’s check it into a revision control system (CVS, Perforce, whatever). Now everyone has to know how to use an RCS (hah!). Plus, documents should then be in text formats for best results (so that versions can be merged and we can have multiple concurrent editors). We’re back to cultural problems here.

My Solution

Wikis. Plain and simple. Wiki software (like MediaWiki, which Wikipedia runs on) brings the following to the table out of the box:

* Free
* 2 Hour set up time (less if you know what you’re doing)
* Easy to use/learn text formatting system that supports all manner of object embedding (diagrams, pictures, etc.)
* Multiple, concurrent authors can work on the same document – or even shards of a document – at the same time.
* Provides instant access to what exactly changed in the document (this is my favorite thing, ever).
* Revision control
* Access control (read/edit)
* Document Search and Indexing
* Auto-generated goodies (Tables of Contents, Categories, etc.)
* End-User machine agnostic (runs in a browser)

Now, Google docs brings a lot of that to the table but it has what I consider to be two glaring flaws:

* It is externally hosted. Good luck convincing the security officers of a large company to host your top-secret project design with a third party. You can’t lock it down behind the firewall, safe as houses in your intranet.
* It requires that you make a Google account. Again, this is a security thing but it’s also a pain in the ass thing. I like to keep my work and personal stuff separate. I have a Google account but I don’t use it (and don’t really want to). A company I used to work for did all their stuff on Google docs. It was a pain in the ass for me to have to constantly remember this weird password that I put in there just to update my estimated hours on a stinkin’ spreadsheet.