They wanted something powerful, flexible and capable of meeting everyone's need. And, while they were at it, they wanted something you could write a few academic papers about. And (they probably didn't admit this even to themselves) they wanted something that kept them in the driver's seat. They wanted an ontology; a classification scheme that could apply to everything on the Internet. A Dewey Decimal System for the web if you will.
Their original idea was to come up with some top-level generic categories and below them would be more specific sub-categories with further sub-categories below that, so that a web page about MG Midgets might be categorized as 'technology / transportation / automobiles / sports cars / British'.
Now the idea of creating a hierarchical structure for the Internet is nothing new. Back in the dark ages there was something called Gopherspace; basically a menu driven directory of text files. Later we had web directories, such as Yahoo! and the Open Directory Project. But the web ontology guys quickly ran into some snags common to all such efforts. The first issue relates to the classifications themselves -- they are open to interpretation and disagreement. Take a look at my example above; Should 'British' come before 'automobiles'? You could certainly argue it either way.
The only way to solve this problem is to have some ontological master creating the categories and saying "Thou shall use these classifications, and no other!" Which creates the second issue: How do you get millions of people to, independently, use the proper categories? Leaving aside intentional mis-categorization (classification spam), you will still get an awful lot of accidents and misunderstandings. And, as all computer programmers know, this leads to a situation known as GIGO. (Garbage in? Garbage out.)
So hierarchical categories won't work. "What could we use in their place?" the ontologists asked. And, being as these guys are mainly academics, mathematicians and computer scientists, they came up with a completely unusable answer: The Resource Description Framework (RDF) and its refinement, the Web Ontology Language (OWL).
Now I'm being a little unfair to the ontologists here, partly because OWL and RDF actually do work. They are entirely rational ways of creating a linkage of meaning between web pages or, for that matter, between any set of things in the world. All you have to do is learn the OWL mapping structures, which includes things like 'transitive' and 'symmetric' properties, and enough of the numerical theory behind network graphs to apply it to RDF.
A few years ago I took a shot at explaining the underlying problem with this approach in my essay Why RDF is hard. For now let's leave it and just accept that the ontologists managed to create a beautiful framework that no-one in the real world is ever going to use.
So, should we just give up on bringing any structure to the rich tangled mass of the web? Well, no. If you read my essay above you will already know that one of the things which makes RDF so powerful is the fact it has emergent properties. The reason RDF is difficult is that you have to get your head around the thing emerging from it before you can use it properly. Creating a chicken and egg paradox.
But what if there was something so simple to use there was no head-stretching required? And what if this thing also had emergent properties? And, furthermore, what if no-one was in charge?
Say hello to tagging. I don't know where tagging was first used, but my introduction to it came when I started using the del.icio.us social bookmarking service as a link blog I call Jack's Tasty Links.
The basic idea is simple: Let everyone come up with their own tags (words) to describe a thing (web page) and let them attach these words to a link pointing to that thing. As many as they like. No limits. No minimums. No set of 'official' tags. Everything is completely arbitrary and up to the user.
Then take a look at the aggregate result of many users. Provide different ways to drill down and limit the results by combining tags. Use an open API so other people can create add ons to the service. That is how del.icio.us works and, let me tell you, it works well.
Very quickly others got in the tagging game. Gmail calls them 'labels'. Flickr uses them to create sets of related photographs. And Thomas Vander Wal came up with a name for what was happening: Folksomomy. (The guru of social software, Clay Shirky, deserves credit for the subsequent wide use of the term.) More recently Clay Shirky has published a powerful essay describing why folksomomy is superior to ontology: Ontology is Overrated -- Categories, Links, and Tags.
The simple beauty of tags is the fact that anyone can do it. The hidden power of tags arises from the fact tags cross service boundaries; a del.icio.us tag and a Flickr tag may be the same word and resources tagged similarly in both services are probably related in some way. No-one knows where this is going, but the funny thing is that this simple little idea which grew from the ground-up is not only catching on, it is already more useful than the top-down ontologist's 'Semantic Web'.
So, how do you get into the game? Well, I do like del.icio.us. But if you have a LiveJournal or a blog you can easily start tagging your posts using Technorati's new tagging service. All you have to do is create some special links at the bottom of your post and ping Technorati. They do the rest.
So, tag, you're it! Go thou out and tag the universe!
EDIT Dec 2010: Not long after I wrote this five years ago, LiveJournal added tags as a feature and Yahoo bought del.icio.us. Just recently Yahoo decided to shutter del.icio.us, causing me (for one) a whole lot of pain. The underlying points remain.