http://norman.walsh.name/2008/08/04/aboutXML 11 55 2008-08-04T16:32:29-04:00 Norman Walsh 2008 Norman Walsh Having an XML server at my disposal is making me think about XML applications differently. I've been writing XML applications for a long time. Arguably, since we spelled XML “es” “gee” “em” “el”. In those years, I'd grown to think of XML applications as being primarily operations on some principle document: a book, a web page, an Atom feed, what have you. (I'm not saying all XML applications are like this, I'm just saying this is how I tended to think about them.) Individual documents were sometimes composed from several files (via entities or XInclude) and some applications operated on a small number of files, but there was always at least some logical sense in which there was “the main file” and its ancillary files. When my application involved a potentially large number of files, I usually massaged them into a single file and used that as one of my small number of files. All of the many and varied sources of information used to present essays in this weblog, for example, are aggregated into a honking big RDF/XML document and that document is used as an ancillary resource when formatting the XML for each essay, the “main file”. One of demos I constructed to learn more about Mark Logic Server was a “W3C Spec Explorer”. I took all of the W3C specs and poured them into the server then I set out to write some XQuery code that would allow me to view the specifications by date, by working group, by editor, and through full-text search (or any combination of those options simultaneously). My starting point for building the sort of faceted navigation that I had in mind was the RDF metadata that the W3C provides. (Not only was I interested in having better full-text search of the specs for myself, I was also interested in exploring RDF in the server.) In the course of learning how best to build this application, I posted a question on the internal “discuss” list asking some fairly basic questions about how to efficiently search RDF's odd serialization. One of the replies that I got suggested (clearly, concisely, and patiently), that I was thinking about the problem from the wrong end. I had gone out and asked the server to give me a fairly big document and now I was trying to grub around inside it to find stuff. Instead, I should “push the constraints to the database”. Don't just ask the server (the database) for the file, ask it for the actual elements I care about. This did two things: first, it made my searches instantaneous or so nearly so as to make no difference. Second, it made me start to think very differently about XML applications. The server's “universal index” over all the content in the database makes it practical (often blindingly fast) to ask questions over an enormous number of documents. Want all the rdf:Description elements of type “REC”, just ask for them: //rdf:Description[rdf:type/@rdf:resource="…#REC"]. That's not all of the descendants of some document, that's all of them anywhere in the database. In fact, in my application, I broke the big RDF document up into a bunch of documents, one for each rdf:Description so that I could ask for /rdf:Description, rather than looking at all descendants anywhere. Had I needed to, I could have limited the search to a particular collection or used any of a number of other mechanisms for making it very targeted. Maybe I'm just discovering something that was obvious to all of you, but I'm now thinking of XML applications over not just a few files, but a whole database. My world is suddenly a lot bigger which is very cool. There's a cool footnote to this essay too (though I'm not actually choosing to make it a footnote, but nevermind). The guy doing all the patient explaining was Dave Kellogg , our CEO. Bonus points for a paradigm shifting answer. That our CEO took the time to read and answer a mundane technical question from a newbie on an internal list set the stage for something that was really driven home to me a couple of weeks ago at my first ever “semi-annual kickoff meeting”: this company is full of excellent people. Every single one of them, as far as I can tell. I'm having a ball.