Is It the APIs, Stupid?

Volume 7, Issue 99; 14 Jun 2004; last modified 08 Oct 2010

You’ve got your regular expressions, your SAX, and your DOM, you’ve even got your RDF as an XML API (I’m not kidding). And now you’ve got another choice in the Java platform. Some thoughts on APIs.

No man's knowledge here can go beyond his experience.

John Locke

Two (more) projects are competing for my (limited) spare cycles these days: docbot, a little IRC bot, and “Where In The World” (WITW), my first attempt to build a “complete” Web Service. Both of these projects have caused me to think seriously about APIs, albeit for very different reasons. This essay is about docbot; I’ll write about WITW some other time.

A Little Background

A couple of the IRC channels that I hang out in have useful bots; a bot is just a computer program masquerading as a user in IRC. Phenny in #rdfig is a good example. Phenny will search through the Unicode character set for code points, perform dictionary and Google queries, and do a half-dozen other useful tasks. For example, I’m using PircBot as the framework for docbot; if I ask Phenny to Google it, she’ll tell me the top answer:

   <ndw> | .g pircbot
<phenny> | pircbot: http://www.jibble.org/pircbot.php

The most useful bot I know of is one you won’t be able to see in action unless you participate in W3C Working Groups: Zakim on the W3C teleconference bridge. Zakim can tell you who’s on the call (using caller ID to associate IRC nicknames with numbers), track a speaker queue, manage an agenda, and perform a whole host of other functions. The highest praise I can give Zakim, and I intend it as high praise indeed, is that I miss it dearly on every other teleconference bridge I use.

DocBot

I decided it would be fun to add a bot to #docbook, so I wrote DocBot. DocBot will answer a few stylized questions about DocBook, as the following exchange demonstrates:

   <ndw> | docbot, what is refname
<docbot> | refname: The name of (one of) the subject(s) of a reference 
<docbot> | page 
   <ndw> | docbot, is refmiscinfo in refinfo
<docbot> | I don't know anything about refinfo 
   <ndw> | docbot, content model of refentry
<docbot> | Content model of refentry: (beginpage?, (%ndxterm.class;)*, 
<docbot> | refentryinfo?,refmeta?, (remark|%link.char.class;)*,
<docbot> | refnamediv+,refsynopsisdiv?, (refsect1+|refsection+)) 
   <ndw> | docbot, is refmiscinfo in refmeta
<docbot> | Yes, refmiscinfo is in the content model of refmeta

I don’t know if it’s really very useful, but it was fun to play with and it was an excuse to learn something about JavaCC.

Implementation

APIs come into the picture when you consider how DocBot knows the answers: it reads an XML document. Right now, the XML document containing the answers is cobbled together from bits of TDG and the DTD. I have plans for better integration into the RELAX NG schema for DocBook NG, but that’s an essay for another time.

An early prototype of DocBot, written in Perl, just grovelled over this document using regular expressions (what could be more natural in Perl?). When I switched to Java, that became less practical.

How then, was I to get at the data? A few options occurred to me:

  1. Use SAX.  Build my own internal data structure from a SAX event stream generated by parsing the XML document.

  2. Use DOM.  Load the XML document as a DOM and build my own internal data structure by walking the DOM.

  3. Use DOM.  Load the XML document as a DOM and simply use that DOM as my internal data structure, writing code to find the answers as necessary.

All eminently practical.

And all of them require writing some rather dull code to walk over this data structure and find the bits needed. Because I’ve been working on JpegRDF recently, I have Jena’s RDF API swapped into my head, so a fourth option occurred to me:

  1. Use RDF.  Convert the XML into RDF and use RDF queries to access the data.

    This isn’t quite as far-fetched as it sounds. The Jena toolkit is a pretty nice API and DocBot’s data really is “triples”: object (element refentry) has property (content model) with value ((beginpage?,…).

    Using Jena, I could reduce all the code to walk over the data structures down to a simple RDQL expression and let someone else’s code do the work:

    SELECT ?rsrc, ?value WHERE
    (?rsrc rdf:type <http://www.w3.org/TR/REC-xml/element>),
    (?rsrc bot:name "refentry"),
    (?rsrc bot:content-model ?value)
    USING bot for <http://nwalsh.com/rdf/docbot#>

So I could take a native XML data source and turn it into RDF just to use a convenient API. I don’t think RDF is naturally the right answer here (RDF evangalists of the world, revolt not; I didn’t say it was a wrong answer, just not obviously the right answer). If you’d been doing this and you’d asked me if you should convert the XML to RDF, I would almost certainly have said “no, what on earth would you do that for?” And yet, I could see doing it for the API.

Then BAM! some distant coding relative of Emeril Lagasse leaps out of the jar file with a hot, smokin’ new idea!

  1. Use XPath.  Load the XML document as a DOM and just grab the answers with XPath.

The XPath API is part of JAXP 1.3 which will be part of Java 1.5 which is now available in public beta.

This is so obviously the right answer that I’m embarrassedDoubly so, considering that I’m one of the leads for the committee that created JAXP 1.3. My excuse, miserable and insufficient though it is, is simply that I haven’t got used to thinking about the new APIs in production code yet. I didn’t think of it sooner.

The XPath API solution is anti-climactic in its simplicity:

XPath xpath = XPathFactory.newInstance().newXPath();
String expr = "//element[@name='refentry']/content-model";
String result = xpath.evaluate(expr, dom);

Separating the text of the XPath expression into a separate string variable is just a convenience because in real life that expression is constructed from some method parameters.

I think there are two lessons here:

  1. The API you choose can have a big impact on the complexity of your application.

  2. The APIs we use shape how we think about a problem. Even though I’m thoroughly familiar with XPath and have been intimately involved in the creation of the Java XPath API, it wasn’t the first thing I thought of because it’s not what I’m used to doing in the context of Java applications.

Neither of those “lessons” is earth shaking news, nor am I the first to observe them but they’ve been driven home again, very clearly, for me.

In point of fact, I wrote the code to use the Jena API and it was only while thinking about this essay that I thought of XPath. Hopefully you’ll forgive the teaser in the abstract on the basis that the XPath API was a surprise to me half way through this essay and I hoped it would be a surprise for you too.

Comments

It seems you have just started scratching the surface of ways to work with XML from an API. See http://www.xml.com/pub/a/2003/07/09/xmlapis.html which shows the state of the art as at last year.

—Posted by Dare Obasanjo on 14 Jun 2004 @ 10:57 UTC #

Interesting. I should probably have said explicitly that my purpose wasn't to catalog all the possibilities, only the ones that crossed my mind as I considered the task at hand.

I also had some web services API issues in mind when I started, but I decide to leave those for another essay.

—Posted by Norman Walsh on 14 Jun 2004 @ 11:46 UTC #

What IRC clients do you prefer on the various platforms? (Solaris, Mac, Win)

—Posted by Scott Hudson on 15 Jun 2004 @ 03:48 UTC #

No way, you can't be kidding me. I thought the use of xpath libraries (i.e. Jaxan) to work on XML was done all the time!

—Posted by Carlos Perez on 21 Jun 2004 @ 07:32 UTC #