Norman.Walsh.name

Creating a DocBook V5.0 DTD

2010-03-18T19:38:13Z

Taking another stab at the long-standing problem of producing DTD (and XSD) versions of the DocBook V5.0 family of schemas.

In the course of preparing the DocBook V5.0 schemas, I devised a process that would convert the DocBook V5.0 RELAX NG grammar into an XML DTD (we get the XSD by running Trang over the DTD). Closer inspection quickly reveals two flaws in this process:

It's incredibly brittle; while it successfully converts the base DocBook schema, it's utterly useless on even a simple customization layer.
It produces utterly crap DTDs.

The former problem is the one that's really causing me pain, though I admit I'm a little embarrassed by the second.

The Publishers Subcommittee identified an XML DTD as a requirement (Come on tool vendors! Get your act together. It's the twenty-first fscking century, already!). My hopes that someone else would fix the problem before I got to it went unfulfilled, so over the last few days I've turned my attention seriously to the problem. A few observations:

The general problem may be insoluble. It may simply be that there's no algorithmic path from the simple and expressive constraints of RELAX NG to the much less expressive constraints of DTDs. Or maybe there are several paths, but the results will never look rational to a human observer. Or maybe the solution is out there, just waiting for some enterprising grad student to find it (nudge, nudge). I'm not sure. I decided that I didn't have time to look for that solution.
If the computer can't solve the whole problem, then we'll have to rely on human intervention to solve the hardest parts. In the DocBook family of schemas, the apparent hard parts are attribute co-constraints and multiple patterns for the same element name.
The solution in both cases is to replace the problematic patterns with a single pattern that is the union of the various options. This creates content models that are too broad: they accept all valid documents, but they also accept some invalid documents. The union of the CALS and HTML table models, for example, is a model in which a element can contain a mixture of CALS elements and HTML elements, among other atrocities. Such was it always with XML DTDs.
The constraints on mixed content in XML DTDs are a pain in the *ss. In a DTD, mixed content must be expressed as (#PCDATA | a | b | c | …)*. The #PCDATA token must come first, the alternatives must be at the top level (no nested parenthesis), and there must be no duplicates among the alternatives.
There are a few places where we allow extensions in “other namespaces.” The elements can contain arbitrary additional metadata elements, for example, and equations can contain any MathML markup. DTDs and namespaces do not play well together. It might be possible to create a DTD that allowed MathML in the appropriate places, but that's more than the minimum needed to declare victory.
Patterns are a little bit like parameter entities. (Ok, a really, really little bit.) It would be nice, where possible, to represent the patterns as parameter entities in the resulting schema. At worst, it does no harm, at best it makes the DTD easier to read and may allow some small amount of customization of the DTD, not that I'd recommend that! And in any event, will solve at least a tiny part of the second problem mentioned above.

With these things in mind, I decided to adopt the following approach:

Create a “DTD” customization layer in RELAX NG that removes the most difficult problems: create unions for the attribute co-constraints, create unions for the elements that are defined by several patterns, remove the elements in “other namespaces” extension points, etc.
Create an “override” document for describing a few more operations. For example, removing the db._phrase pattern and changing all the patterns that use it so that they use db.phrase instead. (If there's an easy way to accomplish that in the RELAX NG customization layer, but it eluded me.)
Massage the modified schema until it's possible to create a DTD from it.

I'd like to say that there was some deep, theoretical insight in the last step, but there wasn't. I just built a pipeline of transformations that got me from A to B. I looked at the document, found something that wouldn't work in a DTD, wrote a transformation to remove it, and added that transformation to the pipeline. Repeat until done. When the result was accepted by an XML parser and accepted a small, valid DocBook document, I called it done.

Here's a 10,000 foot summary of the process.

Starting with the “DTD” customization layer, perform some simplifications. Discard documentation, schematron rules, the start pattern, divisions, etc. Turn interleaves into choices; this is a little risky, but seems to be ok in the DocBook family of schemas. Extract the content of pattern definitions, producing a set of elements and a set of “parameter entities”. Drop schema facet constraints and not allowed content on the floor. Etc.
Apply the overrides, as described above.
Remove “choice” wrappers from around attributes. Fiddle with how “optional” is expressed. In RELAX NG, it's a wrapper, for the some steps in this process, it's more convenient to make it an attribute.
Remove parameter entities that are no longer referenced.
Fiddle with “optional” again, moving the optionality down to the references. (An optional reference to something is the same as a reference to an optional something.)
In subsequent steps, it's going to be convenient to be able to distinguish references to attributes from other references, so turn all elements that point exclusively to attributes into elements.
Flatten chains of references to attributes. (If A points to B points to C points to D which is an attribute, then just make A point to D.)
Fiddle with “optional” again. This time move the optionality up to the attribute declaration. This may require splitting a declaration.
Check for element names defined by more than one pattern. There better not be any.
Remove “empty” parameter entities and references to them.
Pull “text” up. Replace any reference to a parameter entity that contains #PCDATA with a copy of what the parameter entity contains.
Unwrap nested “zero-or-more” elements.
Sort the parameter entities so that we never attempt to use one before it's been declared.
Convert the resulting document into a DTD. Turn parameter entities into !ENTITY declarations, turn elements into !ATTLIST and !ELEMENT declarations, substitute DTD attribute types for the specified types, expand mixed content models, etc.

The most disappointing part of that last step is fully expanding the content model of every element that contains mixed content. I'd been working pretty hard all along to preserve as many pattern names as possible as parameter entities.

The problem is that even though all the relevant parameter entities are simple lists of elements (so they could appear in a mixed content element declaration), sometimes the same element name appears in more than one pattern. So I punted, expanded them all, and removed duplicates. I still think it might be possible to do better.

In retrospect, this isn't too surprising. If you go back to the DocBook V4.x DTDs and study the parameter entity structure [“masochist” -ed], you'll find a few places where we twisted the parameter entity structure pretty hard to avoid exactly this problem.

In any event, the DTD that results from this process is an XML DTD that appears to validate DocBook documents. With different customization and overrides, the DTD version of the publishers schema also seems to work.

I'll get it out in the next day or so for wider testing. It's very likely that there are places where it's not quite right. But it's definitely an improvement over the old process.

Pipeline Notes

The process described above is not wholly unlike what I did before. One significant factor that made this attempt more successful was XProc . It's not impossible to chain together 14 transformations with a big XSLT 2.0 stylesheet and a bunch of modes, but it's a whole lot harder to manage.

Speaking of XProc, I cheated. The pipeline I'm using today will only work in XML Calabash because it relies on a compound extension step: cx:until-unchanged. That step is a little bit like p:for-each except that after each iteration it compares the input document to the result of applying the pipeline and repeats the process (using the output of one iteration as the input of the next) until the result is the same as the input.

It's not impossible to do this without extending XProc, but it requires writing a different recursive pipeline for each looping step. It was more interesting (for me) to see how hard it would be to write a compound extension step. (So shoot me.)

By the way, if you're curious, converting the base DocBook schema to a DTD is a 40 step pipeline (more or less):

INFO: Running pipeline main
INFO: Running xslt rng2dtx
INFO: Running xslt override
INFO: Running cx:until-unchanged remove-choice
INFO: Running xslt attr-remove-choice
INFO: Running xslt attr-remove-choice
INFO: Running xslt attr-remove-choice
INFO: Running xslt attr-remove-choice
INFO: Running cx:until-unchanged remove-unused
INFO: Running xslt attr-remove-unused
INFO: Running xslt attr-remove-unused
INFO: Running xslt attr-remove-unused
INFO: Running xslt attr-optional-to-ref
INFO: Running cx:until-unchanged to-attref
INFO: Running xslt ref-to-attref
INFO: Running xslt ref-to-attref
INFO: Running cx:until-unchanged flatten
INFO: Running xslt flatten-attref
INFO: Running xslt flatten-attref
INFO: Running xslt flatten-attref
INFO: Running xslt attr-optional-to-decl
INFO: Running xslt multiple-gis
INFO: Running xslt remove-empty-pes
INFO: Running cx:until-unchanged pull-up
INFO: Running xslt pull-up-text
INFO: Running xslt pull-up-text
INFO: Running xslt pull-up-text
INFO: Running xslt pull-up-text
INFO: Running cx:until-unchanged unwrap
INFO: Running xslt unwrap-zeroormore
INFO: Running xslt unwrap-zeroormore
INFO: Running cx:until-unchanged sort
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt dtx2dtd

What about a better W3C XML Schema?

The DTD that results from this conversion process is a little bit unsatisfying. It just not what a human being would do if they started from scratch. On the other hand, it doesn't matter much; there's no widespread use for DTDs beyond validation and perhaps guided authoring. (And you ought to be using RELAX NG for that, see previous comment about the twenty-first century.)

The same is not true of W3C XML Schemas. There would (just possibly, maybe) be value in having a better XSD for DocBook. There are data binding tools and other applications that would fare much, much better with DocBook if they were given something that took proper advantage of XSD's native facilities.

I've never had much interest in writing an XSD for DocBook. I'm not likely to ever be persuaded that “type inheritance” is a satisfying abstraction for how content models are related. I never felt that XSD 1.0 was a good foundation for the kind of “human prose” schemas of which DocBook is a typical example. But I'm almost convinced that XSD 1.1 has fixed some of the most inconvenient deficiencies.

My bailing wire and duct tape solution for generating DTDs doesn't seem like it's ever going to be up to the task of doing the conversion properly. I'd be delighted if the aforementioned enterprising grad student built a tool to do the conversion automatically, but if not, I just might (someday) take a crack at hand authoring a proper XSD 1.1 schema for DocBook.

XProc Proposed Recommendation!

2010-03-10T11:01:16Z

I'm pleased to report that XProc: An XML Pipeline Language is now a W3C Proposed Recommendation.

The Proposed Recommendation draft of XProc: An XML Pipeline Language was published yesterday!

I'll save a more reflective post about the process and the result for after we've crossed the last hurdle. In the meantime, here are a few useless statistics. Between start and PR:

We had 169 meetings over 4 years and 73 days, give or take a day or two.
We produced 12 drafts: seven working drafts, two last call working drafts, two candidate recommendations, and one proposed recommendation.
I stood on the podium in front of a conference audience and asserted that we'd be finished “within a year” on at least three separate occasions. Maybe four.
We had four face-to-face meetings. Murray kicked us of in style at his place and we met at three W3C Technical Plenary meetings.
The community generated two complete, interoperable implementations and a number of additional, partial implementations.

And, most important, we developed an active (and growing, I think) XProc user community. On the whole, a success by any metric, I think.

Once again, I'd like to extend my congratulations and heartfelt thanks to the members of the Working Group, reviewers, and implementors that have helped us come this far. We couldn't have done it without you.

Wiki editing with XProc

2010-03-07T21:25:44Z

An example, for better or worse, of automating website interaction with XProc.

What happened was, the DocBook wiki broke. I don't know how or why, but it fell over. The problem, whatever it is, left the wiki immutable and the underlying database in a state of questionable consistency.

Clearly a problem that had to be fixed. I setup a new wiki, running MoinMoin 1.9.2 instead of 1.3.4 [Upgrade much? -ed].

In theory, there's an upgrade path from 1.3.4 to 1.9.2 but I'm sufficiently unsure about the state of the current database that I'm loathe to use it. The last thing I want to do is put the new wiki into some indeterminate state. Instead, I grabbed all the most recent pages from the old wiki, trimmed out a bunch of cruft, and cleaned up the markup a bit (the wiki markup seems to have changed over time).

What I really wanted to do was add all these pages to the new wiki. Easy enough to do with a browser for one or two pages, but several hundred pages was way more than my patience would tolerate.

A quick experiment with HTTP Scoop made it it look pretty easy:

Logging in sets a cookie.
Loading a page that doesn't exist provides a link that you can follow to create the page.
Following that link returns an HTML page containing a form with a place to type the wiki markup and a bunch of hidden fields.
Posting that form back to the server updates the page.

If only I had a tool that could make HTTP requests and process the results…wait, wait, I have one of those!

XProc ought to be up to this job, yes? Yes! In fact, it was reasonably straightfoward. Wanna see how it works? Of course you do. The following pipeline works in XML Calabash version 0.9.20 or later.

I decided to pass the wiki markup as an input and the page name as an option. From the option, I construct the value of the URI for the page.

Next I have to login:

I reverse engineered the way the login form works. I URL encode and pass my username, password, and other parameters to a that POSTS them to the server.

I don't care about the result, so I drop it on the floor with .

I do care about cookies, so I have to store those somewhere. XML Calabash has an extension that lets you manage cookies in named sets. This saves any cookies that come back in the “login” set.

Next, we have to get the page we want to edit.

I use the “login” cookies so that the wiki knows who I am. I also use the cx:depends-on attribute to tell the processor that this step depends on the preceding login step, even though there's no dependency in the flow graph. Without this explicit statement about dependency, the processor might attempt to get the page before performing the login step.

Once again, I don't care about the output so I drop it on the floor. In theory, I have to parse the output and find the “edit” link. In practice, I know how to create it without looking for it in the markup. I'm not even sure I have to do this step, but it is what a browser does and it was easy to do so I left it in.

Now we want to get the page that includes the edit form:

Again, we use the login cookies. And this time we don't drop the output on the floor because we have to extract the hidden fields from the page in order for our subsequent POST to work.

To get the hidden fields, we unescape the markup. XML Calabash uses TagSoup for “text/html” pages, so we'll get well-formed XML.

The loop selects each of the hidden input fields and transforms them into elements. We'll need those later.

Next, we have to construct the for the “savetext” parameter that contains our wiki markup. This one's a bit tricky.

What the hell, I hear you ask, is up with that “replace” value?

Well, see, what's going to appear on the source input port of our pipeline is a element that contains the wiki markup of the page. The replace option is interpolated as an XPath expression, so we have to “quote” the value. This is a common idiom in ^[1]. Except, in this case, the value may contain both double and single quotes, so we need to make sure that they don't result in an invalid XPath expression!

Imagine that this is our element:

"Hello 'world'"

If we do the usual quoting trick, the resulting XPath expression will be:

'"Hello 'world'"'

and that's not a syntactically valid XPath string value. So we use replace to double-up the apostrophes. That gives us

'"Hello ''world''"'

which is what we want. That took me a minute or two, believe you me.

Next we wrap all our elements in a , construct a to hold them, and use to encode them.

Send that off to the server and we're done!

I display the result, after deleting its contents, just to make sure that I got a 200 back.

That little XProc script got all the pages loaded in just a couple of minutes. FTW!

If you're interested, the whole script is available.

^[1]So common that I regret not providing some sort of syntactic shortcut for it. Oh, well, there's always version 1.1.

Where am I?

2010-03-06T21:57:48Z

Or, perhaps more to the point, where was I? And where will I be?

I've long been fascinated by geospatial data. I do a little geocaching . I keep track of the countries I've been in. I use services like Dopplr and TripIt to keep track of my itineraries. I carry a GPS to geotag photographs.

Carrying a mobile phone with a GPS allows me to explore the features of geosocial applications like BrightKite, Gowalla, and Foursquare. I allow Dopplr and Brightkite to update my Fireeagle location.

All very nice, but there were two obvious (to me) deficiencies in this arrangement. First, and most obvious, my data is spread all over someone else's servers. I consider this unacceptable. Any one of these services could get bought, go belly up, or carelessly (or maliciously, I suppose) discard all my data.

The second point is less obvious: how do I tell when I'm home? I don't actually think any of you reading this would have a hard time working out where I live. I'm sure I've left enough digital clues for anyone sufficiently interested to work it out. That said, I can't actually convince myself that it's reasonable to “check in” to any of these geolocation services when I'm home. I'm not sure it's reasonable to do when I'm not home, either, but I do. I think I've set the services up so they only reveal my location to friends anyway.

Not being able to solve the second of these problems made the data sufficiently unreliable (I've been checked into Staples for eleven days?) that the first problem hadn't crossed my “do something about it” threshold.

All that changed a few days ago when Tom Morris happened to mention a clever solution in his Twitter stream. He later documented it in a thoughtful post. The basic idea is this: if you always carry your phone (or other device) on your person, then the presence of that device in your house means your home.

A little hack later and my server always knows when I'm home, give or take a few hours; it can only ping the device when it's on and not in “standby” so there's some latency. But not more than a few hours most days, I expect.

Having solved the second problem, I turned my attention to the first. A few hours of hacking later and the TripIt, Foursquare, Gowalla, Brightkite, and Fireeagle APIs are giving me my data. The hardest part, honestly, was getting over the authentication hurdles. OAuth may be the right answer, but it's not painless to setup a new application.

I started out by pouring all this data into MarkLogic Server. (Well, I would, wouldn't I?). A little XQuery later and I had a normalized view of all my locations. Cool.

But wait, I thought, what about all those GPS tracks? Yes, those belong in there as well! Easily done.

The interesting thing about GPS tracks is that you can (sometimes) interpolate data between points. I do this already when I'm geotagging photographs. By adding “next point” to the normalized data when appropriate, I could expose that in my system as well.

Once that idea was in place, it was clear that an airline flight or train ride (to a lesser extent) might be subject to interpolation as well. A quick tweak to the scripts that normalize TripIt itinerary data took care of that.

At the end of the day, I have an interesting (to me) personal archive of my geolocation over time. It's derived from GPS tracks, explicit checkins, and itineraries. I'm also going to integrate the GPS data that comes from photographs taken with my mobile phone. All very cool to me.

I've also got a web service that I can use for geotagging photographs. I can ask, for example, where was I this morning at 10:00a?

Staples

Apparently, I was at Staples and had been for 8 minutes. No wonder I'm the mayor of Staples.

If I ask where I was at 2006-07-15T13:28:00Z, the answer comes from a GPS track:

That's a location interpolated over five seconds and about 44 feet. Seems pretty reasonable.

Interpolating over airline flights is a little less precise:

BOS

but it's still kind of cool. Also interesting is the fact that itinerary data lets me look forward. Where will I be at 2010-03-12T08:25:00Z?

PRG

Oh yes. I think I have an appointment there. And it'll be more accurate after I've actually checked in, taken photographs, and used my GPS. Sweet.

What your drive knows, and what it doesn't

2010-03-01T20:39:11Z

I recently had occasion to swap hard drives between two essentially identical laptops. A surprising number of apps knew the difference.

I have two essentially identical laptops, loaded 17” MacBook Pro's. One is my personal machine and one belongs to Mark Logic. I use my personal machine most of the time, but the fans have gotten insanely loud. I reported this as a warranty issue and got approval to take it in for service. (Yay!)

Not wanting to be without my laptop for several days, I decided to be clever and swap hard drives. The install videos at Other World Computing couldn't be more straightforward. (I had already bought the appropriate tools when I upgraded to a 500Gb drive.)

Following the swap, I found a curious mixture of systems and applications that could tell.

Not surprisingly, the iTunes authentication system could tell.
Also not surprising, Time Machine could tell. I suppose it's just possible that two folks could have absolutely identical but physically different hard drives. I really don't know how it could tell, though.
Some of the menu bar widgets were different. Apparently whatever makes the little flag icon is in flash memory somewhere and not configured from the hard drive?
The one that really surprised me was VMWare Fusion. It didn't like any of my virtual disks, it thought they were all “in use”. Stealing them back and telling VMWare I'd “moved” them was enough to recover, though, so no harm done.

My laptop is back and the drives have been switched again. I wish the problem had been fixed, but they claimed to be unable to reproduce it. (Boo!) I can tell you right now that it's still a problem.

I should push harder to get the fans replaced, the intermittent high pitched whine is definitely a Bad Thing. Of course, the 17” MacBook Pro is due for a refresh any day now, right?

Demo Jam at XML Prague!

2010-02-23T20:13:39Z

Demo Jam was a huge success at Balisage last year, so we're going to give it a go at XML Prague too!

If you're coming to XML Prague, please plan to come to demo jam!

Here's the scoop: we provide the microphone and the projector. You provide the laptop and the demo. You get five minutes (that's 300 seconds, no more) to demo anything you want, as long as you can reasonably claim that there's some XML in there somewhere. We'd love to see some XQuery and maybe even MarkLogic Server (you could use the free Community License, for example), but that's not necessary.

Judging is by audience participation: if you make the crowd cheer the loudest, you win. Winner walks away with a free pass to the Mark Logic User Conference, May 4-6 in San Francisco, CA, US. Everyone gets a T-shirt.

Bring your demo and your posse to the social evening; demo jam starts at 8:30p. See you there!

XML FTW!

2010-01-25T22:21:37Z

On the serendipitous joy of finding XML.

As I've said before, I'm very reluctant to use your application if it's a roach motel for my data. It would not be fair to say that I'll refuse to use your application, it's just a lot less likely.

For example, when it came to GSD, I decided that open access wasn't as important as picking an application that I'd actually use. If I let myself get distracted by exploring APIs, there'd be other things not getting done! (Priorities!)

Having made my bed, I figured I should see what I was lying in. Today I took a peek at how OmniFocus stores data. Now, the title of this essay no doubt gives away the punch line, so consider for a moment how this would have been done in the time before XML.

…go on, have a think, I'll wait…

In my experience it would probably have been in some proprietary format, almost certainly binary, and utterly opaque. How many tools document(ed) their proprietary data formats? On some platforms, there might have been system services for storing data, some sort of platform-supported database perhaps. Those systems are (often) only marginally better. They produce, instead of an opaque stream of bits, an opaque stream of atomic values. (Don't get me wrong, I've done the reverse-engineering thing on binary formats, I'd prefer the stream of atomic values, believe you me.)

What did I find when I went looking at the OmniFocus data? A directory full of ZIP files. And what's in each ZIP file? Why contents.xml, of course!

Now, it would not be fair to assert that this is perfectly transparent. XML isn't magic. There are clearly some cross-reference relationships in there that will take a little mental gymnastics to decode. But still, I'll trade this:

...

  
  2010-01-21T16:23:08.983Z
  2010-01-24T21:01:41.632Z
  Add server-side support for multipart MIME to tests.xproc.org
  2113929216
  
  2010-01-27T22:00:00.000Z
  2010-01-24T21:01:41.622Z
  parallel

...

for anything I would ever have gotten at any other point in the history of file formats!

XML has its detractors. It would not be fair to say they are all wrong. But I'll take XML over fair any day!

GSD!

2010-01-25T15:13:26Z

Our engineering department has a project management philosophy they describe as GSD. I aspire to GSD.

For me, the part of GSD^[1] that I most often have difficulty with is keeping track of what needs doing. My todo (or want-todo) list is absurdly long. If I feel like castigating myself, I can always find a few things on my list that should have been done by now. It's not that I don't work hard or get a lot done, it's that I don't always prioritize perfectly and sometimes things slip through the cracks.

I've been trying to get better at this. Having an online calendar sync'd with my phone keeps me from accidentally missing meetings and phone calls, so it seems to follow that some sort of online system should be able to help me with my todo list.

My requirements are pretty simple: I want something that's easy to use and I want something that syncs with my mobile device. An online tool is almost, but not quite, as good as something that I can use offline on my PDA.

I don't subscribe to any particular Getting Things Done methodology. Maybe I'll get there someday, but that's not my immediate goal.

I played with Remember The Milk on-and-off last year. It seemed to work pretty well for simple lists, but I wasn't using it consistently because, I think, it wasn't quite powerful enough.

This month, I took a few different systems for a test drive: 2Do, Toodledo, Things, and OmniFocus.

Unfortunately, 2Do is only an iPhone app. It appears that there are plans for the next version to support syncing with Toodledo, but that doesn't exist today. Toodledo is a web-based app and is quite nice, probably plenty sufficient for my needs. On the desktop front, both Things and OmniFocus are probably plenty sufficient as well. (There are no doubt other similar applications, those are just the ones I happened to try. I didn't attempt an exhaustive survey, I've GStD!)

And the winner is: OmniFocus, by a narrow margin. I like the project/context duality that OmniFocus uses (ToodleDo has contexts too, if you turn them on). Mostly it boiled down to the UI: I liked the “feel” of OmniFocus best.

This is an app I plan to force myself to use, so I figured I'd best pick one that felt good. It's also the most expensive, by a pretty wide margin, but c'est la vie.

Will this really work for me? Time will tell. But so far, so good. And I'm already learning to use it in ways I hadn't planned: maintaining shopping lists and travel check lists. Those aren't the sorts of things for which I would have actively sought out software (sometimes a pencil and a piece of paper really is enough), but it's encouraging to me that I have other reasons to be paying attention to my GSD tool.

^[1]Getting Shi␈␈␈Stuff Done!

XML Prague 2010

2010-01-24T18:27:40Z

See you at XML Prague! And a chance to plug some really excellent training.

I'm delighted that my paper proposal for XML Prague was accepted. I'm a little less delighted that the final paper deadline was last week, but I guess that's encouragement to finish it, eh? (I will, I promise.)

I'm going to speak about modular documentation in DocBook, both about the proposal for “assemblies” being developed in the DocBook Technical Committee and about my XProc-based implementation.

I mention these things for only two reasons^[1]: first, to recommend that XML Prague is a conference you need to go to if you're interested in XML technologies. It's that good.

Second, to plug G. Ken Holman ’s XSLT/XPath 1.0 & 2.0 and XQuery 1.0 Hands-on Training class. If you're looking for someone to teach you XSLT and XQuery, you'd be hard pressed to do better than Ken. And if you're interested in XML, these are technologies you need to know. The maximum class size is an astonishing six, so it's practically 1:1. Yet another reason to be in Prague in March

See you there!

^[1]In the interest of full disclosure, I should point out that Mark Logic is a gold sponsor of the conference and Ken's course is being delivered in partnership with our own training services. I don't think that makes me baised, but I guess I wouldn't, would I?

NYMUG: Cloud deployment options

2010-01-24T18:16:47Z

Denise Miura, Sr. Director of Product Management will be speaking about Mark Logic's new offering for the Cloud at our upcoming User Group in New York this Wednesday.

The second Mark Logic New York User Group meeting will be held on Wednesday evening, 27 January 2010, hosted by Steve Kotrch from Simon & Schuster!

What: An opportunity to learn more about MarkLogic Server and collaborate with other MarkLogic users.
When: Wednesday, 27 January 2010, at 6:00pm EST.
Where: 1230 Avenue of the Americas, New York, NY between 48th and 49th streets on Sixth Avenue.
Who: Everyone who shows up, of course! The featured speaker this time is Denise Miura who will explain Mark Logic's new cloud deployment options and describe how they are being used today within the Mark Logic development community. She will demonstrate instantiating a MarkLogic AMI live on Amazon EC2 and talk about best practices for using MarkLogic on the EC2 platform. Finally a preview of the planned cloud-related product enhancements will be provided. This is a great opportunity to provide feedback and influence the cloud computing initiative at Mark Logic.
How: If you plan to attend, please join the mailing list and send your first and last name to cleo dot saab at marklogic dot com.

If you're in New York, please stop by (and please let Cleo know if you plan to stop by).

XProc: Back to Last Call

2009-12-28T14:20:52Z

Early in January, a new XProc draft will appear. It will be a Last Call Working Draft, a step backwards in the process, or maybe just a half-step. The reason is important though: versioning.

The XProc WG has been making steady progress on XProc: An XML Pipeline Language. We saw the start of wide adoption of XProc in 2009 and I think there's every reason to expect more of the same in 2010.

This makes it all the more disappointing to report that we're going back to Last Call. On a personal note, as a TAG alum, it's a bit embarrassing to admit why: versioning.

We received significant and persuasive criticism of our versioning story. In particular, we were pesuaded that requiring a processor to download the declarations for V.next steps in order to process them in a “forwards compatible” manner was too burdensome.

In redrafting the story, we added a version attribute, “compile-time” use-when functionality à la XSLT, and extension functions for more precisely identifying the environment in which the pipeline is running.

We also took the opportunity to fix decisions that were, in retrospect, mistakes, but not in-and-of themselves sufficient to motivate us to return to last call: we changed the rules for connections in option, parameter, and variable bindings so that an explicit isn't required when there's no default readable port, and we changed the rules for parameter input ports so that a binding isn't required when there's at least one explicit . Users are already thanking us.

I didn't get the document through the publication process before the end-of-year publishing moratorium, but you can read the staged draft, if you wish.

Will this really be our last Last Call? I sincerely hope so! I also hope that we can move directly from Last Call to Proposed Recommendation without the formality of another Candidate Recommendation phase in between. There's precedent for doing this, and we've got active implementors.

I'm sometimes frustrated by how long the process takes, but I console myself with the observation that the language is better for our efforts. Rising usage suggests the early adopters, at least, agree with us.

David Alfred Walsh

2009-12-26T20:38:58Z

9 June 1923 — 26 November 2009.

A man may by custom fortify himself against pain, shame, and suchlike accidents; but as to death, we can experience it but once, and are all apprentices when we come to it.

— Montaigne

My father was born in 1923 in Babylon, NY.

David Walsh, age 10

He survived the Great Depression. An enormous tree blew over next to him as he walked home through The Great Hurricane of 1938; he walked away without a scratch. The glider born infantry took him to the China-Burma-India theater in WWII.

Bombay c. 1945

Left to right: Charles Kuhn, York PA; Bill Bride, Beacon NY; Eddy Evans, Boston MA; David Walsh, Babylon, NY

Shrapnel chipped a tooth, but he survived that too. After the war he went to Alaska.

Browerville from the tundra, Mar 1960

My dad taught in Barrow and Nome. After putting out a chimney fire, he walked away from a two story fall off a frozen roof by the lucky stroke of landing feet-first on an oil drum.

Camping in Alaska, c. 1960

He single-handedly built a one-room cabin on a ¼ acre plot in Fairbanks. (I think I remember seeing once a photo showing the scaffolding he built to get the roof beam in place.) He worked for the Fish and Wildlife Service in the summers.

He used to practice orienteering by walking into the Alaskan wilderness on a compass bearing and then walking back out again. On one occasion he stumbled across a downed single-engine plane containing the skeleton of its pilot. His boss laughed when my dad offered to lead a team back to the crash, assuring him that he'd never find it again. Dad's boss was right. There is a lot of wilderness out there.

On another occasion, my dad shot a caribou only to discover as he prepared to dress it that he'd left his knife back in the jeep. Leaning his rifle against a tree, he walked back and got his knife. An enormous brown bear greeted his return by standing on its hind legs and roaring. The bear got the caribou. And the rifle. And the knife, dropped during a hasty retreat.

That wasn't the only caribou that nearly got him killed; on another occasion, one attempted, unsuccessfully, to jump over his jeep. He woke on the side of the road with a caribou hoof protruding into the cab and a nasty gash on his head.

I'm lucky to be here.

Sleeping, June 1970

When my dad left Alaska, he gave the keys to his cabin to a friend. Those keys passed from friend to friend for more than twenty years. In the eighties, the current occupant persuaded my dad to let him buy the cabin. My father signed the deed and mailed it, asking the occupant to please mail the check back. The check came back a couple of weeks later. And it cleared. Luck of the Irish, or something.

From Alaska, my dad traveled to Australia. My mom and dad met in Tasmania. They married in 1961.

Mom and dad, August 1961

I came along a few years later.

Mom, dad, and I

June 1968

I remember my dad singing sea shanties when I was a small boy.

Playing guitar

Time and place unknown

Dad was a naturalist, hunter, trapper, fisherman, scientist, teacher, draftsman, and surveyor. He made beautiful wood carvings. He tied knots. At one time or another, all of them. I have his leather working tools. The old sewing machine on which he made sleeping bags, tents, parkas, rain slickers, and bicycle paniers got lost somewhere along the way. He built two boats.

David Walsh, Sep 2009

After 86 years, entropy won. Entropy always wins. My dad taught me that. And the first and third laws as well.

My father died in 2009 in Norwich, England.

'tis himself

David Alfred Walsh 9 June 1923 - 26 November 2009

Goodbye, dad.

NYMUG Summary

2009-11-12T16:16:03Z

Last night, I spoke at the inaugural New York Mark Logic User Group meeting. I think it was a crowd pleaser, or at least, the punchline at the end was.

The real purpose of a user group is to bring together users (and prospective users). There was much lively discussion after my presentation, which I won't attempt to recapitulate here. The next NYMUG meeting will (most likely) be sometime in January, so please plan to come. I'll post more concrete details here when they're available, and we'll send them to the mailing list, of course.

My biggest problem in preparing for speaking events is figuring out what to talk about, and then figuring out how to stretch that topic to fit in the time allotted.

When I was suggested as the speaker for our inaugural New York meeting, I had to figure out what to talk about. I quickly thought of a topic, but never imagined that it'd be possible. To my delight and surprise, when I shopped the idea around engineering and marketing, there was universal support for the idea. So after I picked my jaw up off the floor, I had to turn my attention to stretching the topic.

The topic I had in mind would easily fit on a couple of slides. That would make for a pretty short talk. At a user group meeting, maybe that wouldn't be all bad, but I felt I needed to say a bit more. I stretched things out by talking about three new and cool features that I thought some folks might not have seen or used yet.

Support for https: and URI rewriting

Support for https: is pretty self-explanatory. Lots of sites have private information (user profiles and passwords, for example) that should not (many would say “must not”) be sent over an insecure communications channel. Furthermore, most users have been trained to look for a secure connection before sending credit card or other financial information over the web.

Support for URI rewriting allows application authors to make cleaner interfaces. I'm a fan of clean URI interfaces. Call me picky, but think it's a lot better to expose the 4th slide in your presentation as http://example.com/slides/nymug/4 than as http://example.com/slides.xqy?deck=nymug.xml&foil=4&format=html.

Until recently, if you wanted to do this with a web site built on top of MarkLogic Server, you had to put up a proxy of some sort (often an Apache web server) to provide https: support and URI rewriting.

Starting with MarkLogic Server V4.1, this is no longer necessary. MarkLogic server now supports https (with your own certificate or a generated, self-signed one) out of the box. The server also supports URI rewriting by allowing you to designate an arbitrary query module to rewrite URIs. Here's the example I used in the presentation:

xquery version "1.0-ml";

declare variable $url as xs:string
        := xdmp:get-request-url();

if (matches($url, "^/slides/([^/]+)/([0-9]+)$"))
then
  replace($url,
          "^/slides/([^/]+)/([0-9]+)$",
          "/slides.xqy?deck=$1.xml&foil=$2")
else
  $url

It's an incomplete, toy example taken from the real code I used on the server behind my presentation, but it gives you a flavor for how it works. Your module starts with the URL that that was used (and access to the headers and other parts of the request), performs any sort of computation that you'd like, and returns the new URI. The new URI then goes into the server and is processed normally.

There may still be good reasons to put a proxy in front of MarkLogic Server (load balancing, etc.), but you no longer have to just to satisfy requests for these two common and simple features.

Office Toolkits

For reasons that will become clear later on (and not only because I find “office applications” to be an inefficient, frustrating, pointless time-suck), I wanted to present this presentation using ordinary web technologies. (Specifically, HTML+CSS+JavaScript served up by MarkLogic Server.)

At the same time, because I was going to talk about new server features, I was required to present a disclaimer:

Disclaimer

All statements describing future releases, estimated release dates and content are plans only, and Mark Logic is under no obligation to develop, include or make available, commercially or otherwise, any specific feature or functionality in any Mark Logic product.

Information is provided for general understanding and informational purposes only, and is subject to change at the sole discretion of Mark Logic in response to changing customer requirements, market conditions, delivery schedules and other factors.

(A disclaimer that applies as much to this weblog essay as it did to my presentation last night, I might add.)

Trouble is, this slide was sent to me in Powerpoint. To use it, I'd have to switch to Powerpoint for the rest of my presentation (yuck!), copy and paste the text (where's the fun in that?), or find some way to incorporate the slide into my DocBook-based slide deck (now that sounds interesting).

Luckily, one of our engineers, Pete Aven has already done all the heavy lifting. Pete's the primary developer of Mark Logic's open source toolkits for Word, Excel, and Powerpoint. Each toolkit provides a pipeline for ingesting office documents into MarkLogic server, an office plugin for using the server from the application, and some XQuery code to work wtih the files in the server.

With that framework in place, it was pretty easy to write a little bit of XQuery code that would extract a slide from a deck and transform it into DocBook. (That's about 20 lines of code, nothing fancy, it extracts paragraphs and bulleted lists from slides, no more, no less.)

The source for my final presentation looks like this:


  
    Transforming XML Development <?lb?>with MarkLogic
    …
  
  

    
      NYMUG!
      …
    

    

    …

a straightforward mixture of hand-authored slides and references to slides from a couple of Powerpoint decks. For the presentation, I edited a Powerpoint slide and ran it back through the process in real time, but that doesn't translate very well to a weblog essay.

Presentation

That left just the last part of my talk, “the big reveal” as it were. Having set this all up so that I can author in DocBook, even including Powerpoint slides, serve it up over https:, and use nice looking URIs, I still have to go the last mile and get the content into the browser.

A couple of obvious ways present themselves. I could serve it up as XML with a stylesheet and let the browser do the work. Could do, but I didn't. I could translate the DocBook markup into (X)HTML using XQuery in MarkLogic Server. Could do, but I didn't.

What I really want, but haven't been able to do, is to transform the DocBook in MarkLogic Server with XSLT. And drum roll … I can haz! cymbal crash

let $doc := xdmp:document-get(concat($ROOT, $xml))/*
let $expanded := local:expand-powerpoint($doc)
let $map := map:map()
let $put := map:put($map, ...("dbp:foil")), $foil)
let $put := map:put($map, ...("dbp:deck")), $deck)
let $put := map:put($map, ...(xs:QName("dbp:format")),
                    $format)
return
  xdmp:xslt-invoke($xslt, $expanded, $map)

Running an internal build, I can demonstrate support for XSLT 2.0 in MarkLogic Server. (Go back and read that disclaimer again now.)

There was some rejoicing at the user group meeting, I hope there's some rejoicing out there over the intertubes too. I'm certainly giddy with delight over the prospects of high-performance XSLT processing in MarkLogic Server V.future.

What more can I say. When you've done your best trick, it's time to get off the stage.

Thanks again to Steve Kotrch and Simon & Schuster for hosting. Hope to see you all in January!

NYMUG: New York Mark Logic Users Group!

2009-11-04T19:32:47Z

The inaugural meeting of the New York Mark Logic User Group will take place on Wednesday evening, 11 November 2009.

Please come join me at the inaugural meeting of the Mark Logic New York User Group on Wednesday evening, 11 November 2009, hosted by Steve Kotrch from Simon & Schuster!

What

I'd say more about what I'm going to say if I'd figured out more of what I'm going to say. The title of my presentation is Transforming XML Development with MarkLogic. I think it'll be a fairly interactive show. Without any spoilers, I'll admit to having some DocBook, 5.0 of course, some Powerpoint, some HTML, and I'm pulling them together in some pretty damn cool ways, if I do say so myself.

If you're in New York, please stop by (and please let Cleo know if you plan to stop by).

See you there!

DocBook V5.0

2009-11-04T18:55:29Z

DocBook V5.0 is an OASIS Standard!

I reported a couple of weeks ago that the ballot to make DocBook V5.0 an OASIS Standard was open. I can now report that the ballot is closed, it closed on 31 October.

To my great satisfaction, I can also report that we easily cleared the process hurdles.

DocBook V5.0 is officially an OASIS Standard.

I'd like to thank everyone who took the time to help us get here: users, reviewers, members of the Technical Committee (past and present), and the OASIS members who voted for us, of course.

Evernote

2009-11-01T21:51:13Z

With a scanner and some Python, I'm an enthusiastic convert to Evernote.

I first tried Evernote more than a year ago. It seemed interesting, especially the ability to extract text from uploaded notes, even handwriting. It's not performing traditional OCR, but it does build a list of likely words in each note. That's enough to make search quite useful.

Here we see a match for “America” in my barely legible scrawl captured by JotNot:

Evernote search

Even though this all seems pretty cool, I never really used it very much. Part of the problem was that I didn't seem to be putting very much into it. If you've only got six notes, you don't need folders or tagging or clever search to find them.

A few weeks ago, Michael Mealling pointed out that one way to fix the data problem was to scan documents directly into Evernote. That turned out to be an excellent suggestion.

For example, I've had a manilla folder of articles torn from magazines in a drawer in my desk for ages. The threshold for getting torn out and shoved in that folder was pretty high because, let's be honest, if I put too many things in that folder, I'll neither remember what's there nor be able to find it if I do remember.

Twenty minutes with a scanner and suddenly I've got something that I can tag, categorize, and search painlessly. What's more it's something I have with me on my laptop or my phone or anywhere I can get to a web browser: it's not in a folder in a drawer thousands of miles away (as my desk is this week, for example).

Of course, then I had a much bigger problem. I am very, very reluctant to put my data in your application if I can't get it back again. Repeat after me: no roach motels. I wish Evernote every success in the world (I'm even doing my part, to the tune of $45/year, to keep them around), but it's not hard to imagine a future in which I've come to rely on them as a repository of important information only to discover some Thursday afternoon that they've gone away or lost all my data or otherwise left me high and dry.

To their credit and my relief, they have an API. It's not the sort of RESTful Web API I've come to expect from these sorts of services, but that's ok. It's published and documented and claims to support Java, Perl, PHP, Python and Ruby out of the box.

A few short hours of hacking and a few hundred lines of Python and I had a backup tool that gets back everything I put into Evernote and an XML representation of the search data. What more could I ask for?

(If you're interested in trying my backup script, you'll need to go through Evernote support to get your own API key and configure a few lines to indicate where you want the files stored, but after that it should work for you too. YMMV, of course.)

My script creates an XML representation (natch!) of the details returned by the Evernote APIs. Eventually, I'll probably decide to fix things so it doesn't create a single potentially enormous file, but it doesn't much matter to me at the moment because I turn around and drop this into the MarkLogic Server instance that I use for my local PIM data.

Now, instead of collecting only the very most interesting articles I see and losing them in a folder, I collect anything that interests me even slightly, confident in the knowledge that I'll always be able to find it, and everything else, with ease. Alongside those articles, you'll find scanned business cards, recipes, specifications, notes, photographs of whiteboards and napkins, receipts, and a host of other interesting words and ideas that I've captured as the universe pushed them past me.

Pretty sweet.

Call for Vote - DocBook V5.0

2009-10-19T19:02:35Z

DocBook V5.0 is ready to become an OASIS Standard!

It's been a long, and sometimes winding, path from DocBook V4.5 to DocBook V5.0, but we're finally approaching the finish line to make it official. The OASIS Call for Votes went out on Friday!

The ballot is open now through 31 October, 2009.

If you belong to an OASIS Member organization, please encourage your representative to vote “Yes”. (The full list of eligible voting members is included on the ballot).

DocBook V5.0 is a significant milestone in the evolution of the standard. Based on RELAX NG and designed with extensibility and flexibility in mind, making DocBook V5.0 an OASIS Standard will provide the solid foundation that we need to continue improving The Source for Documentation™.

(We really do have some cool stuff coming down the pipe, so as they say in Chicago, vote early and vote often!)

Micro-blogging Backup, part the fifth

2009-10-18T20:14:49Z

In which we clean things up.

If you've been using one of the micro-blogging services for a while, you're probably familiar with a set of conventions that have evolved for adding metadata to your status messages. The ones I'm familiar with are:

@user to identify another user.
#tag to add a “tag” to your message.
!group to identify a group (at the moment, this seems only to be an Identi.ca convention).

In addition to those conventions, the use of “URL shorteners” (http://tinyurl.com/, http://bit.ly/, http://is.gd/, etc.) is common. And, finally, although it may not be apparent in the client you use, at the API level, individual status messages may indicate that they are “in-reply-to” some other message.

So far, our micro-blogging backup system doesn't take advantage of any of this extra information.

One of the first things I decided to do was expand shortened URIs. There's no 140 character limit in the database and if you link to something on http://youtube.com/, the odds that I want to follow that link are within ε of zero. I'd like to know before I click.

As long as we're grovelling through the text of each message, it makes sense to expand the other conventions, turning them into the appropriate links.

It also makes sense to download any messages that an existing message is “in-reply-to”. If those messages are also replies, we'll follow them too until the trail ends. This allows us to display whole conversations, even if they involve participants that we don't follow.

All of this can be accomplished with one new module, /modules/cleanup.xqy, and a new top-level query to drive it, /clean-tweets.xqy. The interesting bits are in the cleanup.xqy module:

The actual work is just string manipulation: regular expressions and tokenize, mostly.
Following replies counts against your rate-limit, so we do at most 50 at a time.
To expand URIs, we perform HTTP HEAD requests against the URIs we find in the status messages. In the worst case, some of those may timeout, so we do at most 500 at a time. That way we're unlikely to perform a query that takes so long that it times out.
If you look closely, you'll see that in addition to doing the expansions, we also add new elements to the status document: for mentions of another user, for tags, for groups, and for the host names of expanded URIs.

We'll come back in some future installment and use those for faceted searches (e.g., “find all the messages by @xmlcalabash that include links to tests.xproc.org”).

Pop the two files mentioned above into your setup (if this is your first encounter with my micro-blogging backup series, make sure you start at the beginning).

After you've installed the files, running http://localhost:8330/clean-tweets.xqy will start cleaning up your database. If you've downloaded a lot of messages, you'll have to run it several times. If you have a lot of replies, you'll have to spread the runs over a few hours.

The fact that you sometimes have to run the cleanup scripts several times is a bit inconvenient. I'm experimenting with some JavaScript to improve that, but I'm still looking for better solutions.

Built my own...

2009-10-16T01:31:15Z

Another geekdom right of passage: builing my own box.

As I noted back in June, I've always wanted to build my own box. Now I have.

I don't, in retrospect, claim to have learned what “the right answer” is, but here's the answer I arrived at:

AMD Phenom II X4 955 Black Edition CPU

AMD Phenom II X4 Processor Black Edition

I was more than a little amused at the discrepancy between the size of the CPU and the size of the aluminum and copper monstrosity that sits on top of it to keep it cool:

Cooling apparatus for the CPU
ASUS M4A78T-E AM3 790GX HDMI ATX AMD motherboard

Motherboard in place
4 x G.Skill 2GB 240-pin DDR3 1600 memory

Four of 8Gb
4 x WD Caviar Green 1TB SATA hard drives

Gobs of space
LG Black 8x BD-ROM 16x DVD-ROM 40x CD-ROM SATA LightScribe burner

Optical drive
APEVIA ATX-AS680W-BL 680W ATX12V/EPS12V SLI power supply
XCLIO Windtunnel ATX Full Tower case

Case with power supply in place

I was pleasantly surprised how easy it was to assemble. The documentation was clear and complete, the parts all well labelled; it went together without a hitch.

All systems go!

Glow in the dark

(For a few more pictures, see Custom Build 2009 on Flickr.)

I decided to run bleeding-edge Ubuntu 9.10 Server beta on it. It's not that I wouldn't like to try Solaris and ZFS, but…the days are short and the list of projects is long. Installing Ubuntu is just that little bit easier. Ubuntu 9.10 installed flawlessly.

The ASUS motherboard includes a video controller and assorted other controllers. It claims to do RAID, but a little investigation reveals that it's fakeraid so I abandoned it. Instead, I setup software RAID. The first disk is the boot disk, the remaining three provide 2TB of disk in a RAID5 configuration.

The case, for all those monstrous fans, is pretty quiet, but not silent. I still think the server might get relocated to the basement, except that I worry about the higher humidity.

Anyway, MarkLogic Server is up-and-running and as soon as I get my Micro-blogging backup configuration ported over, I promise I'll write that next installment…

XML Calabash 0.9.15

2009-10-06T01:04:11Z

A new release at last. New features, fewer bugs, and test suite clean again.

Speaking of work (letting the non sequitors pile up), when I wasn't on the beach, I was hacking XML Calabash. I always update the project status page, but in case that's too subtle, perhaps this essay will catch your attention. (You're reading this, so I guess that should be “caught” without the “perhaps”, but nevermind.)

Next in the task queue: supporting Saxon 9.2. I fear this is going to be a bit of a b*tch as I reach down into the guts of Saxon in a few places. Fair warning: I don't plan to attempt to support previous versions of Saxon after I make this switch.

XML Calabash Logging

One of the changes in 0.9.15 is the switch from my own crufty, home grown logging infrastructure to the Java core logging facilities. On the plus side: this gives you a lot more control over the logging. On the minus: you have to deal with the core logging facilities.

Near as I can tell, this has to be done with a properties file. Here's what I do: I set the system property java.util.logging.config.file to point to my own logging configuration file, /Users/ndw/java/logging.properties.

My logging properties file, constructed mostly through trial and error, is shown below. The important bits are:

There are two places to control the amount of detail in the logs. By default, the console handler won't print any messages more detailed than “INFO”. If you want to get more detail, you have to turn this knob appropriately.

I really dislike the default message format, so I wrote a formatter that produces slightly more compact output. You might like it too.

This is the other place where you can control the amount of detail. “ALL” gives you all the messages. You might prefer “SEVERE”, which gives you only the fatal errors.

You can be selective here. If, for some reason, you want to see the gory detail of the XInclude step, but ignore everything else, you could set:

com.xmlcalabash.level=SEVERE
com.xmlcalabash.XInclude=FINEST

I'll try to provide better documentation for the available names.

Here's my current logging properties file:

############################################################
# Logging Configuration File
#
# You can use a different file by specifying a filename
# with the java.util.logging.config.file system property.  
# For example java -Djava.util.logging.config.file=myfile
############################################################

############################################################
#       Global properties
############################################################

# "handlers" specifies a comma separated list of log Handler 
# classes.  These handlers will be installed during VM startup.
# Note that these classes must be on the system classpath.
# By default we only configure a ConsoleHandler, which will only
# show messages at the INFO and above levels.
handlers=java.util.logging.ConsoleHandler

# To also add the FileHandler, use the following line instead.
#handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler

# Default global logging level.
# This specifies which kinds of events are logged across
# all loggers.  For any given facility this global level
# can be overriden by a facility specific level
# Note that the ConsoleHandler also has a separate level
# setting to limit messages printed to the console.
.level = ALL

############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################

# default file output is in user's home directory.
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter

# Limit the message that are printed on the console to INFO and above.
java.util.logging.ConsoleHandler.level=FINE
java.util.logging.ConsoleHandler.formatter=com.xmlcalabash.util.LogFormatter

############################################################
# Facility specific properties.
# Provides extra control for each logger.
############################################################

com.xmlcalabash.level=ALL

I'm fully open to suggestions for better approaches.

XML Summer School ’09

2009-10-05T23:36:02Z

Open source and web technologies at XML Summer School.

Speaking of travelling, a bit of a non sequitor if you aren't reading these essays sequentially, I had just returned from XML Summer School the day before my Caribbean jaunt.

I was delighted to see the return of XML Summer School (September is summer somewhere, I'm sure) and even more delighted to be invited to speak.

Radcliffe Camera

I taught in the “Open Source” and “Web 2.0” tracks, presenting Open Source Application Development and Building Dynamic Web Applications with XML, respectively. I think the sessions went well. (After more than ten years of doing it, I have finally (and suddenly, this year) reached the point where it doesn't make me gut knotting nervous to give presentations. That has to have improved my stage presence.)

Oxford sunset

Beyond the specific courses, what sets Summer School apart is the staggering array of talent lined up to teach. If you've got a question or a problem about something even tangentially related to markup, there's someone at XML Summer School who's thought about it, built it, or deployed it. Probably all three. That we're all good friends, engaging students not just in the classroom but also punting and pub crawling, is just gravy for everyone.

Punters

A bargain at twice the price, I promise.

After Summer School, I snagged a weekend with my folks.

Got seeds?

A relaxing end to a great week not without hard work and long days.

Dominican Republic

2009-10-05T22:51:30Z

A long weekend in the Dominican Republic brings me to country number 15.

The Dominican Republic is country number fifteen for me. Deb had to work, and I did a little work too, but mostly I was free to enjoy the sun and the sea and the sand.

Punta Cana Beach

We don't live in the Caribbean, remind me why.

Moon Palace Pool

Footprints

The Moon Palace resort was lovely (in a big “cruise ship on land” sort of way), and this is not a criticism, just an observation, but clearly there are no OSHA inspectors in the Dominican Republic.

OSHA? What's that?

Or maybe there are no personal injury lawyers, I don't know. The tripping hazard in our room was not the only structural feature that made me do a double take.

SQL to XML

2009-09-26T12:01:23Z

A number of Mac applications store information in SQLite databases. Step one to do something useful with that data is to get it into XML.

I hate having my data squirreled away in proprietary or quasi-proprietary ways. If I can't get my information back out of an app, I'd rather not use it. When I started using the Mac, I switched to iCal and AddressBook: both can export data in standard text formats which I can easily convert to XML.

But exporting the data is a manual process (though I could probably automate it with some clever AppleScript or something, I've never tried). I build a number of views of my address book and calendar data automatically so manual processes don't fit well into my workflow.

It didn't take too long to figure out where iCal stores my appointments or how to pull them together. Having worked out where iCal stores my appointments, I turned my attention to AddressBook.

Long story short: the address data is saved in a database in ~/Library/Application Support/AddressBook/AddressBook-v22.abcddb. After installing the sqlite3 application from MacPorts, I was able to extract a text dump. So far so good.

Here, for example, is a table definition and the first row of data in that table:

CREATE TABLE ZABCDRECORD ( Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER,
Z_OPT INTEGER, ZDISPLAYFLAGS INTEGER, ZMODIFICATIONDATEYEAR INTEGER,
ZCREATIONDATEYEAR INTEGER, ZADDRESSBOOKSOURCE INTEGER, ZISALL INTEGER,
ZME INTEGER, Z19_ME INTEGER, ZINFO INTEGER, ZBIRTHDAYYEAR INTEGER,
ZPRIVACYFLAGS INTEGER, ZNOTE INTEGER, ZADDRESSBOOKSOURCE1 INTEGER,
ZCONTACTINDEX INTEGER, ZSOURCEWHERECONTACTISME INTEGER, ZVERSION
INTEGER, ZSYNCCOUNT INTEGER, ZSHARECOUNT INTEGER, ZADDRESSBOOKSOURCE2
INTEGER, ZMODIFICATIONDATE TIMESTAMP, ZCREATIONDATE TIMESTAMP,
ZMODIFICATIONDATEYEARLESS FLOAT, ZCREATIONDATEYEARLESS FLOAT,
ZBIRTHDAY TIMESTAMP, ZBIRTHDAYYEARLESS FLOAT, ZUNIQUEID VARCHAR, ZNAME
VARCHAR, ZNAMENORMALIZED VARCHAR, ZTMPREMOTELOCATION VARCHAR, ZNAME1
VARCHAR, ZREMOTELOCATION VARCHAR, ZSERIALNUMBER VARCHAR, ZSUFFIX
VARCHAR, ZTITLE VARCHAR, ZTMPHOMEPAGE VARCHAR, ZNICKNAME VARCHAR,
ZORGANIZATION VARCHAR, ZMAIDENNAME VARCHAR, ZIDENTITYUNIQUEID VARCHAR,
ZPHONETICFIRSTNAME VARCHAR, ZDEPARTMENT VARCHAR, ZPHONETICLASTNAME
VARCHAR, ZMIDDLENAME VARCHAR, ZFIRSTNAME VARCHAR, ZIMAGEREFERENCE
VARCHAR, ZJOBTITLE VARCHAR, ZPHONETICMIDDLENAME VARCHAR, ZLASTNAME
VARCHAR, ZSORTINGFIRSTNAME VARCHAR, ZSORTINGLASTNAME VARCHAR,
ZCREATEDVERSION VARCHAR, ZLASTDOTMACACCOUNT VARCHAR, ZLASTSAVEDVERSION
VARCHAR, ZSYNCANCHOR VARCHAR, ZSEARCHELEMENTDATA BLOB,
ZMODIFIEDUNIQUEIDSDATA BLOB );
INSERT INTO "ZABCDRECORD" VALUES(1,18,287,NULL,NULL,2008,NULL,1,NULL,NULL,
3,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,
239119683.670433,NULL,18281283.670433,NULL,NULL,
'93973926-7EF6-40F0-ADBD-8C7BBFC30FA1:ABSubscriptionRecord',NULL,
NULL,NULL,NULL,'local','B7303AAD-DA79-46E6-BC7D-91DAD82AEFB8',
NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,
NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL);

Next, I wrote 150 or so lines of Perl to convert the text into XML.

The XML is designed to be a totally straightforward representation of the table structure of the database. From the preceding SQL statements, sqltoxml produces:


  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

  
    1
    18
    287
    2008
    1
    3
    239119683.670433
    18281283.670433
    93973926-7EF6-40F0-ADBD-8C7BBFC30FA1:ABSubscriptionRecord
    local
    B7303AAD-DA79-46E6-BC7D-91DAD82AEFB8

As you can see, I've made no effort to maintain some aspects of the database (like the primary key), I've simply dropped NULL fields, and I'm relying on the field names to be valid XML NCNames. There are clearly other, equally reasonable, design choices that I could have made.

The resulting XML is the bare minimum needed to switch to XML tools for subsequent downstream processing (turning address book tables into VCards, for example). But it gets the job done.

And maybe it'll come in handy for someone else.

RDFa for DocBook?

2009-09-22T13:20:39Z

Adding RDFa to DocBook would make it possible to add a class of semantic annotations to DocBook without changing the schema. But is that a good idea?

Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information on it.

— Samuel Johnson

When Bob DuCharme introduced the semantic web track at XML Summer School this morning, he mentioned briefly the idea of adding RDFa to vocabularies other than (X)HTML. In particular, he's investigated how to do it in DocBook.

The DocBook TC gets periodic requests to add new inline elements and attributes for bits of metadata. Sometimes the requests are entirely legitimate, in the sense that they're clearly about technical documentation, but seem to apply to such a small audience that the TC is reluctant to add them to all of DocBook.

With this in mind, the idea of adding RDFa has some appeal: we add a few new attributes and henceforth users will be able to add new bits of metadata without having to change the DocBook schema.

But I'm not sure.

First, lots of DocBook elements have more discrete semantics than HTML elements. We don't need to say

Beautiful Sunset

because we have . We don't need to say:

because we have


  
    Some Name
    http://mypubco.com/empid/53234

I'm not suggesting those are exactly the same, they're clearly not, but I'm comfortable that existing DocBook elements are sufficient for the task.

(Yes, you'd need a DocBook-specific tool to extract the metadata, which is a disadvantage, but you probably want one anyway for the existing DocBook semantics.)

Second, it would allow you to construct statements with conflicting or, at best, odd semantics:


  Alice1
  This is from section 2.2.

I can just about imagine a sense in which “Alice1” can be both the title of a section and the Dublin Core creator of the section, but it doesn't make a lot of sense.

Third, Bob's example seems to suggest that it would encourage markup like this:


  The trouble with Bob2
  Alice2

which seems like a bad idea to me.

On the other hand, some of the examples do seem useful for exactly the sort of thing I suggested motivated my interest:

In fairness, Bob set out to recreate the triples from the original tutorial, so some of the markup choices were forced upon him.

So I'm not sure.

Micro-blogging Backup, part the fourth

2009-09-09T15:11:49Z

In which we get to see what our tweets and ’dents look like.

If you haven't been following along, go back and read parts one, two, and for a little background, three first. Now you've got MarkLogic Server up and running and you've been able to download your tweets and the tweets of those you follow. (Tweets or ’dents depending on which microblogging service you prefer; either, actually both, work for me.)

Next, download mbb04.zip and unpack it in the same place where you unpacked mbb02.zip. If you were following the instructions, you've edited some of the files in the “mbb/inst” directory and you may have some sessions saved in CQ, so I have not included those directories in mbb04.zip.

If you've been tinkering with other files, then you want to unpack this zip with some care or you may overwrite your changes. But you won't overwrite changes made to the installation or CQ areas. (By the same token, this distribution is incomplete without mbb02.zip).

With that installed, you now have a CSS file, four modules, and a new “top level” script, show-tweets.xqy. That's the fun one. Point your browser at http://localhost:8330/show-tweets.xqy and you should be rewarded with a list of your status messages from today. (As before, adjust the port number as necessary if you installed the application server on a different port.)

If you don't have any status messages from today, load http://localhost:8330/get-tweets.xqy to download your most recent messages, then try http://localhost:8330/show-tweets.xqy again.

The show-tweets.xqy script accepts query parameters. You can pass:

sdate: To specify the starting date in “YYYY-MM-DD” format. If unspecified, defaults to the ending date.
edate: To specify the ending date. If unspecified, defaults to today.
users: To specify one or more users separated by spaces (+ signs or %20’s in URI-speak). These should match the screen_name values in your account configuration. The value “ALL” is special, it will list tweets for every user, including all your friends.
service: To specify the service. If you setup accounts on multiple services, this will let you limit the result to only those messages on a single service.

Go ahead and give it a try, http://localhost:8330/show-tweets.xqy?sdate=2009-08-01&edate=2009-08-31&users=ALL will show you all the messages by you and those you follow posted in the month of August.

Message Formatting

The messages are sorted in ascending order by date, so that messages read chronologically “down” the page as you'd expect. However, conversations are handled a little bit specially.

Whenever a message is encountered that is part of a conversation (either because it's a reply to another message, or another message exists that is in reply to it), the whole thread is collected together and presented as a unit, like this:

Message threading

I think that makes the results much easier to follow. We'll come back to what to do about threads involving users other than you or your followers later.

There were some bugs in the Identi.ca server that occasionally caused incorrect “in-reply-to” values to be inserted into the data. I think those have been fixed in the server, but there's nothing I can do about the values that are wrong.

URL Rewriting

Those show-tweets.xqy URLs may be sufficient, but they're hardly elegant. I'd be much happier if they were better organized for human consumption.

Luckily, with the URL rewriting features of MarkLogic Server V4.1, this is easily achieved. To begin with, go back into the admin console (http://localhost:8001/) and navigate down through Groups→Default→App Servers in the tree control then select the server you setup for this project.

Near the bottom of that page, you'll find a “url rewriter” field. Specify modules/url-rewriter.xqy as the value for that field and then click “Ok” at either the top or bottom of the page.

The one I provided supports a range of values designed to be more readable:

/my-tweets/ service / users / start-date / end-date: Returns the messages associated with “users” on “service” between the specified start and end dates, e.g., /my-tweets/identica/ndw/2009-08-01/2009-08-15.
/my-tweets/ users / start-date / end-date: Returns the messages associated with “users” on any service between the specified start and end dates, e.g., /my-tweets/ndw/2009-08-01/2009-08-15.
/my-tweets/ start-date / end-date: Returns the messages associated with any user on any service between the specified start and end dates, e.g., /my-tweets/2009-08-01/2009-08-15.
/my-tweets/ start-date: Returns the messages associated with any user on any service between the specified start date and today, e.g., /my-tweets/2009-09-01.
/my-tweets: Returns the messages associated with any user on any service posted today.
/all-tweets/ service / users / start-date / end-date: Returns the messages posted by anyone on “service” between the specified start and end dates.
/all-tweets/ start-date / end-date: Returns the messages posted by anyone on any service between the specified start and end dates.
/all-tweets/ service / start-date: Returns the messages posted by anyone on “service” between the specified start date and today.
/all-tweets/ start-date: Returns the messages posted by anyone on any service between the specified start date and today.

The URL rewriter is just an XQuery that you can change; so it can support any kind of values that you'd like. It recieves the URL that the user entered. Whatever URL it returns is what the server actually responds to. The one I've provided turns

/all-tweets/2009-08-15/2009-08-31

into

/show-tweets.xyq?users=ALL&sdate=2009-08-15&edate=2009-08-31

So nothing else has to change about the code, it all just works.

I'm not sure there's much else that's new or interesting about the other modules provided in this part. If you have any questions, feel free to ask.

Next time, we'll look at dealing with all those ugly shortened URLs and pesky replies that extend beyond our followers.

Micro-blogging Backup, part the third

2009-09-03T20:12:40Z

In which we peel back the covers on what's been built so far.

There's more functionality to come, but first, I thought it might be useful to spend a few minutes looking at what we've got so far.

The setup code in /mbb/init isn't very interesting, and I'm not going to attempt to explain how CQ works, so we'll begin in the /mbb/modules directory.

accounts.xqy

This module contains some utility and convenience functions for dealing with account data. I changed my mind about how to store the data a couple of times early on, so these functions were supposed to protect me a little bit from that. I didn't follow through all the way so the account abstraction is pretty leaky, but I left this module in place anyway.

twitter.xqy

This module is a thin skin over the actual Twitter API. Ideally, I'd flesh this module out to support the rest of the endpoints, but I haven't bothered yet.

One school of thought on this kind of API module is that it should be as thin as possible, providing only the thinest skin over the underlying API. I mostly agree, but I did take a few liberties. If you wanted to adapt this module for some other purpose, you might have reason to carve it a little closer to the bone.

One decision I made was to have the account/rate_limit_status method return the number of calls remaining directly as a number, rather than returning the XML response. That's pretty simple. The other changes I made are a bit deeper.

The Twitter timeline methods are designed to be “paged”; the caller can select the page size and the page they want to retreive. I decided that what I really want is all the pages; so my versions of the timeline methods always request all the pages and return all of the results in a single call (by performing the requisite paging for you, behind the scenes). Twitter limits you to 16 pages, but Identi.ca servers seem to offer more pages. In order to avoid recursing beyond the size of the call stack, I placed an arbitrary limit on the number of pages.

Finally, I decided to protect the caller from exceptions that can occur if the underlying HTTP requests fail. Most of the public methods in twitter.xqy return an element, either the Twitter API response, or a element containing the HTTP error code if an error occurred.

I think an argument could be made for not doing this, for letting the lowest-level API calls throw the exception, but I decided not to. You're free to change that, of course.

twitproc.xqy

This module is mostly responsible for taking Twitter and elements and inserting them into the database. Along the way, we transform them just a little:

I move them from no namespace into the “t:” namespace. First, I subscribe to the position that XML vocabularies should place elements in a namespace. I'm aware that there are people who believe otherwise. They're wrong. Second, XQuery’s interpretation of unqualified names exacerbates the problem. So you could look at this as patching a bug in the Twitter API.
I transform the contents of the element into ISO 8601 format (so it fits more naturally into the data model).
The Twitter APIs return a element embedded in each message. This is probably a net win for limiting round-trip calls to the API, but it doesn't strike me as a very sensible way to store things in the database. I break out the users and store them separately.
I add a few more elments to each status message. These record information about subsequent processing to perform (more about that later), the screen name of the user who uttered the message, and information about who was logged in to retreive this message.

This is a little lazy on my part. Arguably, I should introduce another namespace for these additional elements (so that some future Twitter API change doesn't walk all over them), or maybe not store them in the messages at all. I invite you to fix it if it bothers you.

If you know something about MarkLogic Server, this may sound like a job for document properties. That's a good idea, particularly for the downstream processing markers. However, document properties are associated, as the name suggests, with documents in the database. Later on in the code for displaying messages, we're sometimes going to make a copy of the message (giving it a new parent element). Doing that breaks the association with document properties. I was trying to keep things simple, so I didn't use properties for one set of information and child nodes for another, I just pushed it all into child nodes. My bad.

update.xqy

This module wraps up the functionality of the twitter.xqy and twitproc.xqy modules, getting all the tweets for a user and inserting them into the database. The code for finding the most recent messages by (and not by) a particular user might be interesting to you. Ignore the $tweet-collection variable; it's a holdover from an earlier approach, no longer used.

get-new-tweets.xqy

This module exists only to be invoked from another module. It declares an external variable that identifies a single account then simply calls the get-tweets function from the update.xqy module for that account.

The last bit of code that we've got so far is get-tweets.xqy in the top level of the application server. This module loops over all the accounts that we've defined and, for each one, downloads and inserts any new status messages into the database. It does this by invoking the get-new-tweets.xqy module.

What's all this invoking stuff about?

The server takes a completely safe, transactional approach to database updates. You are guaranteed that every query that updates the database either succeeds in its entirety or fails. One of the things that you aren't allowed to do is make conflicting updates to the same document in the same transaction. You can demonstrate this easily, just run the following expression in CQ:

let $doc := some document
return
  (xdmp:document-insert("/scratch/foo", $doc),
   xdmp:document-insert("/scratch/foo", $doc))

The server will bark “XDMP-CONFLICTINGUPDATES” and no inserts will be made to the database.

Why does this matter to us? Well, imagine that you setup two Twitter accounts in our micro-blogging backup system. Imagine further that both of those accounts follow marklogic.

What's going to happen when we run the backup? Both accounts are going to download all of the status messages on their “friends” timeline, so they're both going to download all of the recent marklogic tweets. And they're both going to try to insert them into the database. And that's going to generate a “conflicting updates” error.

Using the two-step xdmp:invoke dance as shown in get-tweets.xqy and get-new-tweets.xqy avoids this problem. The semantics of xdmp:invoke are that it runs the specified module in a separate transaction.

Since no single user is going to download the same message twice, each transaction will succeed. In fact, some messages will get updated twice in the database, but that doesn't do any harm because the content of the message will be the same in each case.

An alternate approach to this problem is to manage the messages with greater care, identifying duplicates when they occur and not attempting to insert them in the database. This is the approach taken in twitproc.xqy for the simpler problem of dealing with duplicate s.

It would certainly be possible to refactor the code so that the xdmp:invoke call could be avoided, but in this case splitting work into several transactions feels like the more elegant solution. And any performance penalties associated with a few calls to xdmp:invoke are going to be totally swamped by the latency in the underlying HTTP requests, so there isn't really a downside.

What next?

In the next part, we'll push a little further forward, getting some code in place to display the messages we've downloaded. We'll also look at the subsequent processing I hinted at. Further down the road, we'll look at search, and then we'll add some JavaScript , refactor things a bit, and make an AJAXy/Web 2.0 UI for our application.

I hope you're enjoying the ride.

XML+XQuery+Google Voice+Python=WIN!

2009-09-01T13:45:54Z

It's finally possible to put all the pieces together.

I've been storing my PIM data (contacts, appointments, etc.) in XML for ages (and I do mean ages !). Since my Palm days, I've been translating whatever native format my PIM supports into XML. Where necessary (i.e. everywhere), I've used an RDF/N3-like annotation mechanism to support additional metadata.

For example, the “notes” field for the XProc telcon looks like this:

rdf:
p:class telcon
p:access public
p:phone #w3c-zakim
p:code 97762#

That gets parsed into the obvious XML by the conversion process:

telcon
public
#w3c-zakim
97762#

(The phone number “#w3c-zakim” means that there's an entry in my address book with the ID “w3c-zakim”. Why do none of the PIM applications understand that appointments and contacts are related!?)

I've been doing this for years because it's The Right Thing™, even though I've only been able to wring small (er, tiny, perhaps miniscule) amounts of practical value from it.

But no more!

The XML data is stored in my own MarkLogic Server instance (moving from a collection of Perl hacks to the server was one of my first personal projects after I joined Mark Logic). I now have a Google Voice number. And Scott Hillman ’s Python scripts finally let me connect all the dots^[1]!

A little Python hacking, a quick XQuery module, and I can make calls from a shell window.

$ call xproc
W3C XProc WG
+1-617-761-6200 97762#

Dialing +1-617-761-6200...

The call query searches both the appointments on today's calendar and the address book. That means “call seth” works equally well, and calls my boss. For contacts with more than one phone number, I can add “- phone” on the end: “call ndw@nwalsh.com -home” would call me at home, should I ever want to do that.

It's a tiny little thing, but it feels great.

I'm easily amused, I know.

Hmm. I should add an option to send SMS messages, too…

^[1]Well, technically, reconnect, but for all it's coolness, I never got much use out of the DTMF auto dialer.

Micro-blogging Backup, part the second

2009-08-28T18:16:44Z

In which we setup the database one screen at a time and then import our first status messages.

If you were following along yesterday, you've got MarkLogic Server up and running with the Community License. Now it's time to start putting it to work. (Cutting toothpicks with a chainsaw, but hey, you have to start somewhere.)

Well, almost. First, we have to do a little setup.

Download mbb02.zip and unpack it somewhere convenient. I choose /home/ndw/mbb for the purposes of this example, but I'm not sure your home directory is really the best place. Anywhere you'd like though, doesn't matter to me.

Fire up your favorite web browser and connect to the admin interface on port 8001 (http://localhost:8001/, probably); you'll need to login with whatever userid/password combination you selected at installation time.

Once you're there, click “Forests” in the “Configure” tree control in the left hand column and then select the “Create” tab. Enter any name you'd like for the forest and click “ok”. I named mine “mbb”.

Create a forest

Forests are where the server stores XML documents. Trees, as it were. Clever, eh?

Next, choose “Databases” in the tree control and select the “Create” tab again. Enter any name you'd like for the database and click “ok”. I named mine “mbb”. I can't think of a compelling reason to give them different names, but suit yourself.

Create a database

Once you've created a database, you'll be reminded that you need to attach a forest to the database.

You must attach a forest to the database

Click on that link and do so. Remember to click “ok”.

Attach the forest you created

Almost there. Choose “Groups”, “Default”, and “App Servers” in the tree control, then select the “Create HTTP” tab. Enter any name you'd like for the server name, I named mine “mbb”; enter the location where you unpacked the zip file for the root, I used /home/ndw/mbb; and enter an open port value for the port, I used “8330”.

But don't click “ok” just yet. (If you already did, no worries, just click on the app server's name in the tree control.)

Create an HTTP application server

Scroll about half way down the page to change the authentication and default user. Select “application level” for the authentication scheme and “admin” for the default user.

Change the authentication to application-level

This gives your application complete access to the server without having to login. There are lots of ways to make an application more secure, but let's leave all the security knobs for another day. Now scroll to the top or bottom and click “ok”.

At this point, you have a real honest-to-goodness application running on your server. (And yeah, this should all be simpler and easier. I've heard tell of plans to improve it, but nothing I can swear to.)

I included a copy of “CQ”, a browser-based, interactive XQuery environment in the distribution. You can see it if you navigate your browser to http://localhost:8330/cq. (In this and all the following examples, if you chose a different port, use the port number you chose.)

If you click on the “explore” link at the top of the CQ page, you'll see that you've got an empty database.

CQ shows the empty database

Now it's time to configure this particular database for our micro-blogging backup application. Later on, we're going to need some indexes. You could walk through the admin UI to create them, but that's tedious, you only have to do this once, and the admin UI is completely scriptable, so I created a little query to do the grunt work.

Point your web browser at the database configuration script: http://localhost:8330/init/setup-database.xqy. If everything is setup correctly, you'll quickly get a “database configured” message.

Configure the database

Next, we need to configure the microblogging accounts that you want to backup. Like database configuration, you're probably only going to do this once (or at least once in a great while), so I didn't create any sort of UI for it.

In the directory where you unpacked mbb02.zip, open up init/setup-accounts.xqy with your favorite text editor. On lines 57 and 58 replace SCREEN_NAME and PASSWORD with the Twitter username and password that you want to backup.

If you're using Identi.ca instead, you'll have to do a little more editing, but it should be pretty straightfoward. If you're using your own install of the Laconica software, or you're using some other microblogging server, as long as it supports the Twitter API, you should be able to figure out what to do. Feel free to ask if you're not sure.

When you've got all your accounts in place, save the file and point your web browser at it: http://localhost:8330/init/setup-accounts.xqy.

Configure your accounts

If all goes well, you'll get an appropriate “Accounts initialized” message. If you get 500 errors, you messed up the XQuery syntax somewhere. It won't do any harm to run the setup account script more than once, so try making small changes, running the script after each change. If you get stuck, let me know.

If you go back to CQ again and click the “explore” link, you'll see that there are documents in the database now, one for each account you added.

CQ shows the database with one document

Now we're ready to really do something.

Point your web browser at http://localhost:8330/get-tweets.xqy to download your status messages. This may take a while, especially the first time and especially if you entered several accounts.

Download the status messages for your account(s)

If you get a message about “rate limit exceeded”, it means you've done too many interactions with the Twitter API this hour. Wait a bit and try again. Twitter threatens that they'll turn off your account if you flagrantly violate the rate limit, so the MBB queries are pretty careful not to.

The “explore” link in CQ will now show a whole bunch of documents in the database.

CQ shows a database full of documents

You can enter any arbitrary XQuery expressions you'd like into CQ. Here I've asked for a count of all the messages that I've “favorited”.

Arbitrary XQuery expressions evaluated by CQ

In the next parts, we'll look at some of the code behind this functionality in a little more detail, add some XQuery to display the messages, look at how we can augment the messages in useful ways, add searching, and finally pull the pieces together into a useful little app. Well, a little app I think is useful, anyway.

What about my older messages?

Twitter only lets you get at the last 3,200 or so status messages with the Twitter API. If you've got older status messages that you've already backed up, or if you can find some other API to get at them, there are other ways to get them in the database.

I left the skeleton of one of those ways in the init directory, an XProc pipeline that XML Calabash can run to load status messages from existing XML files.

If you've got your old tweets archived in XML, drop me a line and I'll try to point you in the right direction.

Micro-blogging Backup, part the first

2009-08-27T13:23:47Z

What started out as a trivial exercise in backing up my Twitter and Identi.ca posts turned into a little microcosm of XML Server application development. It's something you can deploy for free on your very own MarkLogic Server!

This is the story of the intersection of two ideas:

First, almost no one that I spoke to at Balisage had heard of the Community License for MarkLogic Server, and those few who had thought that it was still limited to just 100Mb of content.

The fact that you can download and play with the best XML server on the planet is something more people should know about! The community license is for non-commercial use only but it's free and it never expires. The previous 100Mb content limit has been upped to 10Gb so there's a lot more room in the sandbox now.
Second, at about the same time, there was a little spike of interest in backing up microblogging data, the status messages that you send to services like Twitter or Identi.ca.

Sturgeon's law applies, of course, to microblogging. And Sturgeon was an optimist. But there's still a lot of useful information out there and I don't want it to disappear under the waves just because some acquisition occurs and the terms of service shift under my feet.

Luckily, the APIs for getting your microblogging content return XML and I have an XML server, so… my first thought was to download the tweets (a “for” loop in Bash and wget will do the trick) and store them in the server. Then I thought, that's silly, the server can download them for me…

From there, my little ten minute exercise grew until I had a (still relatively small) appication that handles oodles of documents from multiple services and accounts, has threaded conversations and account merging, uses indexes, has full-text and faceted search, employs web APIs, uses URI rewriting, and even has some AJAX.

And because status messages are small, it'll run for ages under the community license.

My plan, therefore, is to spin this out over a few essays, building the app from its barest bones to something I'm finding quite useful. If you want to play along, the first step is to go get a copy of MarkLogic Server and install it with the community license. The steps are roughly these:

Download version 4.1 of the server. It runs on Windows, Linux, and Sparc boxes. (It isn't, alas, available for OS X, but it runs just fine under virtualization.)

It also runs just fine on the Debian flavors of Linux, though that's not an officially supported platform. Just make sure you have the bugfixed version of lsb-base.
After it's installed and running, point your web browser at http://localhost:8001/ on the machine where you installed it.
Click on the “free” license button, choose the community license, and click your way through the rest of the install screens.

Congratulations! You know have the most powerful XML chainsaw imaginable at your fingertips. Exactly what to do with it is the subject of part the second and beyond.

Using XML Catalogs and XProc together

2009-07-22T20:15:27Z

XML Calabash, my implementation of XProc, is my go-to tool these days for manipulating XML documents. Adding XML Catalogs into the mix just makes it sweeter.

Recently, I was presented with several hundred books comprised of many thousands of chapters. My goal: load them into the server so that they could become part of a larger application. Easy peasy.

Two snags: all the chapters contained references to named entities declared in an external subset and none of the metadata in each file was actually reliable.

Still pretty straight-forward. Parse the document to expand the entity references, do a little cleanup, and push them into the database. The details of the pipeline aren't that important, the bit I want to highlight today is the parsing.

Everything remained pretty easy until I discovered that there were a half-dozen or more flavors of DTD in use across this corpus. And naturally, every external subset was referenced only by a system identifier with some random, absolute path:

Where “10” was “10”, “11”, “21”, “25”, etc. for some substantial enough set of versions to be go well beyond my limit for tedium.

Luckily, all of them were including a standard suite of ISO entities and (as far as I could easily tell), that's all the entity references ever were.

XML Catalogs to the rescue.

First, grab a recent version of the DTD and stick it somewhere local, then construct the following catalog:

Next, tell XML Calabash to use catalogs. You can do this from the command line, but I set it up in my configuration file, ~/.calabash:


  false

The first few lines just set some defaults I like, it's the last two that are relevant here. I tell XML Calabash to use my XML Resolver catalog implementation for entity and URI resolution.

Now my pipeline simply does The Right Thing™.

When the parser attempts to load the external subset, the catalog resolver returns the local DTD (because all the system identifiers end with “.dtd”). The step doesn't do validation by default, so the fact that some of the files aren't valid according to the particular version of the DTD that I have locally doesn't matter. The entities get expanded correctly. (If any of the documents had relied on other entities only present in a particular version of the DTD, that would have been an error, so I know I didn't miss any.) I do a couple of lightweight transformations on the resulting document and shove it into the database FTW!

Nothing earth shattering here, and not the only way to solve the problem, but one that looks like a nail to my particular hammer of choice at the moment.

Norman.Walsh.name

Creating a DocBook V5.0 DTD

Pipeline Notes

What about a better W3C XML Schema?

XProc Proposed Recommendation!

Wiki editing with XProc

Where am I?

What your drive knows, and what it doesn't

Demo Jam at XML Prague!

XML FTW!

GSD!

XML Prague 2010

NYMUG: Cloud deployment options

XProc: Back to Last Call

David Alfred Walsh

David Walsh, age 10

Bombay c. 1945

Browerville from the tundra, Mar 1960

Camping in Alaska, c. 1960

Sleeping, June 1970

Mom and dad, August 1961

Mom, dad, and I

Playing guitar

David Walsh, Sep 2009

'tis himself

NYMUG Summary

Support for https: and URI rewriting

Office Toolkits

Presentation

NYMUG: New York Mark Logic Users Group!

DocBook V5.0

Evernote

Evernote search

Call for Vote - DocBook V5.0

Micro-blogging Backup, part the fifth

Built my own...

AMD Phenom II X4 Processor Black Edition

Cooling apparatus for the CPU

Motherboard in place

Four of 8Gb

Gobs of space

Optical drive

Case with power supply in place

All systems go!

Glow in the dark

XML Calabash 0.9.15

XML Calabash Logging

XML Summer School ’09

Radcliffe Camera

Oxford sunset

Punters

Got seeds?

Dominican Republic

Punta Cana Beach

Moon Palace Pool

Footprints

OSHA? What's that?

SQL to XML

RDFa for DocBook?

Micro-blogging Backup, part the fourth

Message Formatting

Message threading

URL Rewriting

Micro-blogging Backup, part the third

What's all this invoking stuff about?

What next?

XML+XQuery+Google Voice+Python=WIN!

Micro-blogging Backup, part the second

Create a forest

Create a database

You must attach a forest to the database

Attach the forest you created

Create an HTTP application server

Change the authentication to application-level

CQ shows the empty database

Configure the database

Configure your accounts

CQ shows the database with one document

Download the status messages for your account(s)

CQ shows a database full of documents