Norman.Walsh.name

XML FTW!

2010-01-25T22:21:37Z

On the serendipitous joy of finding XML.

As I've said before, I'm very reluctant to use your application if it's a roach motel for my data. It would not be fair to say that I'll refuse to use your application, it's just a lot less likely.

For example, when it came to GSD, I decided that open access wasn't as important as picking an application that I'd actually use. If I let myself get distracted by exploring APIs, there'd be other things not getting done! (Priorities!)

Having made my bed, I figured I should see what I was lying in. Today I took a peek at how OmniFocus stores data. Now, the title of this essay no doubt gives away the punch line, so consider for a moment how this would have been done in the time before XML.

…go on, have a think, I'll wait…

In my experience it would probably have been in some proprietary format, almost certainly binary, and utterly opaque. How many tools document(ed) their proprietary data formats? On some platforms, there might have been system services for storing data, some sort of platform-supported database perhaps. Those systems are (often) only marginally better. They produce, instead of an opaque stream of bits, an opaque stream of atomic values. (Don't get me wrong, I've done the reverse-engineering thing on binary formats, I'd prefer the stream of atomic values, believe you me.)

What did I find when I went looking at the OmniFocus data? A directory full of ZIP files. And what's in each ZIP file? Why contents.xml, of course!

Now, it would not be fair to assert that this is perfectly transparent. XML isn't magic. There are clearly some cross-reference relationships in there that will take a little mental gymnastics to decode. But still, I'll trade this:

...

  
  2010-01-21T16:23:08.983Z
  2010-01-24T21:01:41.632Z
  Add server-side support for multipart MIME to tests.xproc.org
  2113929216
  
  2010-01-27T22:00:00.000Z
  2010-01-24T21:01:41.622Z
  parallel

...

for anything I would ever have gotten at any other point in the history of file formats!

XML has its detractors. It would not be fair to say they are all wrong. But I'll take XML over fair any day!

GSD!

2010-01-25T15:13:26Z

Our engineering department has a project management philosophy they describe as GSD. I aspire to GSD.

For me, the part of GSD^[1] that I most often have difficulty with is keeping track of what needs doing. My todo (or want-todo) list is absurdly long. If I feel like castigating myself, I can always find a few things on my list that should have been done by now. It's not that I don't work hard or get a lot done, it's that I don't always prioritize perfectly and sometimes things slip through the cracks.

I've been trying to get better at this. Having an online calendar sync'd with my phone keeps me from accidentally missing meetings and phone calls, so it seems to follow that some sort of online system should be able to help me with my todo list.

My requirements are pretty simple: I want something that's easy to use and I want something that syncs with my mobile device. An online tool is almost, but not quite, as good as something that I can use offline on my PDA.

I don't subscribe to any particular Getting Things Done methodology. Maybe I'll get there someday, but that's not my immediate goal.

I played with Remember The Milk on-and-off last year. It seemed to work pretty well for simple lists, but I wasn't using it consistently because, I think, it wasn't quite powerful enough.

This month, I took a few different systems for a test drive: 2Do, Toodledo, Things, and OmniFocus.

Unfortunately, 2Do is only an iPhone app. It appears that there are plans for the next version to support syncing with Toodledo, but that doesn't exist today. Toodledo is a web-based app and is quite nice, probably plenty sufficient for my needs. On the desktop front, both Things and OmniFocus are probably plenty sufficient as well. (There are no doubt other similar applications, those are just the ones I happened to try. I didn't attempt an exhaustive survey, I've GStD!)

And the winner is: OmniFocus, by a narrow margin. I like the project/context duality that OmniFocus uses (ToodleDo has contexts too, if you turn them on). Mostly it boiled down to the UI: I liked the “feel” of OmniFocus best.

This is an app I plan to force myself to use, so I figured I'd best pick one that felt good. It's also the most expensive, by a pretty wide margin, but c'est la vie.

Will this really work for me? Time will tell. But so far, so good. And I'm already learning to use it in ways I hadn't planned: maintaining shopping lists and travel check lists. Those aren't the sorts of things for which I would have actively sought out software (sometimes a pencil and a piece of paper really is enough), but it's encouraging to me that I have other reasons to be paying attention to my GSD tool.

^[1]Getting Shi␈␈␈Stuff Done!

XML Prague 2010

2010-01-24T18:27:40Z

See you at XML Prague! And a chance to plug some really excellent training.

I'm delighted that my paper proposal for XML Prague was accepted. I'm a little less delighted that the final paper deadline was last week, but I guess that's encouragement to finish it, eh? (I will, I promise.)

I'm going to speak about modular documentation in DocBook, both about the proposal for “assemblies” being developed in the DocBook Technical Committee and about my XProc-based implementation.

I mention these things for only two reasons^[1]: first, to recommend that XML Prague is a conference you need to go to if you're interested in XML technologies. It's that good.

Second, to plug G. Ken Holman ’s XSLT/XPath 1.0 & 2.0 and XQuery 1.0 Hands-on Training class. If you're looking for someone to teach you XSLT and XQuery, you'd be hard pressed to do better than Ken. And if you're interested in XML, these are technologies you need to know. The maximum class size is an astonishing six, so it's practically 1:1. Yet another reason to be in Prague in March

See you there!

^[1]In the interest of full disclosure, I should point out that Mark Logic is a gold sponsor of the conference and Ken's course is being delivered in partnership with our own training services. I don't think that makes me baised, but I guess I wouldn't, would I?

NYMUG: Cloud deployment options

2010-01-24T18:16:47Z

Denise Miura, Sr. Director of Product Management will be speaking about Mark Logic's new offering for the Cloud at our upcoming User Group in New York this Wednesday.

The second Mark Logic New York User Group meeting will be held on Wednesday evening, 27 January 2010, hosted by Steve Kotrch from Simon & Schuster!

What: An opportunity to learn more about MarkLogic Server and collaborate with other MarkLogic users.
When: Wednesday, 27 January 2010, at 6:00pm EST.
Where: 1230 Avenue of the Americas, New York, NY between 48th and 49th streets on Sixth Avenue.
Who: Everyone who shows up, of course! The featured speaker this time is Denise Miura who will explain Mark Logic's new cloud deployment options and describe how they are being used today within the Mark Logic development community. She will demonstrate instantiating a MarkLogic AMI live on Amazon EC2 and talk about best practices for using MarkLogic on the EC2 platform. Finally a preview of the planned cloud-related product enhancements will be provided. This is a great opportunity to provide feedback and influence the cloud computing initiative at Mark Logic.
How: If you plan to attend, please join the mailing list and send your first and last name to cleo dot saab at marklogic dot com.

If you're in New York, please stop by (and please let Cleo know if you plan to stop by).

XProc: Back to Last Call

2009-12-28T14:20:52Z

Early in January, a new XProc draft will appear. It will be a Last Call Working Draft, a step backwards in the process, or maybe just a half-step. The reason is important though: versioning.

The XProc WG has been making steady progress on XProc: An XML Pipeline Language. We saw the start of wide adoption of XProc in 2009 and I think there's every reason to expect more of the same in 2010.

This makes it all the more disappointing to report that we're going back to Last Call. On a personal note, as a TAG alum, it's a bit embarrassing to admit why: versioning.

We received significant and persuasive criticism of our versioning story. In particular, we were pesuaded that requiring a processor to download the declarations for V.next steps in order to process them in a “forwards compatible” manner was too burdensome.

In redrafting the story, we added a version attribute, “compile-time” use-when functionality à la XSLT, and extension functions for more precisely identifying the environment in which the pipeline is running.

We also took the opportunity to fix decisions that were, in retrospect, mistakes, but not in-and-of themselves sufficient to motivate us to return to last call: we changed the rules for connections in option, parameter, and variable bindings so that an explicit isn't required when there's no default readable port, and we changed the rules for parameter input ports so that a binding isn't required when there's at least one explicit . Users are already thanking us.

I didn't get the document through the publication process before the end-of-year publishing moratorium, but you can read the staged draft, if you wish.

Will this really be our last Last Call? I sincerely hope so! I also hope that we can move directly from Last Call to Proposed Recommendation without the formality of another Candidate Recommendation phase in between. There's precedent for doing this, and we've got active implementors.

I'm sometimes frustrated by how long the process takes, but I console myself with the observation that the language is better for our efforts. Rising usage suggests the early adopters, at least, agree with us.

David Alfred Walsh

2009-12-26T20:38:58Z

9 June 1923 — 26 November 2009.

A man may by custom fortify himself against pain, shame, and suchlike accidents; but as to death, we can experience it but once, and are all apprentices when we come to it.

— Montaigne

My father was born in 1923 in Babylon, NY.

David Walsh, age 10

He survived the Great Depression. An enormous tree blew over next to him as he walked home through The Great Hurricane of 1938; he walked away without a scratch. The glider born infantry took him to the China-Burma-India theater in WWII.

Bombay c. 1945

Left to right: Charles Kuhn, York PA; Bill Bride, Beacon NY; Eddy Evans, Boston MA; David Walsh, Babylon, NY

Shrapnel chipped a tooth, but he survived that too. After the war he went to Alaska.

Browerville from the tundra, Mar 1960

My dad taught in Barrow and Nome. After putting out a chimney fire, he walked away from a two story fall off a frozen roof by the lucky stroke of landing feet-first on an oil drum.

Camping in Alaska, c. 1960

He single-handedly built a one-room cabin on a ¼ acre plot in Fairbanks. (I think I remember seeing once a photo showing the scaffolding he built to get the roof beam in place.) He worked for the Fish and Wildlife Service in the summers.

He used to practice orienteering by walking into the Alaskan wilderness on a compass bearing and then walking back out again. On one occasion he stumbled across a downed single-engine plane containing the skeleton of its pilot. His boss laughed when my dad offered to lead a team back to the crash, assuring him that he'd never find it again. Dad's boss was right. There is a lot of wilderness out there.

On another occasion, my dad shot a caribou only to discover as he prepared to dress it that he'd left his knife back in the jeep. Leaning his rifle against a tree, he walked back and got his knife. An enormous brown bear greeted his return by standing on its hind legs and roaring. The bear got the caribou. And the rifle. And the knife, dropped during a hasty retreat.

That wasn't the only caribou that nearly got him killed; on another occasion, one attempted, unsuccessfully, to jump over his jeep. He woke on the side of the road with a caribou hoof protruding into the cab and a nasty gash on his head.

I'm lucky to be here.

Sleeping, June 1970

When my dad left Alaska, he gave the keys to his cabin to a friend. Those keys passed from friend to friend for more than twenty years. In the eighties, the current occupant persuaded my dad to let him buy the cabin. My father signed the deed and mailed it, asking the occupant to please mail the check back. The check came back a couple of weeks later. And it cleared. Luck of the Irish, or something.

From Alaska, my dad traveled to Australia. My mom and dad met in Tasmania. They married in 1961.

Mom and dad, August 1961

I came along a few years later.

Mom, dad, and I

June 1968

I remember my dad singing sea shanties when I was a small boy.

Playing guitar

Time and place unknown

Dad was a naturalist, hunter, trapper, fisherman, scientist, teacher, draftsman, and surveyor. He made beautiful wood carvings. He tied knots. At one time or another, all of them. I have his leather working tools. The old sewing machine on which he made sleeping bags, tents, parkas, rain slickers, and bicycle paniers got lost somewhere along the way. He built two boats.

David Walsh, Sep 2009

After 86 years, entropy won. Entropy always wins. My dad taught me that. And the first and third laws as well.

My father died in 2009 in Norwich, England.

'tis himself

David Alfred Walsh 9 June 1923 - 26 November 2009

Goodbye, dad.

NYMUG Summary

2009-11-12T16:16:03Z

Last night, I spoke at the inaugural New York Mark Logic User Group meeting. I think it was a crowd pleaser, or at least, the punchline at the end was.

The real purpose of a user group is to bring together users (and prospective users). There was much lively discussion after my presentation, which I won't attempt to recapitulate here. The next NYMUG meeting will (most likely) be sometime in January, so please plan to come. I'll post more concrete details here when they're available, and we'll send them to the mailing list, of course.

My biggest problem in preparing for speaking events is figuring out what to talk about, and then figuring out how to stretch that topic to fit in the time allotted.

When I was suggested as the speaker for our inaugural New York meeting, I had to figure out what to talk about. I quickly thought of a topic, but never imagined that it'd be possible. To my delight and surprise, when I shopped the idea around engineering and marketing, there was universal support for the idea. So after I picked my jaw up off the floor, I had to turn my attention to stretching the topic.

The topic I had in mind would easily fit on a couple of slides. That would make for a pretty short talk. At a user group meeting, maybe that wouldn't be all bad, but I felt I needed to say a bit more. I stretched things out by talking about three new and cool features that I thought some folks might not have seen or used yet.

Support for https: and URI rewriting

Support for https: is pretty self-explanatory. Lots of sites have private information (user profiles and passwords, for example) that should not (many would say “must not”) be sent over an insecure communications channel. Furthermore, most users have been trained to look for a secure connection before sending credit card or other financial information over the web.

Support for URI rewriting allows application authors to make cleaner interfaces. I'm a fan of clean URI interfaces. Call me picky, but think it's a lot better to expose the 4th slide in your presentation as http://example.com/slides/nymug/4 than as http://example.com/slides.xqy?deck=nymug.xml&foil=4&format=html.

Until recently, if you wanted to do this with a web site built on top of MarkLogic Server, you had to put up a proxy of some sort (often an Apache web server) to provide https: support and URI rewriting.

Starting with MarkLogic Server V4.1, this is no longer necessary. MarkLogic server now supports https (with your own certificate or a generated, self-signed one) out of the box. The server also supports URI rewriting by allowing you to designate an arbitrary query module to rewrite URIs. Here's the example I used in the presentation:

xquery version "1.0-ml";

declare variable $url as xs:string
        := xdmp:get-request-url();

if (matches($url, "^/slides/([^/]+)/([0-9]+)$"))
then
  replace($url,
          "^/slides/([^/]+)/([0-9]+)$",
          "/slides.xqy?deck=$1.xml&foil=$2")
else
  $url

It's an incomplete, toy example taken from the real code I used on the server behind my presentation, but it gives you a flavor for how it works. Your module starts with the URL that that was used (and access to the headers and other parts of the request), performs any sort of computation that you'd like, and returns the new URI. The new URI then goes into the server and is processed normally.

There may still be good reasons to put a proxy in front of MarkLogic Server (load balancing, etc.), but you no longer have to just to satisfy requests for these two common and simple features.

Office Toolkits

For reasons that will become clear later on (and not only because I find “office applications” to be an inefficient, frustrating, pointless time-suck), I wanted to present this presentation using ordinary web technologies. (Specifically, HTML+CSS+JavaScript served up by MarkLogic Server.)

At the same time, because I was going to talk about new server features, I was required to present a disclaimer:

Disclaimer

All statements describing future releases, estimated release dates and content are plans only, and Mark Logic is under no obligation to develop, include or make available, commercially or otherwise, any specific feature or functionality in any Mark Logic product.

Information is provided for general understanding and informational purposes only, and is subject to change at the sole discretion of Mark Logic in response to changing customer requirements, market conditions, delivery schedules and other factors.

(A disclaimer that applies as much to this weblog essay as it did to my presentation last night, I might add.)

Trouble is, this slide was sent to me in Powerpoint. To use it, I'd have to switch to Powerpoint for the rest of my presentation (yuck!), copy and paste the text (where's the fun in that?), or find some way to incorporate the slide into my DocBook-based slide deck (now that sounds interesting).

Luckily, one of our engineers, Pete Aven has already done all the heavy lifting. Pete's the primary developer of Mark Logic's open source toolkits for Word, Excel, and Powerpoint. Each toolkit provides a pipeline for ingesting office documents into MarkLogic server, an office plugin for using the server from the application, and some XQuery code to work wtih the files in the server.

With that framework in place, it was pretty easy to write a little bit of XQuery code that would extract a slide from a deck and transform it into DocBook. (That's about 20 lines of code, nothing fancy, it extracts paragraphs and bulleted lists from slides, no more, no less.)

The source for my final presentation looks like this:


  
    Transforming XML Development <?lb?>with MarkLogic
    …
  
  

    
      NYMUG!
      …
    

    

    …

a straightforward mixture of hand-authored slides and references to slides from a couple of Powerpoint decks. For the presentation, I edited a Powerpoint slide and ran it back through the process in real time, but that doesn't translate very well to a weblog essay.

Presentation

That left just the last part of my talk, “the big reveal” as it were. Having set this all up so that I can author in DocBook, even including Powerpoint slides, serve it up over https:, and use nice looking URIs, I still have to go the last mile and get the content into the browser.

A couple of obvious ways present themselves. I could serve it up as XML with a stylesheet and let the browser do the work. Could do, but I didn't. I could translate the DocBook markup into (X)HTML using XQuery in MarkLogic Server. Could do, but I didn't.

What I really want, but haven't been able to do, is to transform the DocBook in MarkLogic Server with XSLT. And drum roll … I can haz! cymbal crash

let $doc := xdmp:document-get(concat($ROOT, $xml))/*
let $expanded := local:expand-powerpoint($doc)
let $map := map:map()
let $put := map:put($map, ...("dbp:foil")), $foil)
let $put := map:put($map, ...("dbp:deck")), $deck)
let $put := map:put($map, ...(xs:QName("dbp:format")),
                    $format)
return
  xdmp:xslt-invoke($xslt, $expanded, $map)

Running an internal build, I can demonstrate support for XSLT 2.0 in MarkLogic Server. (Go back and read that disclaimer again now.)

There was some rejoicing at the user group meeting, I hope there's some rejoicing out there over the intertubes too. I'm certainly giddy with delight over the prospects of high-performance XSLT processing in MarkLogic Server V.future.

What more can I say. When you've done your best trick, it's time to get off the stage.

Thanks again to Steve Kotrch and Simon & Schuster for hosting. Hope to see you all in January!

NYMUG: New York Mark Logic Users Group!

2009-11-04T19:32:47Z

The inaugural meeting of the New York Mark Logic User Group will take place on Wednesday evening, 11 November 2009.

Please come join me at the inaugural meeting of the Mark Logic New York User Group on Wednesday evening, 11 November 2009, hosted by Steve Kotrch from Simon & Schuster!

What

I'd say more about what I'm going to say if I'd figured out more of what I'm going to say. The title of my presentation is Transforming XML Development with MarkLogic. I think it'll be a fairly interactive show. Without any spoilers, I'll admit to having some DocBook, 5.0 of course, some Powerpoint, some HTML, and I'm pulling them together in some pretty damn cool ways, if I do say so myself.

If you're in New York, please stop by (and please let Cleo know if you plan to stop by).

See you there!

DocBook V5.0

2009-11-04T18:55:29Z

DocBook V5.0 is an OASIS Standard!

I reported a couple of weeks ago that the ballot to make DocBook V5.0 an OASIS Standard was open. I can now report that the ballot is closed, it closed on 31 October.

To my great satisfaction, I can also report that we easily cleared the process hurdles.

DocBook V5.0 is officially an OASIS Standard.

I'd like to thank everyone who took the time to help us get here: users, reviewers, members of the Technical Committee (past and present), and the OASIS members who voted for us, of course.

Evernote

2009-11-01T21:51:13Z

With a scanner and some Python, I'm an enthusiastic convert to Evernote.

I first tried Evernote more than a year ago. It seemed interesting, especially the ability to extract text from uploaded notes, even handwriting. It's not performing traditional OCR, but it does build a list of likely words in each note. That's enough to make search quite useful.

Here we see a match for “America” in my barely legible scrawl captured by JotNot:

Evernote search

Even though this all seems pretty cool, I never really used it very much. Part of the problem was that I didn't seem to be putting very much into it. If you've only got six notes, you don't need folders or tagging or clever search to find them.

A few weeks ago, Michael Mealling pointed out that one way to fix the data problem was to scan documents directly into Evernote. That turned out to be an excellent suggestion.

For example, I've had a manilla folder of articles torn from magazines in a drawer in my desk for ages. The threshold for getting torn out and shoved in that folder was pretty high because, let's be honest, if I put too many things in that folder, I'll neither remember what's there nor be able to find it if I do remember.

Twenty minutes with a scanner and suddenly I've got something that I can tag, categorize, and search painlessly. What's more it's something I have with me on my laptop or my phone or anywhere I can get to a web browser: it's not in a folder in a drawer thousands of miles away (as my desk is this week, for example).

Of course, then I had a much bigger problem. I am very, very reluctant to put my data in your application if I can't get it back again. Repeat after me: no roach motels. I wish Evernote every success in the world (I'm even doing my part, to the tune of $45/year, to keep them around), but it's not hard to imagine a future in which I've come to rely on them as a repository of important information only to discover some Thursday afternoon that they've gone away or lost all my data or otherwise left me high and dry.

To their credit and my relief, they have an API. It's not the sort of RESTful Web API I've come to expect from these sorts of services, but that's ok. It's published and documented and claims to support Java, Perl, PHP, Python and Ruby out of the box.

A few short hours of hacking and a few hundred lines of Python and I had a backup tool that gets back everything I put into Evernote and an XML representation of the search data. What more could I ask for?

(If you're interested in trying my backup script, you'll need to go through Evernote support to get your own API key and configure a few lines to indicate where you want the files stored, but after that it should work for you too. YMMV, of course.)

My script creates an XML representation (natch!) of the details returned by the Evernote APIs. Eventually, I'll probably decide to fix things so it doesn't create a single potentially enormous file, but it doesn't much matter to me at the moment because I turn around and drop this into the MarkLogic Server instance that I use for my local PIM data.

Now, instead of collecting only the very most interesting articles I see and losing them in a folder, I collect anything that interests me even slightly, confident in the knowledge that I'll always be able to find it, and everything else, with ease. Alongside those articles, you'll find scanned business cards, recipes, specifications, notes, photographs of whiteboards and napkins, receipts, and a host of other interesting words and ideas that I've captured as the universe pushed them past me.

Pretty sweet.

Call for Vote - DocBook V5.0

2009-10-19T19:02:35Z

DocBook V5.0 is ready to become an OASIS Standard!

It's been a long, and sometimes winding, path from DocBook V4.5 to DocBook V5.0, but we're finally approaching the finish line to make it official. The OASIS Call for Votes went out on Friday!

The ballot is open now through 31 October, 2009.

If you belong to an OASIS Member organization, please encourage your representative to vote “Yes”. (The full list of eligible voting members is included on the ballot).

DocBook V5.0 is a significant milestone in the evolution of the standard. Based on RELAX NG and designed with extensibility and flexibility in mind, making DocBook V5.0 an OASIS Standard will provide the solid foundation that we need to continue improving The Source for Documentation™.

(We really do have some cool stuff coming down the pipe, so as they say in Chicago, vote early and vote often!)

Micro-blogging Backup, part the fifth

2009-10-18T20:14:49Z

In which we clean things up.

If you've been using one of the micro-blogging services for a while, you're probably familiar with a set of conventions that have evolved for adding metadata to your status messages. The ones I'm familiar with are:

@user to identify another user.
#tag to add a “tag” to your message.
!group to identify a group (at the moment, this seems only to be an Identi.ca convention).

In addition to those conventions, the use of “URL shorteners” (http://tinyurl.com/, http://bit.ly/, http://is.gd/, etc.) is common. And, finally, although it may not be apparent in the client you use, at the API level, individual status messages may indicate that they are “in-reply-to” some other message.

So far, our micro-blogging backup system doesn't take advantage of any of this extra information.

One of the first things I decided to do was expand shortened URIs. There's no 140 character limit in the database and if you link to something on http://youtube.com/, the odds that I want to follow that link are within ε of zero. I'd like to know before I click.

As long as we're grovelling through the text of each message, it makes sense to expand the other conventions, turning them into the appropriate links.

It also makes sense to download any messages that an existing message is “in-reply-to”. If those messages are also replies, we'll follow them too until the trail ends. This allows us to display whole conversations, even if they involve participants that we don't follow.

All of this can be accomplished with one new module, /modules/cleanup.xqy, and a new top-level query to drive it, /clean-tweets.xqy. The interesting bits are in the cleanup.xqy module:

The actual work is just string manipulation: regular expressions and tokenize, mostly.
Following replies counts against your rate-limit, so we do at most 50 at a time.
To expand URIs, we perform HTTP HEAD requests against the URIs we find in the status messages. In the worst case, some of those may timeout, so we do at most 500 at a time. That way we're unlikely to perform a query that takes so long that it times out.
If you look closely, you'll see that in addition to doing the expansions, we also add new elements to the status document: for mentions of another user, for tags, for groups, and for the host names of expanded URIs.

We'll come back in some future installment and use those for faceted searches (e.g., “find all the messages by @xmlcalabash that include links to tests.xproc.org”).

Pop the two files mentioned above into your setup (if this is your first encounter with my micro-blogging backup series, make sure you start at the beginning).

After you've installed the files, running http://localhost:8330/clean-tweets.xqy will start cleaning up your database. If you've downloaded a lot of messages, you'll have to run it several times. If you have a lot of replies, you'll have to spread the runs over a few hours.

The fact that you sometimes have to run the cleanup scripts several times is a bit inconvenient. I'm experimenting with some JavaScript to improve that, but I'm still looking for better solutions.

Built my own...

2009-10-16T01:31:15Z

Another geekdom right of passage: builing my own box.

As I noted back in June, I've always wanted to build my own box. Now I have.

I don't, in retrospect, claim to have learned what “the right answer” is, but here's the answer I arrived at:

AMD Phenom II X4 955 Black Edition CPU

AMD Phenom II X4 Processor Black Edition

I was more than a little amused at the discrepancy between the size of the CPU and the size of the aluminum and copper monstrosity that sits on top of it to keep it cool:

Cooling apparatus for the CPU
ASUS M4A78T-E AM3 790GX HDMI ATX AMD motherboard

Motherboard in place
4 x G.Skill 2GB 240-pin DDR3 1600 memory

Four of 8Gb
4 x WD Caviar Green 1TB SATA hard drives

Gobs of space
LG Black 8x BD-ROM 16x DVD-ROM 40x CD-ROM SATA LightScribe burner

Optical drive
APEVIA ATX-AS680W-BL 680W ATX12V/EPS12V SLI power supply
XCLIO Windtunnel ATX Full Tower case

Case with power supply in place

I was pleasantly surprised how easy it was to assemble. The documentation was clear and complete, the parts all well labelled; it went together without a hitch.

All systems go!

Glow in the dark

(For a few more pictures, see Custom Build 2009 on Flickr.)

I decided to run bleeding-edge Ubuntu 9.10 Server beta on it. It's not that I wouldn't like to try Solaris and ZFS, but…the days are short and the list of projects is long. Installing Ubuntu is just that little bit easier. Ubuntu 9.10 installed flawlessly.

The ASUS motherboard includes a video controller and assorted other controllers. It claims to do RAID, but a little investigation reveals that it's fakeraid so I abandoned it. Instead, I setup software RAID. The first disk is the boot disk, the remaining three provide 2TB of disk in a RAID5 configuration.

The case, for all those monstrous fans, is pretty quiet, but not silent. I still think the server might get relocated to the basement, except that I worry about the higher humidity.

Anyway, MarkLogic Server is up-and-running and as soon as I get my Micro-blogging backup configuration ported over, I promise I'll write that next installment…

XML Calabash 0.9.15

2009-10-06T01:04:11Z

A new release at last. New features, fewer bugs, and test suite clean again.

Speaking of work (letting the non sequitors pile up), when I wasn't on the beach, I was hacking XML Calabash. I always update the project status page, but in case that's too subtle, perhaps this essay will catch your attention. (You're reading this, so I guess that should be “caught” without the “perhaps”, but nevermind.)

Next in the task queue: supporting Saxon 9.2. I fear this is going to be a bit of a b*tch as I reach down into the guts of Saxon in a few places. Fair warning: I don't plan to attempt to support previous versions of Saxon after I make this switch.

XML Calabash Logging

One of the changes in 0.9.15 is the switch from my own crufty, home grown logging infrastructure to the Java core logging facilities. On the plus side: this gives you a lot more control over the logging. On the minus: you have to deal with the core logging facilities.

Near as I can tell, this has to be done with a properties file. Here's what I do: I set the system property java.util.logging.config.file to point to my own logging configuration file, /Users/ndw/java/logging.properties.

My logging properties file, constructed mostly through trial and error, is shown below. The important bits are:

There are two places to control the amount of detail in the logs. By default, the console handler won't print any messages more detailed than “INFO”. If you want to get more detail, you have to turn this knob appropriately.

I really dislike the default message format, so I wrote a formatter that produces slightly more compact output. You might like it too.

This is the other place where you can control the amount of detail. “ALL” gives you all the messages. You might prefer “SEVERE”, which gives you only the fatal errors.

You can be selective here. If, for some reason, you want to see the gory detail of the XInclude step, but ignore everything else, you could set:

com.xmlcalabash.level=SEVERE
com.xmlcalabash.XInclude=FINEST

I'll try to provide better documentation for the available names.

Here's my current logging properties file:

############################################################
# Logging Configuration File
#
# You can use a different file by specifying a filename
# with the java.util.logging.config.file system property.  
# For example java -Djava.util.logging.config.file=myfile
############################################################

############################################################
#       Global properties
############################################################

# "handlers" specifies a comma separated list of log Handler 
# classes.  These handlers will be installed during VM startup.
# Note that these classes must be on the system classpath.
# By default we only configure a ConsoleHandler, which will only
# show messages at the INFO and above levels.
handlers=java.util.logging.ConsoleHandler

# To also add the FileHandler, use the following line instead.
#handlers= java.util.logging.FileHandler, java.util.logging.ConsoleHandler

# Default global logging level.
# This specifies which kinds of events are logged across
# all loggers.  For any given facility this global level
# can be overriden by a facility specific level
# Note that the ConsoleHandler also has a separate level
# setting to limit messages printed to the console.
.level = ALL

############################################################
# Handler specific properties.
# Describes specific configuration info for Handlers.
############################################################

# default file output is in user's home directory.
java.util.logging.FileHandler.pattern = %h/java%u.log
java.util.logging.FileHandler.limit = 50000
java.util.logging.FileHandler.count = 1
java.util.logging.FileHandler.formatter = java.util.logging.XMLFormatter

# Limit the message that are printed on the console to INFO and above.
java.util.logging.ConsoleHandler.level=FINE
java.util.logging.ConsoleHandler.formatter=com.xmlcalabash.util.LogFormatter

############################################################
# Facility specific properties.
# Provides extra control for each logger.
############################################################

com.xmlcalabash.level=ALL

I'm fully open to suggestions for better approaches.

XML Summer School ’09

2009-10-05T23:36:02Z

Open source and web technologies at XML Summer School.

Speaking of travelling, a bit of a non sequitor if you aren't reading these essays sequentially, I had just returned from XML Summer School the day before my Caribbean jaunt.

I was delighted to see the return of XML Summer School (September is summer somewhere, I'm sure) and even more delighted to be invited to speak.

Radcliffe Camera

I taught in the “Open Source” and “Web 2.0” tracks, presenting Open Source Application Development and Building Dynamic Web Applications with XML, respectively. I think the sessions went well. (After more than ten years of doing it, I have finally (and suddenly, this year) reached the point where it doesn't make me gut knotting nervous to give presentations. That has to have improved my stage presence.)

Oxford sunset

Beyond the specific courses, what sets Summer School apart is the staggering array of talent lined up to teach. If you've got a question or a problem about something even tangentially related to markup, there's someone at XML Summer School who's thought about it, built it, or deployed it. Probably all three. That we're all good friends, engaging students not just in the classroom but also punting and pub crawling, is just gravy for everyone.

Punters

A bargain at twice the price, I promise.

After Summer School, I snagged a weekend with my folks.

Got seeds?

A relaxing end to a great week not without hard work and long days.

Dominican Republic

2009-10-05T22:51:30Z

A long weekend in the Dominican Republic brings me to country number 15.

The Dominican Republic is country number fifteen for me. Deb had to work, and I did a little work too, but mostly I was free to enjoy the sun and the sea and the sand.

Punta Cana Beach

We don't live in the Caribbean, remind me why.

Moon Palace Pool

Footprints

The Moon Palace resort was lovely (in a big “cruise ship on land” sort of way), and this is not a criticism, just an observation, but clearly there are no OSHA inspectors in the Dominican Republic.

OSHA? What's that?

Or maybe there are no personal injury lawyers, I don't know. The tripping hazard in our room was not the only structural feature that made me do a double take.

SQL to XML

2009-09-26T12:01:23Z

A number of Mac applications store information in SQLite databases. Step one to do something useful with that data is to get it into XML.

I hate having my data squirreled away in proprietary or quasi-proprietary ways. If I can't get my information back out of an app, I'd rather not use it. When I started using the Mac, I switched to iCal and AddressBook: both can export data in standard text formats which I can easily convert to XML.

But exporting the data is a manual process (though I could probably automate it with some clever AppleScript or something, I've never tried). I build a number of views of my address book and calendar data automatically so manual processes don't fit well into my workflow.

It didn't take too long to figure out where iCal stores my appointments or how to pull them together. Having worked out where iCal stores my appointments, I turned my attention to AddressBook.

Long story short: the address data is saved in a database in ~/Library/Application Support/AddressBook/AddressBook-v22.abcddb. After installing the sqlite3 application from MacPorts, I was able to extract a text dump. So far so good.

Here, for example, is a table definition and the first row of data in that table:

CREATE TABLE ZABCDRECORD ( Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER,
Z_OPT INTEGER, ZDISPLAYFLAGS INTEGER, ZMODIFICATIONDATEYEAR INTEGER,
ZCREATIONDATEYEAR INTEGER, ZADDRESSBOOKSOURCE INTEGER, ZISALL INTEGER,
ZME INTEGER, Z19_ME INTEGER, ZINFO INTEGER, ZBIRTHDAYYEAR INTEGER,
ZPRIVACYFLAGS INTEGER, ZNOTE INTEGER, ZADDRESSBOOKSOURCE1 INTEGER,
ZCONTACTINDEX INTEGER, ZSOURCEWHERECONTACTISME INTEGER, ZVERSION
INTEGER, ZSYNCCOUNT INTEGER, ZSHARECOUNT INTEGER, ZADDRESSBOOKSOURCE2
INTEGER, ZMODIFICATIONDATE TIMESTAMP, ZCREATIONDATE TIMESTAMP,
ZMODIFICATIONDATEYEARLESS FLOAT, ZCREATIONDATEYEARLESS FLOAT,
ZBIRTHDAY TIMESTAMP, ZBIRTHDAYYEARLESS FLOAT, ZUNIQUEID VARCHAR, ZNAME
VARCHAR, ZNAMENORMALIZED VARCHAR, ZTMPREMOTELOCATION VARCHAR, ZNAME1
VARCHAR, ZREMOTELOCATION VARCHAR, ZSERIALNUMBER VARCHAR, ZSUFFIX
VARCHAR, ZTITLE VARCHAR, ZTMPHOMEPAGE VARCHAR, ZNICKNAME VARCHAR,
ZORGANIZATION VARCHAR, ZMAIDENNAME VARCHAR, ZIDENTITYUNIQUEID VARCHAR,
ZPHONETICFIRSTNAME VARCHAR, ZDEPARTMENT VARCHAR, ZPHONETICLASTNAME
VARCHAR, ZMIDDLENAME VARCHAR, ZFIRSTNAME VARCHAR, ZIMAGEREFERENCE
VARCHAR, ZJOBTITLE VARCHAR, ZPHONETICMIDDLENAME VARCHAR, ZLASTNAME
VARCHAR, ZSORTINGFIRSTNAME VARCHAR, ZSORTINGLASTNAME VARCHAR,
ZCREATEDVERSION VARCHAR, ZLASTDOTMACACCOUNT VARCHAR, ZLASTSAVEDVERSION
VARCHAR, ZSYNCANCHOR VARCHAR, ZSEARCHELEMENTDATA BLOB,
ZMODIFIEDUNIQUEIDSDATA BLOB );
INSERT INTO "ZABCDRECORD" VALUES(1,18,287,NULL,NULL,2008,NULL,1,NULL,NULL,
3,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,
239119683.670433,NULL,18281283.670433,NULL,NULL,
'93973926-7EF6-40F0-ADBD-8C7BBFC30FA1:ABSubscriptionRecord',NULL,
NULL,NULL,NULL,'local','B7303AAD-DA79-46E6-BC7D-91DAD82AEFB8',
NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,
NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL);

Next, I wrote 150 or so lines of Perl to convert the text into XML.

The XML is designed to be a totally straightforward representation of the table structure of the database. From the preceding SQL statements, sqltoxml produces:


  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  
  

  
    1
    18
    287
    2008
    1
    3
    239119683.670433
    18281283.670433
    93973926-7EF6-40F0-ADBD-8C7BBFC30FA1:ABSubscriptionRecord
    local
    B7303AAD-DA79-46E6-BC7D-91DAD82AEFB8

As you can see, I've made no effort to maintain some aspects of the database (like the primary key), I've simply dropped NULL fields, and I'm relying on the field names to be valid XML NCNames. There are clearly other, equally reasonable, design choices that I could have made.

The resulting XML is the bare minimum needed to switch to XML tools for subsequent downstream processing (turning address book tables into VCards, for example). But it gets the job done.

And maybe it'll come in handy for someone else.

RDFa for DocBook?

2009-09-22T13:20:39Z

Adding RDFa to DocBook would make it possible to add a class of semantic annotations to DocBook without changing the schema. But is that a good idea?

Knowledge is of two kinds. We know a subject ourselves, or we know where we can find information on it.

— Samuel Johnson

When Bob DuCharme introduced the semantic web track at XML Summer School this morning, he mentioned briefly the idea of adding RDFa to vocabularies other than (X)HTML. In particular, he's investigated how to do it in DocBook.

The DocBook TC gets periodic requests to add new inline elements and attributes for bits of metadata. Sometimes the requests are entirely legitimate, in the sense that they're clearly about technical documentation, but seem to apply to such a small audience that the TC is reluctant to add them to all of DocBook.

With this in mind, the idea of adding RDFa has some appeal: we add a few new attributes and henceforth users will be able to add new bits of metadata without having to change the DocBook schema.

But I'm not sure.

First, lots of DocBook elements have more discrete semantics than HTML elements. We don't need to say

Beautiful Sunset

because we have . We don't need to say:

because we have


  
    Some Name
    http://mypubco.com/empid/53234

I'm not suggesting those are exactly the same, they're clearly not, but I'm comfortable that existing DocBook elements are sufficient for the task.

(Yes, you'd need a DocBook-specific tool to extract the metadata, which is a disadvantage, but you probably want one anyway for the existing DocBook semantics.)

Second, it would allow you to construct statements with conflicting or, at best, odd semantics:


  Alice1
  This is from section 2.2.

I can just about imagine a sense in which “Alice1” can be both the title of a section and the Dublin Core creator of the section, but it doesn't make a lot of sense.

Third, Bob's example seems to suggest that it would encourage markup like this:


  The trouble with Bob2
  Alice2

which seems like a bad idea to me.

On the other hand, some of the examples do seem useful for exactly the sort of thing I suggested motivated my interest:

In fairness, Bob set out to recreate the triples from the original tutorial, so some of the markup choices were forced upon him.

So I'm not sure.

Micro-blogging Backup, part the fourth

2009-09-09T15:11:49Z

In which we get to see what our tweets and ’dents look like.

If you haven't been following along, go back and read parts one, two, and for a little background, three first. Now you've got MarkLogic Server up and running and you've been able to download your tweets and the tweets of those you follow. (Tweets or ’dents depending on which microblogging service you prefer; either, actually both, work for me.)

Next, download mbb04.zip and unpack it in the same place where you unpacked mbb02.zip. If you were following the instructions, you've edited some of the files in the “mbb/inst” directory and you may have some sessions saved in CQ, so I have not included those directories in mbb04.zip.

If you've been tinkering with other files, then you want to unpack this zip with some care or you may overwrite your changes. But you won't overwrite changes made to the installation or CQ areas. (By the same token, this distribution is incomplete without mbb02.zip).

With that installed, you now have a CSS file, four modules, and a new “top level” script, show-tweets.xqy. That's the fun one. Point your browser at http://localhost:8330/show-tweets.xqy and you should be rewarded with a list of your status messages from today. (As before, adjust the port number as necessary if you installed the application server on a different port.)

If you don't have any status messages from today, load http://localhost:8330/get-tweets.xqy to download your most recent messages, then try http://localhost:8330/show-tweets.xqy again.

The show-tweets.xqy script accepts query parameters. You can pass:

sdate: To specify the starting date in “YYYY-MM-DD” format. If unspecified, defaults to the ending date.
edate: To specify the ending date. If unspecified, defaults to today.
users: To specify one or more users separated by spaces (+ signs or %20’s in URI-speak). These should match the screen_name values in your account configuration. The value “ALL” is special, it will list tweets for every user, including all your friends.
service: To specify the service. If you setup accounts on multiple services, this will let you limit the result to only those messages on a single service.

Go ahead and give it a try, http://localhost:8330/show-tweets.xqy?sdate=2009-08-01&edate=2009-08-31&users=ALL will show you all the messages by you and those you follow posted in the month of August.

Message Formatting

The messages are sorted in ascending order by date, so that messages read chronologically “down” the page as you'd expect. However, conversations are handled a little bit specially.

Whenever a message is encountered that is part of a conversation (either because it's a reply to another message, or another message exists that is in reply to it), the whole thread is collected together and presented as a unit, like this:

Message threading

I think that makes the results much easier to follow. We'll come back to what to do about threads involving users other than you or your followers later.

There were some bugs in the Identi.ca server that occasionally caused incorrect “in-reply-to” values to be inserted into the data. I think those have been fixed in the server, but there's nothing I can do about the values that are wrong.

URL Rewriting

Those show-tweets.xqy URLs may be sufficient, but they're hardly elegant. I'd be much happier if they were better organized for human consumption.

Luckily, with the URL rewriting features of MarkLogic Server V4.1, this is easily achieved. To begin with, go back into the admin console (http://localhost:8001/) and navigate down through Groups→Default→App Servers in the tree control then select the server you setup for this project.

Near the bottom of that page, you'll find a “url rewriter” field. Specify modules/url-rewriter.xqy as the value for that field and then click “Ok” at either the top or bottom of the page.

The one I provided supports a range of values designed to be more readable:

/my-tweets/ service / users / start-date / end-date: Returns the messages associated with “users” on “service” between the specified start and end dates, e.g., /my-tweets/identica/ndw/2009-08-01/2009-08-15.
/my-tweets/ users / start-date / end-date: Returns the messages associated with “users” on any service between the specified start and end dates, e.g., /my-tweets/ndw/2009-08-01/2009-08-15.
/my-tweets/ start-date / end-date: Returns the messages associated with any user on any service between the specified start and end dates, e.g., /my-tweets/2009-08-01/2009-08-15.
/my-tweets/ start-date: Returns the messages associated with any user on any service between the specified start date and today, e.g., /my-tweets/2009-09-01.
/my-tweets: Returns the messages associated with any user on any service posted today.
/all-tweets/ service / users / start-date / end-date: Returns the messages posted by anyone on “service” between the specified start and end dates.
/all-tweets/ start-date / end-date: Returns the messages posted by anyone on any service between the specified start and end dates.
/all-tweets/ service / start-date: Returns the messages posted by anyone on “service” between the specified start date and today.
/all-tweets/ start-date: Returns the messages posted by anyone on any service between the specified start date and today.

The URL rewriter is just an XQuery that you can change; so it can support any kind of values that you'd like. It recieves the URL that the user entered. Whatever URL it returns is what the server actually responds to. The one I've provided turns

/all-tweets/2009-08-15/2009-08-31

into

/show-tweets.xyq?users=ALL&sdate=2009-08-15&edate=2009-08-31

So nothing else has to change about the code, it all just works.

I'm not sure there's much else that's new or interesting about the other modules provided in this part. If you have any questions, feel free to ask.

Next time, we'll look at dealing with all those ugly shortened URLs and pesky replies that extend beyond our followers.

Micro-blogging Backup, part the third

2009-09-03T20:12:40Z

In which we peel back the covers on what's been built so far.

There's more functionality to come, but first, I thought it might be useful to spend a few minutes looking at what we've got so far.

The setup code in /mbb/init isn't very interesting, and I'm not going to attempt to explain how CQ works, so we'll begin in the /mbb/modules directory.

accounts.xqy

This module contains some utility and convenience functions for dealing with account data. I changed my mind about how to store the data a couple of times early on, so these functions were supposed to protect me a little bit from that. I didn't follow through all the way so the account abstraction is pretty leaky, but I left this module in place anyway.

twitter.xqy

This module is a thin skin over the actual Twitter API. Ideally, I'd flesh this module out to support the rest of the endpoints, but I haven't bothered yet.

One school of thought on this kind of API module is that it should be as thin as possible, providing only the thinest skin over the underlying API. I mostly agree, but I did take a few liberties. If you wanted to adapt this module for some other purpose, you might have reason to carve it a little closer to the bone.

One decision I made was to have the account/rate_limit_status method return the number of calls remaining directly as a number, rather than returning the XML response. That's pretty simple. The other changes I made are a bit deeper.

The Twitter timeline methods are designed to be “paged”; the caller can select the page size and the page they want to retreive. I decided that what I really want is all the pages; so my versions of the timeline methods always request all the pages and return all of the results in a single call (by performing the requisite paging for you, behind the scenes). Twitter limits you to 16 pages, but Identi.ca servers seem to offer more pages. In order to avoid recursing beyond the size of the call stack, I placed an arbitrary limit on the number of pages.

Finally, I decided to protect the caller from exceptions that can occur if the underlying HTTP requests fail. Most of the public methods in twitter.xqy return an element, either the Twitter API response, or a element containing the HTTP error code if an error occurred.

I think an argument could be made for not doing this, for letting the lowest-level API calls throw the exception, but I decided not to. You're free to change that, of course.

twitproc.xqy

This module is mostly responsible for taking Twitter and elements and inserting them into the database. Along the way, we transform them just a little:

I move them from no namespace into the “t:” namespace. First, I subscribe to the position that XML vocabularies should place elements in a namespace. I'm aware that there are people who believe otherwise. They're wrong. Second, XQuery’s interpretation of unqualified names exacerbates the problem. So you could look at this as patching a bug in the Twitter API.
I transform the contents of the element into ISO 8601 format (so it fits more naturally into the data model).
The Twitter APIs return a element embedded in each message. This is probably a net win for limiting round-trip calls to the API, but it doesn't strike me as a very sensible way to store things in the database. I break out the users and store them separately.
I add a few more elments to each status message. These record information about subsequent processing to perform (more about that later), the screen name of the user who uttered the message, and information about who was logged in to retreive this message.

This is a little lazy on my part. Arguably, I should introduce another namespace for these additional elements (so that some future Twitter API change doesn't walk all over them), or maybe not store them in the messages at all. I invite you to fix it if it bothers you.

If you know something about MarkLogic Server, this may sound like a job for document properties. That's a good idea, particularly for the downstream processing markers. However, document properties are associated, as the name suggests, with documents in the database. Later on in the code for displaying messages, we're sometimes going to make a copy of the message (giving it a new parent element). Doing that breaks the association with document properties. I was trying to keep things simple, so I didn't use properties for one set of information and child nodes for another, I just pushed it all into child nodes. My bad.

update.xqy

This module wraps up the functionality of the twitter.xqy and twitproc.xqy modules, getting all the tweets for a user and inserting them into the database. The code for finding the most recent messages by (and not by) a particular user might be interesting to you. Ignore the $tweet-collection variable; it's a holdover from an earlier approach, no longer used.

get-new-tweets.xqy

This module exists only to be invoked from another module. It declares an external variable that identifies a single account then simply calls the get-tweets function from the update.xqy module for that account.

The last bit of code that we've got so far is get-tweets.xqy in the top level of the application server. This module loops over all the accounts that we've defined and, for each one, downloads and inserts any new status messages into the database. It does this by invoking the get-new-tweets.xqy module.

What's all this invoking stuff about?

The server takes a completely safe, transactional approach to database updates. You are guaranteed that every query that updates the database either succeeds in its entirety or fails. One of the things that you aren't allowed to do is make conflicting updates to the same document in the same transaction. You can demonstrate this easily, just run the following expression in CQ:

let $doc := some document
return
  (xdmp:document-insert("/scratch/foo", $doc),
   xdmp:document-insert("/scratch/foo", $doc))

The server will bark “XDMP-CONFLICTINGUPDATES” and no inserts will be made to the database.

Why does this matter to us? Well, imagine that you setup two Twitter accounts in our micro-blogging backup system. Imagine further that both of those accounts follow marklogic.

What's going to happen when we run the backup? Both accounts are going to download all of the status messages on their “friends” timeline, so they're both going to download all of the recent marklogic tweets. And they're both going to try to insert them into the database. And that's going to generate a “conflicting updates” error.

Using the two-step xdmp:invoke dance as shown in get-tweets.xqy and get-new-tweets.xqy avoids this problem. The semantics of xdmp:invoke are that it runs the specified module in a separate transaction.

Since no single user is going to download the same message twice, each transaction will succeed. In fact, some messages will get updated twice in the database, but that doesn't do any harm because the content of the message will be the same in each case.

An alternate approach to this problem is to manage the messages with greater care, identifying duplicates when they occur and not attempting to insert them in the database. This is the approach taken in twitproc.xqy for the simpler problem of dealing with duplicate s.

It would certainly be possible to refactor the code so that the xdmp:invoke call could be avoided, but in this case splitting work into several transactions feels like the more elegant solution. And any performance penalties associated with a few calls to xdmp:invoke are going to be totally swamped by the latency in the underlying HTTP requests, so there isn't really a downside.

What next?

In the next part, we'll push a little further forward, getting some code in place to display the messages we've downloaded. We'll also look at the subsequent processing I hinted at. Further down the road, we'll look at search, and then we'll add some JavaScript , refactor things a bit, and make an AJAXy/Web 2.0 UI for our application.

I hope you're enjoying the ride.

XML+XQuery+Google Voice+Python=WIN!

2009-09-01T13:45:54Z

It's finally possible to put all the pieces together.

I've been storing my PIM data (contacts, appointments, etc.) in XML for ages (and I do mean ages !). Since my Palm days, I've been translating whatever native format my PIM supports into XML. Where necessary (i.e. everywhere), I've used an RDF/N3-like annotation mechanism to support additional metadata.

For example, the “notes” field for the XProc telcon looks like this:

rdf:
p:class telcon
p:access public
p:phone #w3c-zakim
p:code 97762#

That gets parsed into the obvious XML by the conversion process:

telcon
public
#w3c-zakim
97762#

(The phone number “#w3c-zakim” means that there's an entry in my address book with the ID “w3c-zakim”. Why do none of the PIM applications understand that appointments and contacts are related!?)

I've been doing this for years because it's The Right Thing™, even though I've only been able to wring small (er, tiny, perhaps miniscule) amounts of practical value from it.

But no more!

The XML data is stored in my own MarkLogic Server instance (moving from a collection of Perl hacks to the server was one of my first personal projects after I joined Mark Logic). I now have a Google Voice number. And Scott Hillman ’s Python scripts finally let me connect all the dots^[1]!

A little Python hacking, a quick XQuery module, and I can make calls from a shell window.

$ call xproc
W3C XProc WG
+1-617-761-6200 97762#

Dialing +1-617-761-6200...

The call query searches both the appointments on today's calendar and the address book. That means “call seth” works equally well, and calls my boss. For contacts with more than one phone number, I can add “- phone” on the end: “call ndw@nwalsh.com -home” would call me at home, should I ever want to do that.

It's a tiny little thing, but it feels great.

I'm easily amused, I know.

Hmm. I should add an option to send SMS messages, too…

^[1]Well, technically, reconnect, but for all it's coolness, I never got much use out of the DTMF auto dialer.

Micro-blogging Backup, part the second

2009-08-28T18:16:44Z

In which we setup the database one screen at a time and then import our first status messages.

If you were following along yesterday, you've got MarkLogic Server up and running with the Community License. Now it's time to start putting it to work. (Cutting toothpicks with a chainsaw, but hey, you have to start somewhere.)

Well, almost. First, we have to do a little setup.

Download mbb02.zip and unpack it somewhere convenient. I choose /home/ndw/mbb for the purposes of this example, but I'm not sure your home directory is really the best place. Anywhere you'd like though, doesn't matter to me.

Fire up your favorite web browser and connect to the admin interface on port 8001 (http://localhost:8001/, probably); you'll need to login with whatever userid/password combination you selected at installation time.

Once you're there, click “Forests” in the “Configure” tree control in the left hand column and then select the “Create” tab. Enter any name you'd like for the forest and click “ok”. I named mine “mbb”.

Create a forest

Forests are where the server stores XML documents. Trees, as it were. Clever, eh?

Next, choose “Databases” in the tree control and select the “Create” tab again. Enter any name you'd like for the database and click “ok”. I named mine “mbb”. I can't think of a compelling reason to give them different names, but suit yourself.

Create a database

Once you've created a database, you'll be reminded that you need to attach a forest to the database.

You must attach a forest to the database

Click on that link and do so. Remember to click “ok”.

Attach the forest you created

Almost there. Choose “Groups”, “Default”, and “App Servers” in the tree control, then select the “Create HTTP” tab. Enter any name you'd like for the server name, I named mine “mbb”; enter the location where you unpacked the zip file for the root, I used /home/ndw/mbb; and enter an open port value for the port, I used “8330”.

But don't click “ok” just yet. (If you already did, no worries, just click on the app server's name in the tree control.)

Create an HTTP application server

Scroll about half way down the page to change the authentication and default user. Select “application level” for the authentication scheme and “admin” for the default user.

Change the authentication to application-level

This gives your application complete access to the server without having to login. There are lots of ways to make an application more secure, but let's leave all the security knobs for another day. Now scroll to the top or bottom and click “ok”.

At this point, you have a real honest-to-goodness application running on your server. (And yeah, this should all be simpler and easier. I've heard tell of plans to improve it, but nothing I can swear to.)

I included a copy of “CQ”, a browser-based, interactive XQuery environment in the distribution. You can see it if you navigate your browser to http://localhost:8330/cq. (In this and all the following examples, if you chose a different port, use the port number you chose.)

If you click on the “explore” link at the top of the CQ page, you'll see that you've got an empty database.

CQ shows the empty database

Now it's time to configure this particular database for our micro-blogging backup application. Later on, we're going to need some indexes. You could walk through the admin UI to create them, but that's tedious, you only have to do this once, and the admin UI is completely scriptable, so I created a little query to do the grunt work.

Point your web browser at the database configuration script: http://localhost:8330/init/setup-database.xqy. If everything is setup correctly, you'll quickly get a “database configured” message.

Configure the database

Next, we need to configure the microblogging accounts that you want to backup. Like database configuration, you're probably only going to do this once (or at least once in a great while), so I didn't create any sort of UI for it.

In the directory where you unpacked mbb02.zip, open up init/setup-accounts.xqy with your favorite text editor. On lines 57 and 58 replace SCREEN_NAME and PASSWORD with the Twitter username and password that you want to backup.

If you're using Identi.ca instead, you'll have to do a little more editing, but it should be pretty straightfoward. If you're using your own install of the Laconica software, or you're using some other microblogging server, as long as it supports the Twitter API, you should be able to figure out what to do. Feel free to ask if you're not sure.

When you've got all your accounts in place, save the file and point your web browser at it: http://localhost:8330/init/setup-accounts.xqy.

Configure your accounts

If all goes well, you'll get an appropriate “Accounts initialized” message. If you get 500 errors, you messed up the XQuery syntax somewhere. It won't do any harm to run the setup account script more than once, so try making small changes, running the script after each change. If you get stuck, let me know.

If you go back to CQ again and click the “explore” link, you'll see that there are documents in the database now, one for each account you added.

CQ shows the database with one document

Now we're ready to really do something.

Point your web browser at http://localhost:8330/get-tweets.xqy to download your status messages. This may take a while, especially the first time and especially if you entered several accounts.

Download the status messages for your account(s)

If you get a message about “rate limit exceeded”, it means you've done too many interactions with the Twitter API this hour. Wait a bit and try again. Twitter threatens that they'll turn off your account if you flagrantly violate the rate limit, so the MBB queries are pretty careful not to.

The “explore” link in CQ will now show a whole bunch of documents in the database.

CQ shows a database full of documents

You can enter any arbitrary XQuery expressions you'd like into CQ. Here I've asked for a count of all the messages that I've “favorited”.

Arbitrary XQuery expressions evaluated by CQ

In the next parts, we'll look at some of the code behind this functionality in a little more detail, add some XQuery to display the messages, look at how we can augment the messages in useful ways, add searching, and finally pull the pieces together into a useful little app. Well, a little app I think is useful, anyway.

What about my older messages?

Twitter only lets you get at the last 3,200 or so status messages with the Twitter API. If you've got older status messages that you've already backed up, or if you can find some other API to get at them, there are other ways to get them in the database.

I left the skeleton of one of those ways in the init directory, an XProc pipeline that XML Calabash can run to load status messages from existing XML files.

If you've got your old tweets archived in XML, drop me a line and I'll try to point you in the right direction.

Micro-blogging Backup, part the first

2009-08-27T13:23:47Z

What started out as a trivial exercise in backing up my Twitter and Identi.ca posts turned into a little microcosm of XML Server application development. It's something you can deploy for free on your very own MarkLogic Server!

This is the story of the intersection of two ideas:

First, almost no one that I spoke to at Balisage had heard of the Community License for MarkLogic Server, and those few who had thought that it was still limited to just 100Mb of content.

The fact that you can download and play with the best XML server on the planet is something more people should know about! The community license is for non-commercial use only but it's free and it never expires. The previous 100Mb content limit has been upped to 10Gb so there's a lot more room in the sandbox now.
Second, at about the same time, there was a little spike of interest in backing up microblogging data, the status messages that you send to services like Twitter or Identi.ca.

Sturgeon's law applies, of course, to microblogging. And Sturgeon was an optimist. But there's still a lot of useful information out there and I don't want it to disappear under the waves just because some acquisition occurs and the terms of service shift under my feet.

Luckily, the APIs for getting your microblogging content return XML and I have an XML server, so… my first thought was to download the tweets (a “for” loop in Bash and wget will do the trick) and store them in the server. Then I thought, that's silly, the server can download them for me…

From there, my little ten minute exercise grew until I had a (still relatively small) appication that handles oodles of documents from multiple services and accounts, has threaded conversations and account merging, uses indexes, has full-text and faceted search, employs web APIs, uses URI rewriting, and even has some AJAX.

And because status messages are small, it'll run for ages under the community license.

My plan, therefore, is to spin this out over a few essays, building the app from its barest bones to something I'm finding quite useful. If you want to play along, the first step is to go get a copy of MarkLogic Server and install it with the community license. The steps are roughly these:

Download version 4.1 of the server. It runs on Windows, Linux, and Sparc boxes. (It isn't, alas, available for OS X, but it runs just fine under virtualization.)

It also runs just fine on the Debian flavors of Linux, though that's not an officially supported platform. Just make sure you have the bugfixed version of lsb-base.
After it's installed and running, point your web browser at http://localhost:8001/ on the machine where you installed it.
Click on the “free” license button, choose the community license, and click your way through the rest of the install screens.

Congratulations! You know have the most powerful XML chainsaw imaginable at your fingertips. Exactly what to do with it is the subject of part the second and beyond.

Using XML Catalogs and XProc together

2009-07-22T20:15:27Z

XML Calabash, my implementation of XProc, is my go-to tool these days for manipulating XML documents. Adding XML Catalogs into the mix just makes it sweeter.

Recently, I was presented with several hundred books comprised of many thousands of chapters. My goal: load them into the server so that they could become part of a larger application. Easy peasy.

Two snags: all the chapters contained references to named entities declared in an external subset and none of the metadata in each file was actually reliable.

Still pretty straight-forward. Parse the document to expand the entity references, do a little cleanup, and push them into the database. The details of the pipeline aren't that important, the bit I want to highlight today is the parsing.

Everything remained pretty easy until I discovered that there were a half-dozen or more flavors of DTD in use across this corpus. And naturally, every external subset was referenced only by a system identifier with some random, absolute path:

Where “10” was “10”, “11”, “21”, “25”, etc. for some substantial enough set of versions to be go well beyond my limit for tedium.

Luckily, all of them were including a standard suite of ISO entities and (as far as I could easily tell), that's all the entity references ever were.

XML Catalogs to the rescue.

First, grab a recent version of the DTD and stick it somewhere local, then construct the following catalog:

Next, tell XML Calabash to use catalogs. You can do this from the command line, but I set it up in my configuration file, ~/.calabash:


  false

The first few lines just set some defaults I like, it's the last two that are relevant here. I tell XML Calabash to use my XML Resolver catalog implementation for entity and URI resolution.

Now my pipeline simply does The Right Thing™.

When the parser attempts to load the external subset, the catalog resolver returns the local DTD (because all the system identifiers end with “.dtd”). The step doesn't do validation by default, so the fact that some of the files aren't valid according to the particular version of the DTD that I have locally doesn't matter. The entities get expanded correctly. (If any of the documents had relied on other entities only present in a particular version of the DTD, that would have been an error, so I know I didn't miss any.) I do a couple of lightweight transformations on the resulting document and shove it into the database FTW!

Nothing earth shattering here, and not the only way to solve the problem, but one that looks like a nail to my particular hammer of choice at the moment.

Not exactly XProc

2009-06-23T22:27:55Z

One advantage of being an implementor is that I can play with languages that the Working Group didn't approve.

I've implemented a number of XProc extensions, and have plans for at least a few more, but so far they've all used standard extension mechanisms.

On the train ride home Monday night, I decided to do something different. Implementor's prerogative.

The XProc specification states that all variables, options, and parameters are string values. On the whole, I think this is a useful simplification:

All of the options used by the standard atomic steps have convenient string representations: they don't need more complex structures.
In an XPath 1.0 implementation there are only a few data types anyway (remember, there was a time when we thought we might finish before the XSLT/XQuery WGs). [Ah, optimism! -ed ]
Using strings simplifies serialization issues for steps like .

But it's frustrating in one particular area, XSLT parameters and XQuery external variables can have more complex values. The fact that XProc doesn't support this means that there are some stylesheets and queries that can't be fully supported by XProc.

Early on, I proposed that we allow parameters at least to contain either strings or documents, but I couldn't get working group support for the idea. (I think they'll come around, but not in 1.0.)

I've wondered, ever since my idea got left on the cutting room floor, how hard it would be to support arbitrary XDM values in XProc.

So I implemented it.

Turns out it's not very hard at all. I extended the RuntimeValue object to preserve the original XDM value of the expression instead of discarding it after computing its string value. In and , instead of using the string value for parameters and external variables, respectively, I use the XDM value. Everywhere else, I continue to use the string value so this change has no impact on other atomic steps.

In compound steps, I made a change analagous to the changes for and , when setting up the environment for evaluating XPath expressions, I use the XDM values of options and variables instead of the string values. This means that user-defined pipelines can accept and use XDM values.

The hardest part, by far, was changing the step and the interpretation of documents to support an extended serialization for arbitrary XDM values.

All of which means that you can do things like this:






  
    
      value
      value2
      
        
          Some fragment. How doc/p is useful
          in a configuration file, I don't know.

The param-seq option of our user-defined cx:foo step expects a sequence (even though this is silly thing to do in this case).

We extract items from this sequence to establish the values of the stylesheet parameters.

Back out in our main pipeline, we extract values from the configuration file and store them in variables. (We don't have to do this, of course, we could have computed the sequence directly with XPath expressions.)

Pay particular attention to the first value . This XPath expression selects a node; in standard XProc, this would automatically become a string. Using the general values extension, this will remain a node, which may not be what was intended.

The second value uses string() to explicitly make the parameter into a string. The third example also selects a node.

Finally, we pass all of these values to the cx:foo step as a sequence . In standard XProc, this sequence would be collapsed into a single string value, but it will remain a sequence if we use the general values extension.

Run through a standard XProc processor, here is the expected result:


   valuevalue2
          Some fragment. How doc/p is useful
          in a configuration file, I don't know.

We get the string value of all the variables, options, and parameters with the param-seq option compressed to a single string value.

But if we enable the general values extension (with -X general-values on the command line with XML Calabash version 0.9.12), we get a different result:


   
      value
   
   value2
   
      
                Some fragment. How doc/p is useful
          in a configuration file, I don't know.

Here our sequence has been passed successfully and each of the individual values has been preserved all the way through to XSLT.

With the general values extension, XML Calabash does not implement XProc 1.0! It implements a closely related, but entirely non-standard language which you cannot expect to interoperate with other implementations.

There are still a few obvious weaknesses in this extension.

Implementing a non-standard extension is a bad thing. I probably should disable it completely.
There should be a mechanism (an as attribute, probably) to selectively enable this behavior. This would also allow for type-checking the values passed around.
The serialization used by is incompletely supported. Although the serialization identifies the type of atomic values, the code which interprets this serialization ignores the types. Integers may go in, but strings come out.

This is an experimental feature. It may or may not survive over the long run. Comments most welcome.

Remember: if you enable this extension, you are not running a conformant XProc processor. Your gun, your bullet, your foot.

Java vs. AJAX

2009-06-10T20:22:10Z

Watching the twitter stream from JavaOne go by, I was initially surprised by the apparent frontal assault on AJAX. It seemed like an odd target at first; on further reflection, not so much.

Pointing the big guns of JavaOne at AJAX seemed very strange when I first saw it go by. A bit like attacking tractors or escalators: it's a bit of infrastructure, a programming technique, not a competing vendor or product. But later, the possible significance dawned on me (maybe I'm slow on the uptake and this is obvious to everyone else, I dunno).

AJAX is a useful technique for building rich, information-centric applications, the sorts of applications that help readers get the most value from what they see and help companies get the most value out of the content they have.

As it turns out, these are exactly the sorts of applications that are easy to build on top of MarkLogic Server (or, more broadly, with similar technologies; see also XRX). The application runs in the browser (standard HTML+CSS+JavaScript) and uses AJAX (with XML or JSON, that's not really significant for the point I want to make) to provide a dynamic, interactive experience for the user. The server can very quickly search through large amounts of content and transform what it finds back into the nuggets needed to satisfy the AJAX requests. The result: easy to build, easy to deploy, responsive applications that make users happy: win, win, win.

What's significant about this isn't how efficient it is, or how agile, what's significant is what's missing: there's no relational database (because that's a dumb way to store the rich content that drives applications like this) and there's no Java stack (because, well, because you just don't need anything as awkward and complex as a Java application server to get the job done^[1]).

So, just maybe, AJAX is a kind of competitor to Oracle+Java. Credit where it's due, I never would have seen that.

[ Yes, I'm coming late to the party. I put this in the “wait 24 hours” bucket and then forgot about it for a few days. –ed ]

^[1]Java fans: please put down the flame throwers. I like Java. I'm a Java fan. It's my go to language for building applications (like XML Calabash). I just never drank the “application server” kool aid. If that was the answer, I always used to think, you must be asking the wrong question.

Building my own…

2009-06-03T16:55:11Z

I've always wanted to pick out the components and build my own computer. Maybe the time has come.

I've always wanted to build my own computer, but I've never done it. For years, my primary computer has been a laptop and it's not really practical to home-build one of those. For servers, I've just muddled along with spare boxes and cast-offs. Extra disk and memory in almost anything running Linux will get you a long way.

My current cobbled together setup involves a slow, noisy desktop box in a closet and a handful of USB enclosures on my desk. Not making me happy. So what do I want?

I'd like a walloping big slug of disk space, in some sort of redundant/RAID configuration for storage and backups.
I'd like the system to function as a media server (for some vague definition of what media server means). It'd be nice if I could stream music to my laptop and video to the PS/3 in the living room.
I'd like it to be able to build applications on it. That is, it should run a web server and ideally MarkLogic Server.
I'd like it to be quiet and consume only reasonable amounts of power.
Naturally, I'd like it to be not too expensive.

That's probably the order of priority, too, though quiet is pretty important and I'm pretty cheap.

I'm following what Adam Retter is building. Dave Pawson is building one too (and pointed me to Tom's Hardware for more ideas). Coincidentally, I see from this month's Linux Journal that the Western Digital My Book World Edition is a hackable NAS. That might be the least expensive way to go.

But I still lean towards a Solaris box with four (or more) drive bays running ZFS.

I wonder what the right answer is?

Balisage 2009

2009-06-03T13:41:54Z

Bring out yer demo for free beer!

Only passion, great passion, can elevate the soul to great things.

— Denis Diderot

I'm delighted to learn that the i's have been dotted, the t's crossed, and Mark Logic is now an official sponsor of Balisage 2009. Good on us! You all know what a great conference I think it is, so I won't repeat myself. Much.

In addition to straight-up sponsoring the conference, we're also going to run a “beer and demo jam” on Tuesday night. I'm not sure we've worked out all the details yet, but I know what we're using as a model.

One evening of each of our company meetings is devoted to “Demo Jam” hosted with great panache by Matt Turner (author of Discovering XQuery). All the engineers and consultants are encouraged to demo something. A crowd gathers. Everyone has a few beers and then it's off to the races: five minutes to show anything you want. The winner is judged by the loudest audience response (measured in true geek fashion, as carefully as one can under the circumstances, with a dB meter and everything). It's great fun all around.

So, I don't know exactly what we'll do for beer and demos at Balisage, but if you're coming, bring along your favorite project. Obviously it doesn't have to be Mark Logic related, or XQuery related, or even, for that matter, probably, XML related, though I'm sorta guessing most of them will be.

Good cheer and free beer, FTW!

Miscellany

2009-05-29T17:12:26Z

On speaking engagements (two excellent conferences), anniversaries, and VoIP.

The years teach us much which the days never knew.

— Ralph Waldo Emerson

On the whole, I try not to travel more than I have to. I try to mitigate the travel by planning ahead. My plans this year now officially include (among other things) two marvelous events: Balisage, my absolute favorite XML gig, in Montréal in August, and XML Summer School in Oxford in September. At Balisage, I'll be talking about XProc, at XML Summer School, open source XML applications and Web 2.0 technologies.

Both events offer an opportunity to talk to some of the sharpest markup technogists in the world about the problems, peeves, or practices that you care most about. Highly recommended!

In other news, my calendar reminds me that I decloaked this weblog six years ago today. My output has been down a bit of late, I know, but I still enjoy writing when I have the chance and I have (renewed) plans for tinkering with the technologies that underly the site (more fun for me, perhaps, than you, but an important reason why I do it). Thanks for reading!

Lastly, perhaps leastly in this mixed bag, I've decided to give Skype a try in a serious way, mostly because it's incredibly inexpensive but also because I've been using my cable company's VoIP technology for months now without any problems. (I tried Vonage a few years ago, but it was a total failure for me.)

If you've a reason to call, my Skype ID is my first and last names (“norman” and “walsh”, respectively) all run together. The inbound phone number is +1-413-284-4793. That's a Palmer number, they tell me, basically one town over, because they didn't offer any numbers in Belchertown. (I've only paid for the number through August, 2009. So if it's September 2009 or after, you might want to ask before you call, in case this experiment fails.)

Be seeing you! Or hearing you! Or both!

Perhaps the penultimate XProc draft

2009-05-28T16:44:39Z

Today, the XML Processing Model Working Group published a new working draft. Not the very last working draft, but possibly very close.

For the past few months, the XML Processing Model Working Group has been busily resolving issues raised during the CR. So busy with XProc , in fact, that we forgot about our heartbeat requirement.

Today, we published a new working draft of XProc: An XML Pipeline Language . In addition to editorial improvements and clarifications, this draft contains a small number of significant changes:

Changed and . An output port produces a sequence if that port produces a sequence in any subpipeline.
Changed . Added primary output port result.

Taken together, these two changes make it much easier for pipeline authors to write steps where one of the branches uses a .

Changed . Added arg-separator, path-separator, and failure-threshold. Input can be zero or one document only. Added support for a result code.

Implementor experience revealed that our design for was insufficient. These changes fix problems with platform-specific path separators and dealing with arguments that contain spaces.

Clarified . Interaction with HTTP redirects and cookies is now explicit; the interaction of media types and serialization, several aspects of multipart messages, and a number of other areas have been clarified as well.

The step was the subject of a lot of discussion. We made quite a few changes, almost exclusively clarifications.

Clarified the interpretation of base URIs and the xml:base attribute.

This last change clarifies that the inherent base URI of a node can exist independent of an xml:base attribute. In particular, removing the xml:base attribute does not change the inherent base URI. (Though adding one does change the base URI, of course, so perhaps “independent” wasn't exactly the right word.)

We're almost done

The bottom line is that we really are almost finished. The test suite still needs to be fleshed out, and our implementors need to get to complete coverage, but I think that language evolution is coming to an end.

Ironically, in the time between requesting publication of today's draft and today, the WG identified and corrected one more issue. We added a p:value-available() function to allow pipeline authors to identify options that have no specified value.

Clearly, we can't absolutely promise nothing else will change, but the chair is setting the bar pretty high.

Norman.Walsh.name

XML FTW!

GSD!

XML Prague 2010

NYMUG: Cloud deployment options

XProc: Back to Last Call

David Alfred Walsh

David Walsh, age 10

Bombay c. 1945

Browerville from the tundra, Mar 1960

Camping in Alaska, c. 1960

Sleeping, June 1970

Mom and dad, August 1961

Mom, dad, and I

Playing guitar

David Walsh, Sep 2009

'tis himself

NYMUG Summary

Support for https: and URI rewriting

Office Toolkits

Presentation

NYMUG: New York Mark Logic Users Group!

DocBook V5.0

Evernote

Evernote search

Call for Vote - DocBook V5.0

Micro-blogging Backup, part the fifth

Built my own...

AMD Phenom II X4 Processor Black Edition

Cooling apparatus for the CPU

Motherboard in place

Four of 8Gb

Gobs of space

Optical drive

Case with power supply in place

All systems go!

Glow in the dark

XML Calabash 0.9.15

XML Calabash Logging

XML Summer School ’09

Radcliffe Camera

Oxford sunset

Punters

Got seeds?

Dominican Republic

Punta Cana Beach

Moon Palace Pool

Footprints

OSHA? What's that?

SQL to XML

RDFa for DocBook?

Micro-blogging Backup, part the fourth

Message Formatting

Message threading

URL Rewriting

Micro-blogging Backup, part the third

What's all this invoking stuff about?

What next?

XML+XQuery+Google Voice+Python=WIN!

Micro-blogging Backup, part the second

Create a forest

Create a database

You must attach a forest to the database

Attach the forest you created

Create an HTTP application server

Change the authentication to application-level

CQ shows the empty database

Configure the database

Configure your accounts

CQ shows the database with one document

Download the status messages for your account(s)

CQ shows a database full of documents

Arbitrary XQuery expressions evaluated by CQ

What about my older messages?

Micro-blogging Backup, part the first

Using XML Catalogs and XProc together

Not exactly XProc

Java vs. AJAX

Building my own…

Balisage 2009