The Flickr Uploader has never been a model of stability in my experience. Recently, it's become entirely unusable, it simply doesn't work for the current set of pictures that I want to upload.

There are lots of alternative uploaders, but I'm also interested in exploring the capabilities of XProc. Specifications and test suites aside, you don't really know what you can do with a language until you try.

Perhaps uploading photographs to Flickr isn't the sort of application best achieved by XProc, but I'm certain that interacting with real world web service APIs is something that XProc pipelines will need to do.

Uploading photographs requires authenticated access. As a prerequisite for that, you need to go through a little authentication dance with Flickr. I'm not going to try to automate that part; I'll assume you're in possession of an API key, authentication token, and shared secret. Authenticated access is achieved by “signing” the API call; in brief, concatentate the shared secret with the parameters (sorted into alphabetical order) and their values, compute the MD5 hash of that string, and add that value as a new parameter. Can we do this in XProc?

Sorting the parameters is easy. The only interesting part of that pipeline is the <p:parameters> step. It's there to turn what might possibly be a sequence of documents on the parameters port into a single document for XSLT^[1].

Before we dive into the signing step, there's one more complication. To simplify processing for the caller, I'm going to allow the “secret” to be either a parameter or an option. The Flickr APIs of course don't allow the secret to be a parameter because that would expose the secret which would defeat its purpose. But from an XProc perspective, it's convenient to have a <c:param-set> that contains the key, token, and secret which we can just pass around together.

That means the signing step will have to be prepared for the secret to come from either place and will have to explicitly exclude the secret from the signed parameters.

Let's begin! Our pipeline, flickr:sign-api takes a set of parameters and an optional secret. It returns the signed parameters as a <c:param-set> on the result port.

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:c="http://www.w3.org/ns/xproc-step"
                xmlns:flickr="http://xmlcalabash.com/ns/extensions/flickr"
                xmlns:cx="http://xmlcalabash.com/ns/extensions"
                xmlns:err="http://www.w3.org/ns/xproc-error"
                type="flickr:sign-api" name="main"
                exclude-inline-prefixes="err" version="1.0">
  <p:input port="parameters" kind="parameter"/>
  <p:output port="result"/>
  <p:option name="secret"/>

The first step is to run the parameters through a <p:parameters> step, as discussed above, then we go looking for the secret. This is XProc, so everything has to be XML. The result of the choose is an XML document that contains the secret selected from (preferentially) the secret option or from the secret parameter. It's an error to call this pipeline without passing a secret.

  <p:parameters name="params">
    <p:input port="parameters">
      <p:pipe step="main" port="parameters"/>
    </p:input>
  </p:parameters>

  <p:choose name="secret">
    <p:when test="p:value-available('secret')">
      <p:output port="result"/>

      <p:string-replace match="/secret/text()">
        <p:input port="source">
          <p:inline><secret>@@</secret></p:inline>
        </p:input>
        <p:with-option name="replace" select="concat('&quot;',$secret,'&quot;')"/>
      </p:string-replace>
    </p:when>

    <p:when test="/c:param-set/c:param[@name='secret']">
      <p:xpath-context>
        <p:pipe step="params" port="result"/>
      </p:xpath-context>
      <p:output port="result"/>

      <p:variable name="value"
                  select="string(/c:param-set/c:param[@name='secret']/@value)">
        <p:pipe step="params" port="result"/>
      </p:variable>

      <p:string-replace match="/secret/text()">
        <p:input port="source">
          <p:inline><secret>@@</secret></p:inline>
        </p:input>
        <p:with-option name="replace" select="concat('&quot;',$value,'&quot;')"/>
      </p:string-replace>
    </p:when>

    <p:otherwise>
      <p:output port="result"/>

      <p:error code="err:XX01">
        <p:input port="source">
          <p:inline>
            <message>flickr:sign-api called without secret</message>
          </p:inline>
        </p:input>
      </p:error>
    </p:otherwise>
  </p:choose>

Next we sort them.

  <cx:sort-parameters name="sorted">
    <p:input port="parameters">
      <p:pipe step="main" port="parameters"/>
    </p:input>
  </cx:sort-parameters>

Now we have a <c:param-set> containing all of the parameters in alphabetical order. From this we must construct a string that we can sign. There's no two ways about it, string manipulation in XProc is not as easy as XML manipulation. Here's what we're going to do: for each parameter (except the secret , if it's there) we're going to construct a dummy document that contains the name and value concatenated together. If we pass in <c:param name="foo" value="bar"/>, we'll pass back <doc>foobar</doc>.

There's one interesting complication here. We're going to use <p:string-replace> to do the heavy lifting, but that will make the context node the node that we're replacing. From that context node, we'll have no way to get back to the <c:param> that's on the current port. We work around this by storing the values we need in variables.

  <p:for-each name="loop">
    <p:iteration-source select="//c:param[@name != 'secret']"/>

    <!-- Flickr params should always be NCNames, but just in case... -->
    <p:variable name="name"
                select="if (contains(/*/@name, ':'))
                        then substring-after(/*/@name, ':')
                        else string(/*/@name)"/>

    <p:variable name="value" select="string(/*/@value)"/>

    <p:string-replace match="/doc/text()">
      <p:input port="source"><p:inline><doc>@@</doc></p:inline></p:input>
      <p:with-option name="replace"
                     select="'concat(&quot;',$name,'&quot;,&quot;',$value,'&quot;)'">
        <p:pipe step="loop" port="current"/>
      </p:with-option>
    </p:string-replace>
  </p:for-each>

The output of the loop step is a sequence of <doc> documents. We'll turn that back into a single document by adding a wrapper and we'll prepend the secret value to the beginning of the document.

  <p:wrap-sequence wrapper="inner-wrapper"/>

  <p:insert match="/inner-wrapper" position="first-child">
    <p:input port="insertion">
      <p:pipe step="secret" port="result"/>
    </p:input>
  </p:insert>

We're getting there! Now we've got something that looks like this:

<inner-wrapper>
<secret>somehexvalue</secret>
<doc>api_keysomelonghexvalue;</doc>
<doc>api_tokenanotherlonghexvalue;</doc>
<doc>methodsomemethodname;</doc>
</inner-wrapper>

Recall your XPath semantics, if we ask for the string-value of that document, we'll get the string we want hash. We have a <p:hash> step, so we can do that part. But if we replace the wrapper with the hash of it's string value, the result won't be well-formed because it won't have a document element anymore. We get around that by adding another level of wrapper first.

  <p:wrap-sequence wrapper="wrapper"/>

  <p:hash match="/wrapper/inner-wrapper" algorithm="md" version="5" name="hash">
    <p:with-option name="value" select="string(.)"/>
  </p:hash>

The output of <p:hash> is a document that looks like this:

<wrapper>somehexstring</wrapper>

Now if we can stick that back in a <c:param> element and stick that parameter back into the <c:param-set>, we're done. We'll use <p:string-replace> to put the string back into an element, but here we encounter the first really odd wrinkle in the XProc design.

Like before, we need to put the hash in a variable so that we can get at it when the context is changed by <p:string-replace>. But variables can only appear at the beginning of a compound step. Therefore, we have to introduce a <p:group>.

WTF!? I hear you cry. Yes, well, look at it this way: the order of steps in the pipeline is determined by the connections between them, not strictly by the order in which they appear in the pipeline document. If you put a variable declaration between to steps and those steps get moved, where does the variable go? In order to answer that question, you have to know how it fits into the flow graph. To answer that question, you'd have to parse and analyze the XPath expression that defines its value because that variable's value might depend on the values of other variables. That was more than we were willing to demand of implementations. Instead, we simply limited variables to the beginning of compound steps. That assures that ordinary step analysis produces a consistent result for variables. Most of the time, it's what you want to do anyway. But every now and then, you run into this weird situation where you have to insert a <p:group>. Imperfect, I agree.

C'est la vie. Here's the <p:group>, the variable, and the <p:string-replace> call.

  <p:group>
    <p:variable name="hash" select="string(/)">
      <p:pipe step="hash" port="result"/>
    </p:variable>

    <p:string-replace match="/c:param/@value" name="api_sig">
      <p:input port="source">
        <p:inline><c:param name="api_sig" value="@@"/></p:inline>
      </p:input>
      <p:with-option name="replace" select="concat('&quot;',$hash,'&quot;')"/>
    </p:string-replace>

The last step is to combine this new signature parameter with the original input parameters. Remember that the user may have passed the secret in as a parameter, so we start by deleting that one.

    <p:delete match="c:param[@name='secret']">
      <p:input port="source">
        <p:pipe step="params" port="result"/>
      </p:input>
    </p:delete>

Then we insert the signature into the resulting <c:param-set> and that's what we send out the result port.

    <p:insert match="/c:param-set" position="last-child">
      <p:input port="insertion">
        <p:pipe step="api_sig" port="result"/>
      </p:input>
    </p:insert>
  </p:group>
</p:declare-step>

The input parameters have been augmented with a signature that will satisfy the Flickr authentication requirements. Now we're read to move on to actually using the API with our signing pipeline.

In a future essay we'll look at making a “Flickr service” pipeline and an upload pipeline.

http://norman.walsh.name/2010/04/12/xprocbook 2010-04-12T16:51:34Z 2010-08-30T19:52:29Z XProc XProcBook

I guess it's official, I'm writing another book.

I guess it's official, I'm writing another book.

After I wrote Making TeX Work, I wondered if I'd ever write another book. The effort required to get from a seventy page draft to a finished book was much larger, and harder, than I imagined.

Then I wrote DocBook: The Definitive Guide, and could honestly claim not to be a one-hit wonder in publishing. O'Reilly's even publishing the second edition, covering DocBook V5.0!

I really wasn't sure I'd ever write a third book. But XProc looks to be ramping up in the world, and Richard Hamilton ’s been giving me little nudges to write one for his XML Press. Sometime over the last couple of months, the possible thought of maybe doing that turned into a few pages of outline then a few dozen reference pages then a nice (XProc-based!) toolchain for building it.

I guess I'm writing another book.

You can follow along, if you're so inclined, at http://xprocbook.com/. That's pretty rough and it's early days. There's no actual text up there yet, but that's got more to do with laziness on my part than anything else.

For my first two books, O'Reilly's art department provided the cover design and all the details. For this one, more of it's on my plate. So here's my first idea: crowd-source the cover. If you've got an image (of your own, naturally) or other design that you think would make a good cover, please let me know. You'll get a cover credit and my gratitude if we pick your design for the final cover!

http://norman.walsh.name/2010/04/09/xmldefault 2010-04-09T15:27:38Z 2010-08-30T19:52:29Z XML XProc

A look at the intersection of the XML model PI, the XML stylesheet PI, and XProc.

A look at the intersection of the XML model PI, the XML stylesheet PI, and XProc.

What is the “default XML processing model?” That question has been open for a long time, since the very beginning of XML really. There are a lot of different opinions, some of them captured in the W3C’s Technical Architecture Group discussion of the issue “xmlFunctions-34”. (Disclaimer: I contributed to that issue while I was a member of the TAG.)

It's in the charter of the XML Processing Model Working Group , which I chair, to provide an answer to this question. I don't think it has an answer. I don't subscribe to the notion that XML documents have one and only one intrinsic meaning. I think the best we can do is describe one (or a few) possible models and give them labels. That will allow the authors of other specifications, and applications, to say “we do TYPEX processing on XML documents”, where “ TYPEX ” is one of the labels. That'll give us a shorthand for talking about some common processing models.

That may not seem very satisfying. Maybe it isn't. The point of this essay isn't to make or defend that position. When the WG produces it's first public working draft of a document that attempts to answer the “default XML processing model” question, I'll let you know. The right answer isn't what I think it is, it's what community consensus drives us to.

No, the point of this essay is something else, something a little more complicated than I think we would reasonably expect to put in that document (though I could be wrong).

The XML community has had an Associating Style Sheets with XML documents specification for a long time. It will soon have an Associating Schemas with XML documents specification. (That link is to an early editor's draft, there's nothing official yet, but it's coming soon.)

What are the two most common things that many (not all!) users want to do with XML documents? Validate them and transform them.

Well, if the document tells you how to validate it and how to style it, then isn't one possible answer to the default processing question simple: validate like I say and style like I say? If it is, shouldn't we be able to express that processing using an XProc pipeline?

Of course we should. And we can: default.xpl.

I'm happy and relieved to find that we can express that processing in XProc. I'm a little, but only a little, surprised to see how complex that pipeline is. Weighing in at 320+ lines it works for a few narrow cases. I still need to integrate support for RELAX NG compact syntax schemas and NVDL processing, at least. I may also want to support a few more stylesheet options, I'm not sure.

http://norman.walsh.name/2010/04/08/mluc10-demojam 2010-04-08T13:55:10Z 2010-08-30T19:52:29Z MLUC10 MarkLogic

Bring out yer demos! We're going to jam! This is your chance to show off in front of the largest annual gathering of XML and XQuery experts in the world at the Mark Logic User Conference.

Bring out yer demos! We're going to jam! This is your chance to show off in front of the largest annual gathering of XML and XQuery experts in the world at the Mark Logic User Conference.

With two successful demo jam's under my belt (XML Prague this year and Balisage last year), marketing has decided to put me on stage with Matt Turner at the big show: we're going to run “Demo Jam” at our annual user conference.

We've never had a shortage of entries (despite my sometimes fear that we will), and our user conference is big, so we're going to introduce a little process this time with a “call for demos”. We'll pick ten entrants to compete at the event. Top prize for your trouble: a shiny new iPad.

Anyone attending the conference can compete, so register today and propose your demo!

What's a demo jam? 

It works like this: you get a microphone, a projector, and 5 minutes to show off your stuff. After all the demos have run, the audience cheers for their favorite. Loudest ovation walks away with the ’pad.

The only hard-and-fast rule of demo jam is you get five minutes, 300 seconds, not a second more.

Now, in my best high-speed, voiceover-lawyer voice, here are the official contest rules:

DEMO JAM!

Attend our DemoJam at the opening reception! Submit your best 5 minute demo now, present it on May 3rd, get the cheers of the attendees and win an iPad!

Mark Logic will hold a demo contest, on May 3rd, 2010 at the Mark Logic User Conference opening reception. Everyone is welcome! The demo must be of technical nature and cannot take more than 5 minutes. The crowd will vote on the winner.

DemoJam started as an internal Mark Logic contest that we have expanded to include our developer community. Because of the great enthusiasm this event receives, we have to put a ceiling on the number of demos to be presented. Submit your demo now and we will invite the top 10 to compete at the User Conference event.

Submit your demos by April 21st at demojam at marklogic dot com. Submissions should include a brief textual description of the demo. You're welcome, but not required, to provide screen shots or screen casts as well.

Mark Logic will provide an Internet connection but presenters are encouraged to bring their own laptops.

It's great fun. Hope to see you there, on stage or in the audience!

http://norman.walsh.name/2010/04/06/engineering 2010-04-06T20:21:42Z 2010-08-30T19:52:29Z MarkLogic SelfReference

I had a lot of fun last week hanging out with my new colleagues in engineering. Not that you should care. The part you should care about is: we're hiring.

I had a lot of fun last week hanging out with my new colleagues in engineering. Not that you should care. The part you should care about is: we're hiring.

As I mentioned a month or so ago, I've changed jobs inside Mark Logic. I've moved from a pre-sales/consulting field position in the media team to the application services team in engineering. The short version of the story about how this happened is that Micah Dubinko , whom most of my regular readers will certainly know, made a compelling offer: a chance to work with a great team, a chance to build cool stuff, and a better opportunity, really, to make a lasting difference for the company and our customers. I think. (But I still love our media peeps and I'll always be a “publishing guy”.)

It's been more than a decade since I spent a week just hanging out with engineers in the office. That is a lot of fun. Not “I'm ready to relocate” fun, I'm going to continue to be a full-time work-from-homer, but still fun and stimulating.

So I had fun (yay me!), but that's not really the point. The point is that you can have fun too! If you're smart and motivated and want to work on an amazing product with the largest collection of uniformly great people I've ever met, we should talk.

I can tell you that my group is looking for at least two people. And, lest you think working on MarkLogic Server is all behind-the-scenes heavy coding, we'd particularly like to talk to someone with the sort of UI/JavaScript/HTML experience that can really make an application shine. Oh, and if you really love to be behind-the-scenes doing heavy coding on a honking big, screaming fast, hugely scalable powerhouse, we can hook you up there too! And in the field, if that's your thing. I don't know all the details, but I can put you in touch with the folks who do.

Anyway, the short story is, I'm having the best time and I like to share. There's plenty of room to join in the fun! If you're interested, get in touch.

http://norman.walsh.name/2010/03/18/rng2dtd 2010-03-18T19:38:13Z 2010-08-30T19:52:29Z DocBook W3CXMLSchema XML XProc

Taking another stab at the long-standing problem of producing DTD (and XSD) versions of the DocBook V5.0 family of schemas.

Taking another stab at the long-standing problem of producing DTD (and XSD) versions of the DocBook V5.0 family of schemas.

In the course of preparing the DocBook V5.0 schemas, I devised a process that would convert the DocBook V5.0 RELAX NG grammar into an XML DTD (we get the XSD by running Trang over the DTD). Closer inspection quickly reveals two flaws in this process:

1. It's incredibly brittle; while it successfully converts the base DocBook schema, it's utterly useless on even a simple customization layer.

2. It produces utterly crap DTDs.

The former problem is the one that's really causing me pain, though I admit I'm a little embarrassed by the second.

The Publishers Subcommittee identified an XML DTD as a requirement (Come on tool vendors! Get your act together. It's the twenty-first fscking century, already!). My hopes that someone else would fix the problem before I got to it went unfulfilled, so over the last few days I've turned my attention seriously to the problem. A few observations:

• The general problem may be insoluble. It may simply be that there's no algorithmic path from the simple and expressive constraints of RELAX NG to the much less expressive constraints of DTDs. Or maybe there are several paths, but the results will never look rational to a human observer. Or maybe the solution is out there, just waiting for some enterprising grad student to find it (nudge, nudge). I'm not sure. I decided that I didn't have time to look for that solution.

• If the computer can't solve the whole problem, then we'll have to rely on human intervention to solve the hardest parts. In the DocBook family of schemas, the apparent hard parts are attribute co-constraints and multiple patterns for the same element name.

• The solution in both cases is to replace the problematic patterns with a single pattern that is the union of the various options. This creates content models that are too broad: they accept all valid documents, but they also accept some invalid documents. The union of the CALS and HTML table models, for example, is a model in which a <tbody> element can contain a mixture of CALS <row> elements and HTML <tr> elements, among other atrocities. Such was it always with XML DTDs.

• The constraints on mixed content in XML DTDs are a pain in the *ss. In a DTD, mixed content must be expressed as (#PCDATA | a | b | c | …)*. The #PCDATA token must come first, the alternatives must be at the top level (no nested parenthesis), and there must be no duplicates among the alternatives.

• There are a few places where we allow extensions in “other namespaces.” The <info> elements can contain arbitrary additional metadata elements, for example, and equations can contain any MathML markup. DTDs and namespaces do not play well together. It might be possible to create a DTD that allowed MathML in the appropriate places, but that's more than the minimum needed to declare victory.

• Patterns are a little bit like parameter entities. (Ok, a really, really little bit.) It would be nice, where possible, to represent the patterns as parameter entities in the resulting schema. At worst, it does no harm, at best it makes the DTD easier to read and may allow some small amount of customization of the DTD, not that I'd recommend that! And in any event, will solve at least a tiny part of the second problem mentioned above.

With these things in mind, I decided to adopt the following approach:

1. Create a “DTD” customization layer in RELAX NG that removes the most difficult problems: create unions for the attribute co-constraints, create unions for the elements that are defined by several patterns, remove the elements in “other namespaces” extension points, etc.

2. Create an “override” document for describing a few more operations. For example, removing the db._phrase pattern and changing all the patterns that use it so that they use db.phrase instead. (If there's an easy way to accomplish that in the RELAX NG customization layer, but it eluded me.)

3. Massage the modified schema until it's possible to create a DTD from it.

I'd like to say that there was some deep, theoretical insight in the last step, but there wasn't. I just built a pipeline of transformations that got me from A to B. I looked at the document, found something that wouldn't work in a DTD, wrote a transformation to remove it, and added that transformation to the pipeline. Repeat until done. When the result was accepted by an XML parser and accepted a small, valid DocBook document, I called it done.

Here's a 10,000 foot summary of the process.

1. Starting with the “DTD” customization layer, perform some simplifications. Discard documentation, schematron rules, the start pattern, divisions, etc. Turn interleaves into choices; this is a little risky, but seems to be ok in the DocBook family of schemas. Extract the content of pattern definitions, producing a set of elements and a set of “parameter entities”. Drop schema facet constraints and not allowed content on the floor. Etc.

2. Apply the overrides, as described above.

3. Remove “choice” wrappers from around attributes. Fiddle with how “optional” is expressed. In RELAX NG, it's a wrapper, for the some steps in this process, it's more convenient to make it an attribute.

4. Remove parameter entities that are no longer referenced.

5. Fiddle with “optional” again, moving the optionality down to the references. (An optional reference to something is the same as a reference to an optional something.)

6. In subsequent steps, it's going to be convenient to be able to distinguish references to attributes from other references, so turn all <ref> elements that point exclusively to attributes into <attref> elements.

7. Flatten chains of references to attributes. (If A points to B points to C points to D which is an attribute, then just make A point to D.)

8. Fiddle with “optional” again. This time move the optionality up to the attribute declaration. This may require splitting a declaration.

9. Check for element names defined by more than one pattern. There better not be any.

10. Remove “empty” parameter entities and references to them.

11. Pull “text” up. Replace any reference to a parameter entity that contains #PCDATA with a copy of what the parameter entity contains.

12. Unwrap nested “zero-or-more” elements.

13. Sort the parameter entities so that we never attempt to use one before it's been declared.

14. Convert the resulting document into a DTD. Turn parameter entities into !ENTITY declarations, turn elements into !ATTLIST and !ELEMENT declarations, substitute DTD attribute types for the specified types, expand mixed content models, etc.

The most disappointing part of that last step is fully expanding the content model of every element that contains mixed content. I'd been working pretty hard all along to preserve as many pattern names as possible as parameter entities.

The problem is that even though all the relevant parameter entities are simple lists of elements (so they could appear in a mixed content element declaration), sometimes the same element name appears in more than one pattern. So I punted, expanded them all, and removed duplicates. I still think it might be possible to do better.

In retrospect, this isn't too surprising. If you go back to the DocBook V4.x DTDs and study the parameter entity structure [“masochist” -ed], you'll find a few places where we twisted the parameter entity structure pretty hard to avoid exactly this problem.

In any event, the DTD that results from this process is an XML DTD that appears to validate DocBook documents. With different customization and overrides, the DTD version of the publishers schema also seems to work.

I'll get it out in the next day or so for wider testing. It's very likely that there are places where it's not quite right. But it's definitely an improvement over the old process.

Pipeline Notes 

The process described above is not wholly unlike what I did before. One significant factor that made this attempt more successful was XProc . It's not impossible to chain together 14 transformations with a big XSLT 2.0 stylesheet and a bunch of modes, but it's a whole lot harder to manage.

Speaking of XProc, I cheated. The pipeline I'm using today will only work in XML Calabash because it relies on a compound extension step: cx:until-unchanged. That step is a little bit like p:for-each except that after each iteration it compares the input document to the result of applying the pipeline and repeats the process (using the output of one iteration as the input of the next) until the result is the same as the input.

It's not impossible to do this without extending XProc, but it requires writing a different recursive pipeline for each looping step. It was more interesting (for me) to see how hard it would be to write a compound extension step. (So shoot me.)

By the way, if you're curious, converting the base DocBook schema to a DTD is a 40 step pipeline (more or less):

INFO: Running pipeline main
INFO: Running xslt rng2dtx
INFO: Running xslt override
INFO: Running cx:until-unchanged remove-choice
INFO: Running xslt attr-remove-choice
INFO: Running xslt attr-remove-choice
INFO: Running xslt attr-remove-choice
INFO: Running xslt attr-remove-choice
INFO: Running cx:until-unchanged remove-unused
INFO: Running xslt attr-remove-unused
INFO: Running xslt attr-remove-unused
INFO: Running xslt attr-remove-unused
INFO: Running xslt attr-optional-to-ref
INFO: Running cx:until-unchanged to-attref
INFO: Running xslt ref-to-attref
INFO: Running xslt ref-to-attref
INFO: Running cx:until-unchanged flatten
INFO: Running xslt flatten-attref
INFO: Running xslt flatten-attref
INFO: Running xslt flatten-attref
INFO: Running xslt attr-optional-to-decl
INFO: Running xslt multiple-gis
INFO: Running xslt remove-empty-pes
INFO: Running cx:until-unchanged pull-up
INFO: Running xslt pull-up-text
INFO: Running xslt pull-up-text
INFO: Running xslt pull-up-text
INFO: Running xslt pull-up-text
INFO: Running cx:until-unchanged unwrap
INFO: Running xslt unwrap-zeroormore
INFO: Running xslt unwrap-zeroormore
INFO: Running cx:until-unchanged sort
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt sort-pe
INFO: Running xslt dtx2dtd

What about a better W3C XML Schema? 

The DTD that results from this conversion process is a little bit unsatisfying. It just not what a human being would do if they started from scratch. On the other hand, it doesn't matter much; there's no widespread use for DTDs beyond validation and perhaps guided authoring. (And you ought to be using RELAX NG for that, see previous comment about the twenty-first century.)

The same is not true of W3C XML Schemas. There would (just possibly, maybe) be value in having a better XSD for DocBook. There are data binding tools and other applications that would fare much, much better with DocBook if they were given something that took proper advantage of XSD's native facilities.

I've never had much interest in writing an XSD for DocBook. I'm not likely to ever be persuaded that “type inheritance” is a satisfying abstraction for how content models are related. I never felt that XSD 1.0 was a good foundation for the kind of “human prose” schemas of which DocBook is a typical example. But I'm almost convinced that XSD 1.1 has fixed some of the most inconvenient deficiencies.

My bailing wire and duct tape solution for generating DTDs doesn't seem like it's ever going to be up to the task of doing the conversion properly. I'd be delighted if the aforementioned enterprising grad student built a tool to do the conversion automatically, but if not, I just might (someday) take a crack at hand authoring a proper XSD 1.1 schema for DocBook.

http://norman.walsh.name/2010/03/10/XProcProposedRecommendation 2010-03-10T11:01:16Z 2010-08-30T19:52:29Z W3C XML XProc

I'm pleased to report that XProc: An XML Pipeline Language is now a W3C Proposed Recommendation.

I'm pleased to report that XProc: An XML Pipeline Language is now a W3C Proposed Recommendation.

The Proposed Recommendation draft of XProc: An XML Pipeline Language was published yesterday!

I'll save a more reflective post about the process and the result for after we've crossed the last hurdle. In the meantime, here are a few useless statistics. Between start and PR:

• We had 169 meetings over 4 years and 73 days, give or take a day or two.

• We produced 12 drafts: seven working drafts, two last call working drafts, two candidate recommendations, and one proposed recommendation.

• I stood on the podium in front of a conference audience and asserted that we'd be finished “within a year” on at least three separate occasions. Maybe four.

• We had four face-to-face meetings. Murray kicked us of in style at his place and we met at three W3C Technical Plenary meetings.

• The community generated two complete, interoperable implementations and a number of additional, partial implementations.

And, most important, we developed an active (and growing, I think) XProc user community. On the whole, a success by any metric, I think.

Once again, I'd like to extend my congratulations and heartfelt thanks to the members of the Working Group, reviewers, and implementors that have helped us come this far. We couldn't have done it without you.

http://norman.walsh.name/2010/03/07/wikiEdit 2010-03-07T21:25:44Z 2010-08-30T19:52:29Z Calabash TheWeb XProc

An example, for better or worse, of automating website interaction with XProc.

An example, for better or worse, of automating website interaction with XProc.

What happened was, the DocBook wiki broke. I don't know how or why, but it fell over. The problem, whatever it is, left the wiki immutable and the underlying database in a state of questionable consistency.

Clearly a problem that had to be fixed. I setup a new wiki, running MoinMoin 1.9.2 instead of 1.3.4 [Upgrade much? -ed].

In theory, there's an upgrade path from 1.3.4 to 1.9.2 but I'm sufficiently unsure about the state of the current database that I'm loathe to use it. The last thing I want to do is put the new wiki into some indeterminate state. Instead, I grabbed all the most recent pages from the old wiki, trimmed out a bunch of cruft, and cleaned up the markup a bit (the wiki markup seems to have changed over time).

What I really wanted to do was add all these pages to the new wiki. Easy enough to do with a browser for one or two pages, but several hundred pages was way more than my patience would tolerate.

A quick experiment with HTTP Scoop made it it look pretty easy:

• Logging in sets a cookie.

• Loading a page that doesn't exist provides a link that you can follow to create the page.

• Following that link returns an HTML page containing a form with a place to type the wiki markup and a bunch of hidden fields.

• Posting that form back to the server updates the page.

If only I had a tool that could make HTTP requests and process the results…wait, wait, I have one of those!

XProc ought to be up to this job, yes? Yes! In fact, it was reasonably straightfoward. Wanna see how it works? Of course you do. The following pipeline works in XML Calabash version 0.9.20 or later.

I decided to pass the wiki markup as an input and the page name as an option. From the option, I construct the value of the URI for the page.

<p:declare-step version='1.0' xmlns:p="http://www.w3.org/ns/xproc" name="main"
                xmlns:cx="http://xmlcalabash.com/ns/extensions"
                xmlns:c="http://www.w3.org/ns/xproc-step"
                xmlns:html="http://www.w3.org/1999/xhtml">
  <p:input port="source"/>
  <p:output port="result"/>
  <p:option name="page" required="true"/>

  <p:variable name="pageuri" select="concat('http://wiki.example.com/',$page)"/>

Next I have to login:

  <p:www-form-urlencode match="/c:request/c:body/text()">
    <p:input port="source">
      <p:inline>
        <c:request method="POST"
                   href="http://wiki.example.com/DocBookWikiWelcome">
          <c:body content-type="application/x-www-form-urlencoded">@@HERE@@</c:body>
        </c:request>
      </p:inline>
    </p:input>
    <p:input port="parameters">
      <p:inline>
        <c:param-set>
          <c:param name="action" value="login"/>
          <c:param name="name" value="NormanWalsh"/>
          <c:param name="password" value="MYPASSWORD"/>
          <c:param name="login" value="Login"/>
        </c:param-set>
      </p:inline>
    </p:input>
  </p:www-form-urlencode>

  <p:http-request cx:cookies="login" name="login"/>

  <p:sink/>

I reverse engineered the way the login form works. I URL encode and pass my username, password, and other parameters to a <p:http-request> that POSTS them to the server.

I don't care about the result, so I drop it on the floor with <p:sink>.

I do care about cookies, so I have to store those somewhere. XML Calabash has an extension that lets you manage cookies in named sets. This <p:http-request> saves any cookies that come back in the “login” set.

Next, we have to get the page we want to edit.

  <p:string-replace match="/c:request/@href" cx:depends-on="login">
    <p:input port="source">
      <p:inline>
        <c:request method="GET" href="@@HERE@@"/>
      </p:inline>
    </p:input>
    <p:with-option name="replace" select="concat('&quot;', $pageuri, '&quot;')"/>
  </p:string-replace>

  <p:http-request cx:cookies="login" name="getpage"/>

  <p:sink/>

I use the “login” cookies so that the wiki knows who I am. I also use the cx:depends-on attribute to tell the processor that this step depends on the preceding login step, even though there's no dependency in the flow graph. Without this explicit statement about dependency, the processor might attempt to get the page before performing the login step.

Once again, I don't care about the output so I drop it on the floor. In theory, I have to parse the output and find the “edit” link. In practice, I know how to create it without looking for it in the markup. I'm not even sure I have to do this step, but it is what a browser does and it was easy to do so I left it in.

Now we want to get the page that includes the edit form:

  <p:string-replace match="/c:request/@href" cx:depends-on="getpage">
    <p:input port="source">
      <p:inline>
        <c:request method="GET" detailed="false" href="@@HERE@@"/>
      </p:inline>
    </p:input>
    <p:with-option name="replace" select="concat('&quot;', $pageuri, '?action=edit&quot;')"/>
  </p:string-replace>

  <p:http-request cx:cookies="login" name="getpageedit"/>

Again, we use the login cookies. And this time we don't drop the output on the floor because we have to extract the hidden fields from the page in order for our subsequent POST to work.

  <p:unescape-markup namespace="http://www.w3.org/1999/xhtml"
                     content-type="text/html" name="unescape"/>

  <p:for-each name="for-each">
    <p:iteration-source select="//html:input[@type='hidden']"/>
    <p:output port="result"/>

    <p:string-replace match="c:param/@name">
      <p:input port="source">
        <p:inline><c:param name="name" value="value"/></p:inline>
      </p:input>
      <p:with-option name="replace" select="concat('&quot;',/*/@name,'&quot;')"/>
    </p:string-replace>

    <p:string-replace match="c:param/@value">
      <p:with-option name="replace" select="concat('&quot;',/*/@value, '&quot;')">
        <p:pipe step="for-each" port="current"/>
      </p:with-option>
    </p:string-replace>
  </p:for-each>

To get the hidden fields, we unescape the markup. XML Calabash uses TagSoup for “text/html” pages, so we'll get well-formed XML.

The <p:for-each> loop selects each of the hidden input fields and transforms them into <c:param> elements. We'll need those later.

Next, we have to construct the <c:param> for the “savetext” parameter that contains our wiki markup. This one's a bit tricky.

  <p:string-replace name="savetext" match="/c:param/@value">
    <p:input port="source">
      <p:inline>
        <c:param name="savetext" value="@@HERE@@"/>
      </p:inline>
    </p:input>
    <p:with-option name="replace" select='concat("&apos;",replace(c:data,"&apos;","&apos;&apos;"),"&apos;")'>
      <p:pipe step="main" port="source"/>
    </p:with-option>
  </p:string-replace>

What the hell, I hear you ask, is up with that “replace” value?

Well, see, what's going to appear on the source input port of our pipeline is a <c:data> element that contains the wiki markup of the page. The replace option is interpolated as an XPath expression, so we have to “quote” the value. This is a common idiom in <p:string-replace> ^[1]. Except, in this case, the value may contain both double and single quotes, so we need to make sure that they don't result in an invalid XPath expression!

Imagine that this is our <c:data> element:

<c:data>"Hello 'world'"</c:data>

If we do the usual quoting trick, the resulting XPath expression will be:

'"Hello 'world'"'

and that's not a syntactically valid XPath string value. So we use replace to double-up the apostrophes. That gives us

'"Hello ''world''"'

which is what we want. That took me a minute or two, believe you me.

Next we wrap all our <c:param> elements in a <c:param-set>, construct a <c:request> to hold them, and use <p:www-form-urlencode> to encode them.

  <p:wrap-sequence name="wrap" wrapper="c:param-set">
    <p:input port="source">
      <p:pipe step="for-each" port="result"/>
      <p:pipe step="savetext" port="result"/>
    </p:input>
  </p:wrap-sequence>

  <p:string-replace match="/c:request/@href" cx:depends-on="wrap">
    <p:input port="source">
      <p:inline>
        <c:request method="POST" detailed="true" href="@@HERE@@">
          <c:body content-type="application/x-www-form-urlencoded">@@HERE@@</c:body>
        </c:request>
      </p:inline>
    </p:input>
    <p:with-option name="replace" select="concat('&quot;', $pageuri, '&quot;')"/>
  </p:string-replace>

  <p:www-form-urlencode match="/c:request/c:body/text()">
    <p:input port="parameters">
      <p:pipe step="wrap" port="result"/>
    </p:input>
  </p:www-form-urlencode>

Send that off to the server and we're done!

  <p:http-request cx:cookies="login"/>

  <p:delete match="/c:response/*"/>

</p:declare-step>

I display the result, after deleting its contents, just to make sure that I got a 200 back.

That little XProc script got all the pages loaded in just a couple of minutes. FTW!

If you're interested, the whole script is available.

---

^[1]So common that I regret not providing some sort of syntactic shortcut for it. Oh, well, there's always version 1.1.

http://norman.walsh.name/2010/03/06/where 2010-03-06T21:57:48Z 2010-08-30T19:52:29Z SelfReference

Or, perhaps more to the point, where was I? And where will I be?

Or, perhaps more to the point, where was I? And where will I be?

I've long been fascinated by geospatial data. I do a little geocaching . I keep track of the countries I've been in. I use services like Dopplr and TripIt to keep track of my itineraries. I carry a GPS to geotag photographs.

Carrying a mobile phone with a GPS allows me to explore the features of geosocial applications like BrightKite, Gowalla, and Foursquare. I allow Dopplr and Brightkite to update my Fireeagle location.

All very nice, but there were two obvious (to me) deficiencies in this arrangement. First, and most obvious, my data is spread all over someone else's servers. I consider this unacceptable. Any one of these services could get bought, go belly up, or carelessly (or maliciously, I suppose) discard all my data.

The second point is less obvious: how do I tell when I'm home? I don't actually think any of you reading this would have a hard time working out where I live. I'm sure I've left enough digital clues for anyone sufficiently interested to work it out. That said, I can't actually convince myself that it's reasonable to “check in” to any of these geolocation services when I'm home. I'm not sure it's reasonable to do when I'm not home, either, but I do. I think I've set the services up so they only reveal my location to friends anyway.

Not being able to solve the second of these problems made the data sufficiently unreliable (I've been checked into Staples for eleven days?) that the first problem hadn't crossed my “do something about it” threshold.

All that changed a few days ago when Tom Morris happened to mention a clever solution in his Twitter stream. He later documented it in a thoughtful post. The basic idea is this: if you always carry your phone (or other device) on your person, then the presence of that device in your house means your home.

A little hack later and my server always knows when I'm home, give or take a few hours; it can only ping the device when it's on and not in “standby” so there's some latency. But not more than a few hours most days, I expect.

Having solved the second problem, I turned my attention to the first. A few hours of hacking later and the TripIt, Foursquare, Gowalla, Brightkite, and Fireeagle APIs are giving me my data. The hardest part, honestly, was getting over the authentication hurdles. OAuth may be the right answer, but it's not painless to setup a new application.

I started out by pouring all this data into MarkLogic Server. (Well, I would, wouldn't I?). A little XQuery later and I had a normalized view of all my locations. Cool.

But wait, I thought, what about all those GPS tracks? Yes, those belong in there as well! Easily done.

The interesting thing about GPS tracks is that you can (sometimes) interpolate data between points. I do this already when I'm geotagging photographs. By adding “next point” to the normalized data when appropriate, I could expose that in my system as well.

Once that idea was in place, it was clear that an airline flight or train ride (to a lesser extent) might be subject to interpolation as well. A quick tweak to the scripts that normalize TripIt itinerary data took care of that.

At the end of the day, I have an interesting (to me) personal archive of my geolocation over time. It's derived from GPS tracks, explicit checkins, and itineraries. I'm also going to integrate the GPS data that comes from photographs taken with my mobile phone. All very cool to me.

I've also got a web service that I can use for geotagging photographs. I can ask, for example, where was I this morning at 10:00a?

<point lat="42.360633" long="-72.543451"
       timestamp="2010-03-06T14:51:53Z"
       duration="PT8M7S" seconds="487">Staples</point>

Apparently, I was at Staples and had been for 8 minutes. No wonder I'm the mayor of Staples.

If I ask where I was at 2006-07-15T13:28:00Z, the answer comes from a GPS track:

<path start-lat="42.376713753" start-long="-72.516739368"
      end-lat="42.376821041" end-long="-72.516653538"
      timestamp="2006-07-15T18:27:58Z" end-timestamp="2006-07-15T18:28:03Z"
      total-distance="0.00842566695064306"
      total-duration="PT5S" total-seconds="5" velocity="6.06648020446301"
      duration="PT2S" seconds="2" distance="0.00337026678025723"
      lat="42.3768" long="-72.5167"/>

That's a location interpolated over five seconds and about 44 feet. Seems pretty reasonable.

Interpolating over airline flights is a little less precise:

<path start-lat="42.363611" start-long="-71.006111"
      end-lat="18.3375" end-long="-64.969444"
      timestamp="2010-01-28T14:05:00Z" end-timestamp="2010-01-28T18:00:00Z"
      total-distance="1697.2841796875"
      total-duration="PT3H55M" total-seconds="14100" velocity="433.349152260638"
      duration="PT55M" seconds="3300" distance="397.236722905585"
      lat="36.7753" long="-69.2873">BOS</path>

but it's still kind of cool. Also interesting is the fact that itinerary data lets me look forward. Where will I be at 2010-03-12T08:25:00Z?

<point lat="50.1" long="14.266667"
       timestamp="2010-03-12T08:25:00Z"
       duration="P1DT1H55M" seconds="93300">PRG</point>

Oh yes. I think I have an appointment there. And it'll be more accurate after I've actually checked in, taken photographs, and used my GPS. Sweet.

http://norman.walsh.name/2010/03/01/driveKnows 2010-03-01T20:39:11Z 2010-08-30T19:52:29Z Laptop OSX SelfReference

I recently had occasion to swap hard drives between two essentially identical laptops. A surprising number of apps knew the difference.

I recently had occasion to swap hard drives between two essentially identical laptops. A surprising number of apps knew the difference.

I have two essentially identical laptops, loaded 17” MacBook Pro's. One is my personal machine and one belongs to Mark Logic. I use my personal machine most of the time, but the fans have gotten insanely loud. I reported this as a warranty issue and got approval to take it in for service. (Yay!)

Not wanting to be without my laptop for several days, I decided to be clever and swap hard drives. The install videos at Other World Computing couldn't be more straightforward. (I had already bought the appropriate tools when I upgraded to a 500Gb drive.)

Following the swap, I found a curious mixture of systems and applications that could tell.

• Not surprisingly, the iTunes authentication system could tell.

• Also not surprising, Time Machine could tell. I suppose it's just possible that two folks could have absolutely identical but physically different hard drives. I really don't know how it could tell, though.

• Some of the menu bar widgets were different. Apparently whatever makes the little flag icon is in flash memory somewhere and not configured from the hard drive?

• The one that really surprised me was VMWare Fusion. It didn't like any of my virtual disks, it thought they were all “in use”. Stealing them back and telling VMWare I'd “moved” them was enough to recover, though, so no harm done.

My laptop is back and the drives have been switched again. I wish the problem had been fixed, but they claimed to be unable to reproduce it. (Boo!) I can tell you right now that it's still a problem.

I should push harder to get the fans replaced, the intermittent high pitched whine is definitely a Bad Thing. Of course, the 17” MacBook Pro is due for a refresh any day now, right?

http://norman.walsh.name/2010/02/23/demojam 2010-02-23T20:13:39Z 2010-08-30T19:52:29Z MarkLogic XML XMLPrague2010

Demo Jam was a huge success at Balisage last year, so we're going to give it a go at XML Prague too!

Demo Jam was a huge success at Balisage last year, so we're going to give it a go at XML Prague too!

If you're coming to XML Prague, please plan to come to demo jam!

Here's the scoop: we provide the microphone and the projector. You provide the laptop and the demo. You get five minutes (that's 300 seconds, no more) to demo anything you want, as long as you can reasonably claim that there's some XML in there somewhere. We'd love to see some XQuery and maybe even MarkLogic Server (you could use the free Community License, for example), but that's not necessary.

Judging is by audience participation: if you make the crowd cheer the loudest, you win. Winner walks away with a free pass to the Mark Logic User Conference, May 4-6 in San Francisco, CA, US. Everyone gets a T-shirt.

Bring your demo and your posse to the social evening; demo jam starts at 8:30p. See you there!

http://norman.walsh.name/2010/01/25/xml 2010-01-25T22:21:37Z 2010-08-30T19:52:29Z Software XML

On the serendipitous joy of finding XML.

On the serendipitous joy of finding XML.

As I've said before, I'm very reluctant to use your application if it's a roach motel for my data. It would not be fair to say that I'll refuse to use your application, it's just a lot less likely.

For example, when it came to GSD, I decided that open access wasn't as important as picking an application that I'd actually use. If I let myself get distracted by exploring APIs, there'd be other things not getting done! (Priorities!)

Having made my bed, I figured I should see what I was lying in. Today I took a peek at how OmniFocus stores data. Now, the title of this essay no doubt gives away the punch line, so consider for a moment how this would have been done in the time before XML.

…go on, have a think, I'll wait…

In my experience it would probably have been in some proprietary format, almost certainly binary, and utterly opaque. How many tools document(ed) their proprietary data formats? On some platforms, there might have been system services for storing data, some sort of platform-supported database perhaps. Those systems are (often) only marginally better. They produce, instead of an opaque stream of bits, an opaque stream of atomic values. (Don't get me wrong, I've done the reverse-engineering thing on binary formats, I'd prefer the stream of atomic values, believe you me.)

What did I find when I went looking at the OmniFocus data? A directory full of ZIP files. And what's in each ZIP file? Why contents.xml, of course!

Now, it would not be fair to assert that this is perfectly transparent. XML isn't magic. There are clearly some cross-reference relationships in there that will take a little mental gymnastics to decode. But still, I'll trade this:

...
<task id="pJhk6REkEHC" op="update">
  <task idref="ggQv63WgCbw"/>
  <added>2010-01-21T16:23:08.983Z</added>
  <modified>2010-01-24T21:01:41.632Z</modified>
  <name>Add server-side support for multipart MIME to tests.xproc.org</name>
  <rank>2113929216</rank>
  <context idref="jYnYAAVroBT"/>
  <due>2010-01-27T22:00:00.000Z</due>
  <completed>2010-01-24T21:01:41.622Z</completed>
  <order>parallel</order>
</task>
...

for anything I would ever have gotten at any other point in the history of file formats!

XML has its detractors. It would not be fair to say they are all wrong. But I'll take XML over fair any day!

http://norman.walsh.name/2010/01/25/gsd 2010-01-25T15:13:26Z 2010-08-30T19:52:29Z OSX Software

Our engineering department has a project management philosophy they describe as GSD. I aspire to GSD.

Our engineering department has a project management philosophy they describe as GSD. I aspire to GSD.

For me, the part of GSD^[1] that I most often have difficulty with is keeping track of what needs doing. My todo (or want-todo) list is absurdly long. If I feel like castigating myself, I can always find a few things on my list that should have been done by now. It's not that I don't work hard or get a lot done, it's that I don't always prioritize perfectly and sometimes things slip through the cracks.

I've been trying to get better at this. Having an online calendar sync'd with my phone keeps me from accidentally missing meetings and phone calls, so it seems to follow that some sort of online system should be able to help me with my todo list.

My requirements are pretty simple: I want something that's easy to use and I want something that syncs with my mobile device. An online tool is almost, but not quite, as good as something that I can use offline on my PDA.

I don't subscribe to any particular Getting Things Done methodology. Maybe I'll get there someday, but that's not my immediate goal.

I played with Remember The Milk on-and-off last year. It seemed to work pretty well for simple lists, but I wasn't using it consistently because, I think, it wasn't quite powerful enough.

This month, I took a few different systems for a test drive: 2Do, Toodledo, Things, and OmniFocus.

Unfortunately, 2Do is only an iPhone app. It appears that there are plans for the next version to support syncing with Toodledo, but that doesn't exist today. Toodledo is a web-based app and is quite nice, probably plenty sufficient for my needs. On the desktop front, both Things and OmniFocus are probably plenty sufficient as well. (There are no doubt other similar applications, those are just the ones I happened to try. I didn't attempt an exhaustive survey, I've GStD!)

And the winner is: OmniFocus, by a narrow margin. I like the project/context duality that OmniFocus uses (ToodleDo has contexts too, if you turn them on). Mostly it boiled down to the UI: I liked the “feel” of OmniFocus best.

This is an app I plan to force myself to use, so I figured I'd best pick one that felt good. It's also the most expensive, by a pretty wide margin, but c'est la vie.

Will this really work for me? Time will tell. But so far, so good. And I'm already learning to use it in ways I hadn't planned: maintaining shopping lists and travel check lists. Those aren't the sorts of things for which I would have actively sought out software (sometimes a pencil and a piece of paper really is enough), but it's encouraging to me that I have other reasons to be paying attention to my GSD tool.

---

^[1]Getting Shi␈␈␈Stuff Done!

http://norman.walsh.name/2010/01/24/xmlprague 2010-01-24T18:27:40Z 2010-08-30T19:52:29Z DocBook XMLPrague2010 XProc

See you at XML Prague! And a chance to plug some really excellent training.

See you at XML Prague! And a chance to plug some really excellent training.

I'm delighted that my paper proposal for XML Prague was accepted. I'm a little less delighted that the final paper deadline was last week, but I guess that's encouragement to finish it, eh? (I will, I promise.)

I'm going to speak about modular documentation in DocBook, both about the proposal for “assemblies” being developed in the DocBook Technical Committee and about my XProc-based implementation.

I mention these things for only two reasons^[1]: first, to recommend that XML Prague is a conference you need to go to if you're interested in XML technologies. It's that good.

Second, to plug G. Ken Holman ’s XSLT/XPath 1.0 & 2.0 and XQuery 1.0 Hands-on Training class. If you're looking for someone to teach you XSLT and XQuery, you'd be hard pressed to do better than Ken. And if you're interested in XML, these are technologies you need to know. The maximum class size is an astonishing six, so it's practically 1:1. Yet another reason to be in Prague in March

See you there!

---

^[1]In the interest of full disclosure, I should point out that Mark Logic is a gold sponsor of the conference and Ken's course is being delivered in partnership with our own training services. I don't think that makes me baised, but I guess I wouldn't, would I?

http://norman.walsh.name/2010/01/24/nymug 2010-01-24T18:16:47Z 2010-08-30T19:52:29Z MarkLogic

Denise Miura, Sr. Director of Product Management will be speaking about Mark Logic's new offering for the Cloud at our upcoming User Group in New York this Wednesday.

Denise Miura, Sr. Director of Product Management will be speaking about Mark Logic's new offering for the Cloud at our upcoming User Group in New York this Wednesday.

The second Mark Logic New York User Group meeting will be held on Wednesday evening, 27 January 2010, hosted by Steve Kotrch from Simon & Schuster!

What

An opportunity to learn more about MarkLogic Server and collaborate with other MarkLogic users.

When

Wednesday, 27 January 2010, at 6:00pm EST.

Where

1230 Avenue of the Americas, New York, NY between 48th and 49th streets on Sixth Avenue.

Who

Everyone who shows up, of course! The featured speaker this time is Denise Miura who will explain Mark Logic's new cloud deployment options and describe how they are being used today within the Mark Logic development community. She will demonstrate instantiating a MarkLogic AMI live on Amazon EC2 and talk about best practices for using MarkLogic on the EC2 platform. Finally a preview of the planned cloud-related product enhancements will be provided. This is a great opportunity to provide feedback and influence the cloud computing initiative at Mark Logic.

How

If you plan to attend, please join the mailing list and send your first and last name to cleo dot saab at marklogic dot com.

If you're in New York, please stop by (and please let Cleo know if you plan to stop by).

http://norman.walsh.name/2009/12/28/xproc-lc 2009-12-28T14:20:52Z 2009-12-28T15:24:41Z W3C XProc

Early in January, a new XProc draft will appear. It will be a Last Call Working Draft, a step backwards in the process, or maybe just a half-step. The reason is important though: versioning.

Early in January, a new XProc draft will appear. It will be a Last Call Working Draft, a step backwards in the process, or maybe just a half-step. The reason is important though: versioning.

The XProc WG has been making steady progress on XProc: An XML Pipeline Language. We saw the start of wide adoption of XProc in 2009 and I think there's every reason to expect more of the same in 2010.

This makes it all the more disappointing to report that we're going back to Last Call. On a personal note, as a TAG alum, it's a bit embarrassing to admit why: versioning.

We received significant and persuasive criticism of our versioning story. In particular, we were pesuaded that requiring a processor to download the declarations for V.next steps in order to process them in a “forwards compatible” manner was too burdensome.

In redrafting the story, we added a version attribute, “compile-time” use-when functionality à la XSLT, and extension functions for more precisely identifying the environment in which the pipeline is running.

We also took the opportunity to fix decisions that were, in retrospect, mistakes, but not in-and-of themselves sufficient to motivate us to return to last call: we changed the rules for connections in option, parameter, and variable bindings so that an explicit <p:empty> isn't required when there's no default readable port, and we changed the rules for parameter input ports so that a binding isn't required when there's at least one explicit <p:with-param>. Users are already thanking us.

I didn't get the document through the publication process before the end-of-year publishing moratorium, but you can read the staged draft, if you wish.

Will this really be our last Last Call? I sincerely hope so! I also hope that we can move directly from Last Call to Proposed Recommendation without the formality of another Candidate Recommendation phase in between. There's precedent for doing this, and we've got active implementors.

I'm sometimes frustrated by how long the process takes, but I console myself with the observation that the language is better for our efforts. Rising usage suggests the early adopters, at least, agree with us.

http://norman.walsh.name/2009/12/26/dad 2009-12-26T20:38:58Z 2009-12-31T14:40:33Z People

9 June 1923 — 26 November 2009.

9 June 1923 — 26 November 2009.

A man may by custom fortify himself against pain, shame, and suchlike accidents; but as to death, we can experience it but once, and are all apprentices when we come to it.

— Montaigne

My father was born in 1923 in Babylon, NY.

David Walsh, age 10

He survived the Great Depression. An enormous tree blew over next to him as he walked home through The Great Hurricane of 1938; he walked away without a scratch. The glider born infantry took him to the China-Burma-India theater in WWII.

Bombay c. 1945

Left to right: Charles Kuhn, York PA; Bill Bride, Beacon NY; Eddy Evans, Boston MA; David Walsh, Babylon, NY

Shrapnel chipped a tooth, but he survived that too. After the war he went to Alaska.

Browerville from the tundra, Mar 1960

My dad taught in Barrow and Nome. After putting out a chimney fire, he walked away from a two story fall off a frozen roof by the lucky stroke of landing feet-first on an oil drum.

Camping in Alaska, c. 1960

He single-handedly built a one-room cabin on a ¼ acre plot in Fairbanks. (I think I remember seeing once a photo showing the scaffolding he built to get the roof beam in place.) He worked for the Fish and Wildlife Service in the summers.

He used to practice orienteering by walking into the Alaskan wilderness on a compass bearing and then walking back out again. On one occasion he stumbled across a downed single-engine plane containing the skeleton of its pilot. His boss laughed when my dad offered to lead a team back to the crash, assuring him that he'd never find it again. Dad's boss was right. There is a lot of wilderness out there.

On another occasion, my dad shot a caribou only to discover as he prepared to dress it that he'd left his knife back in the jeep. Leaning his rifle against a tree, he walked back and got his knife. An enormous brown bear greeted his return by standing on its hind legs and roaring. The bear got the caribou. And the rifle. And the knife, dropped during a hasty retreat.

That wasn't the only caribou that nearly got him killed; on another occasion, one attempted, unsuccessfully, to jump over his jeep. He woke on the side of the road with a caribou hoof protruding into the cab and a nasty gash on his head.

I'm lucky to be here.

Sleeping, June 1970

When my dad left Alaska, he gave the keys to his cabin to a friend. Those keys passed from friend to friend for more than twenty years. In the eighties, the current occupant persuaded my dad to let him buy the cabin. My father signed the deed and mailed it, asking the occupant to please mail the check back. The check came back a couple of weeks later. And it cleared. Luck of the Irish, or something.

From Alaska, my dad traveled to Australia. My mom and dad met in Tasmania. They married in 1961.

Mom and dad, August 1961

I came along a few years later.

Mom, dad, and I

June 1968

I remember my dad singing sea shanties when I was a small boy.

Playing guitar

Time and place unknown

Dad was a naturalist, hunter, trapper, fisherman, scientist, teacher, draftsman, and surveyor. He made beautiful wood carvings. He tied knots. At one time or another, all of them. I have his leather working tools. The old sewing machine on which he made sleeping bags, tents, parkas, rain slickers, and bicycle paniers got lost somewhere along the way. He built two boats.

David Walsh, Sep 2009

After 86 years, entropy won. Entropy always wins. My dad taught me that. And the first and third laws as well.

My father died in 2009 in Norwich, England.

'tis himself

David Alfred Walsh 9 June 1923 - 26 November 2009

Goodbye, dad.