Accessing Flickr with XProc

The Flickr Uploader has never been a model of stability in my experience. Recently, it's become entirely unusable, it simply doesn't work for the current set of pictures that I want to upload.

There are lots of alternative uploaders, but I'm also interested in exploring the capabilities of XProc. Specifications and test suites aside, you don't really know what you can do with a language until you try.

Perhaps uploading photographs to Flickr isn't the sort of application best achieved by XProc, but I'm certain that interacting with real world web service APIs is something that XProc pipelines will need to do.

Uploading photographs requires authenticated access. As a prerequisite for that, you need to go through a little authentication dance with Flickr. I'm not going to try to automate that part; I'll assume you're in possession of an API key, authentication token, and shared secret. Authenticated access is achieved by “signing” the API call; in brief, concatentate the shared secret with the parameters (sorted into alphabetical order) and their values, compute the MD5 hash of that string, and add that value as a new parameter. Can we do this in XProc?

Sorting the parameters is easy. The only interesting part of that pipeline is the p:parameters step. It's there to turn what might possibly be a sequence of documents on the parameters port into a single document for XSLTIf you think using XSLT for the sorting part is “cheating” of some sort, I don't agree. And if you think that I should just punt and do the whole exercise in XSLT, I don't agree with you about that, either..

Before we dive into the signing step, there's one more complication. To simplify processing for the caller, I'm going to allow the “secret” to be either a parameter or an option. The Flickr APIs of course don't allow the secret to be a parameter because that would expose the secret which would defeat its purpose. But from an XProc perspective, it's convenient to have a c:param-set that contains the key, token, and secret which we can just pass around together.

That means the signing step will have to be prepared for the secret to come from either place and will have to explicitly exclude the secret from the signed parameters.

Let's begin! Our pipeline, flickr:sign-api takes a set of parameters and an optional secret. It returns the signed parameters as a c:param-set on the result port.

<p:declare-step xmlns:p="http://www.w3.org/ns/xproc"
                xmlns:c="http://www.w3.org/ns/xproc-step"
                xmlns:flickr="http://xmlcalabash.com/ns/extensions/flickr"
                xmlns:cx="http://xmlcalabash.com/ns/extensions"
                xmlns:err="http://www.w3.org/ns/xproc-error"
                type="flickr:sign-api" name="main"
                exclude-inline-prefixes="err" version="1.0">
  <p:input port="parameters" kind="parameter"/>
  <p:output port="result"/>
  <p:option name="secret"/>

The first step is to run the parameters through a p:parameters step, as discussed above, then we go looking for the secret. This is XProc, so everything has to be XML. The result of the choose is an XML document that contains the secret selected from (preferentially) the secret option or from the secret parameter. It's an error to call this pipeline without passing a secret.

  <p:parameters name="params">
    <p:input port="parameters">
      <p:pipe step="main" port="parameters"/>
    </p:input>
  </p:parameters>

  <p:choose name="secret">
    <p:when test="p:value-available('secret')">
      <p:output port="result"/>

      <p:string-replace match="/secret/text()">
        <p:input port="source">
          <p:inline><secret>@@</secret></p:inline>
        </p:input>
        <p:with-option name="replace" select="concat('&quot;',$secret,'&quot;')"/>
      </p:string-replace>
    </p:when>

    <p:when test="/c:param-set/c:param[@name='secret']">
      <p:xpath-context>
        <p:pipe step="params" port="result"/>
      </p:xpath-context>
      <p:output port="result"/>

      <p:variable name="value"
                  select="string(/c:param-set/c:param[@name='secret']/@value)">
        <p:pipe step="params" port="result"/>
      </p:variable>

      <p:string-replace match="/secret/text()">
        <p:input port="source">
          <p:inline><secret>@@</secret></p:inline>
        </p:input>
        <p:with-option name="replace" select="concat('&quot;',$value,'&quot;')"/>
      </p:string-replace>
    </p:when>

    <p:otherwise>
      <p:output port="result"/>

      <p:error code="err:XX01">
        <p:input port="source">
          <p:inline>
            <message>flickr:sign-api called without secret</message>
          </p:inline>
        </p:input>
      </p:error>
    </p:otherwise>
  </p:choose>

Next we sort them.

  <cx:sort-parameters name="sorted">
    <p:input port="parameters">
      <p:pipe step="main" port="parameters"/>
    </p:input>
  </cx:sort-parameters>

Now we have a c:param-set containing all of the parameters in alphabetical order. From this we must construct a string that we can sign. There's no two ways about it, string manipulation in XProc is not as easy as XML manipulation. Here's what we're going to do: for each parameter (except the secret, if it's there) we're going to construct a dummy document that contains the name and value concatenated together. If we pass in <c:param name="foo" value="bar"/>, we'll pass back <doc>foobar</doc>.

There's one interesting complication here. We're going to use p:string-replace to do the heavy lifting, but that will make the context node the node that we're replacing. From that context node, we'll have no way to get back to the c:param that's on the current port. We work around this by storing the values we need in variables.

  <p:for-each name="loop">
    <p:iteration-source select="//c:param[@name != 'secret']"/>

    <!-- Flickr params should always be NCNames, but just in case... -->
    <p:variable name="name"
                select="if (contains(/*/@name, ':'))
                        then substring-after(/*/@name, ':')
                        else string(/*/@name)"/>

    <p:variable name="value" select="string(/*/@value)"/>

    <p:string-replace match="/doc/text()">
      <p:input port="source"><p:inline><doc>@@</doc></p:inline></p:input>
      <p:with-option name="replace"
                     select="'concat(&quot;',$name,'&quot;,&quot;',$value,'&quot;)'">
        <p:pipe step="loop" port="current"/>
      </p:with-option>
    </p:string-replace>
  </p:for-each>

The output of the loop step is a sequence of doc documents. We'll turn that back into a single document by adding a wrapper and we'll prepend the secret value to the beginning of the document.

  <p:wrap-sequence wrapper="inner-wrapper"/>

  <p:insert match="/inner-wrapper" position="first-child">
    <p:input port="insertion">
      <p:pipe step="secret" port="result"/>
    </p:input>
  </p:insert>

We're getting there! Now we've got something that looks like this:

<inner-wrapper>
<secret>somehexvalue</secret>
<doc>api_keysomelonghexvalue;</doc>
<doc>api_tokenanotherlonghexvalue;</doc>
<doc>methodsomemethodname;</doc>
</inner-wrapper>

Recall your XPath semantics, if we ask for the string-value of that document, we'll get the string we want hash. We have a p:hash step, so we can do that part. But if we replace the wrapper with the hash of it's string value, the result won't be well-formed because it won't have a document element anymore. We get around that by adding another level of wrapper first.

  <p:wrap-sequence wrapper="wrapper"/>

  <p:hash match="/wrapper/inner-wrapper" algorithm="md" version="5" name="hash">
    <p:with-option name="value" select="string(.)"/>
  </p:hash>

The output of p:hash is a document that looks like this:

<wrapper>somehexstring</wrapper>

Now if we can stick that back in a c:param element and stick that parameter back into the c:param-set, we're done. We'll use p:string-replace to put the string back into an element, but here we encounter the first really odd wrinkle in the XProc design.

Like before, we need to put the hash in a variable so that we can get at it when the context is changed by p:string-replace. But variables can only appear at the beginning of a compound step. Therefore, we have to introduce a p:group.

C'est la vie. Here's the p:group, the variable, and the p:string-replace call.

  <p:group>
    <p:variable name="hash" select="string(/)">
      <p:pipe step="hash" port="result"/>
    </p:variable>

    <p:string-replace match="/c:param/@value" name="api_sig">
      <p:input port="source">
        <p:inline><c:param name="api_sig" value="@@"/></p:inline>
      </p:input>
      <p:with-option name="replace" select="concat('&quot;',$hash,'&quot;')"/>
    </p:string-replace>

The last step is to combine this new signature parameter with the original input parameters. Remember that the user may have passed the secret in as a parameter, so we start by deleting that one.

    <p:delete match="c:param[@name='secret']">
      <p:input port="source">
        <p:pipe step="params" port="result"/>
      </p:input>
    </p:delete>

Then we insert the signature into the resulting c:param-set and that's what we send out the result port.

    <p:insert match="/c:param-set" position="last-child">
      <p:input port="insertion">
        <p:pipe step="api_sig" port="result"/>
      </p:input>
    </p:insert>
  </p:group>
</p:declare-step>

The input parameters have been augmented with a signature that will satisfy the Flickr authentication requirements. Now we're read to move on to actually using the API with our signing pipeline.

In a future essay we'll look at making a “Flickr service” pipeline and an upload pipeline.