Using Pygments from XSLT

This is just plain fun.

Syntax highlighting source code listings can make them easier to read (it certainly makes them prettier). Pygments is an excellent syntax highlighter. It's a python tool that can transform source code listings into nice, clean HTML using spans and class attributes.

I've integrated a syntax highlighter into my XML workflow many times and in a variety of ways, most often using make or ant or an XProc pipeline and XInclude.

The topic comes up periodically on the DocBook Apps mailing list. When it came up most recently, Jirka suggested that it would interesting to try integrating Pygments directly into Saxon with Jython. I thought that would be just the ticket. (It helped that I had just been digging around in the extension functions part of Saxon a few days earlier.)

There's no distribution for it yet, but it works and you can try it out with the latest commits to the DocBook XSLT 2.0 stylesheets. But first, you have to do a little setup.

If something goes wrong in Pygments, your whole stylesheet will crash. At some point, I'll go back and work a little harder to be defensive about errors.

A few notes:

Ironically, these pages are formatted with MarkLogic Server which isn't Saxon and so nothing about Jython has any relevance at all.

What I did for MarkLogic was implement the extension as a web service that I can call from a stylesheet running on MarkLogic Server. The fact that XSLT makes this “just work” is really nice. Here's the place where the stylesheet decides what to do:

<xsl:choose>
  <xsl:when test="contains(@role,'nopygments') or string-length(.) &gt; 9000
                  or self::db:literallayout or exists(*)">
    <xsl:apply-templates/>
  </xsl:when>

  <xsl:when use-when="function-available('xdmp:http-post')"
            test="$pygmenter-uri != ''">
    <xsl:sequence select="ext:highlight(string(.), string(@language))"/>
  </xsl:when>

  <xsl:when use-when="function-available('ext:highlight')"
            test="true()">
    <xsl:sequence select="ext:highlight(string(.), string(@language))"/>
  </xsl:when>

  <xsl:otherwise>
    <xsl:apply-templates/>
  </xsl:otherwise>
</xsl:choose>

Later on, there's a declaration for the ext:highlight function. Note that it's presence is also conditional. It'll only be seen by MarkLogic Server. On Saxon, the function is provided through the processor's extension mechanisms; it can't be implemented in XSLT.

<xsl:function use-when="function-available('xdmp:http-post')"
              name="ext:highlight" as="node()*">
  <xsl:param name="code"/>
  <xsl:param name="language"/>

  <xsl:variable name="code-node" as="text()">
    <xsl:value-of select="$code"/>
  </xsl:variable>

  <xsl:variable name="highlighted"
                select="xdmp:http-post(concat($pygmenter-uri,'?language=',$language),(),$code-node)"/>

  <xsl:sequence select="$highlighted[2]//h:pre/node()"/>
</xsl:function>

Anyway, that's how the formatting is being done on these pages. Here's the endpoint, in case you're interested (it serves as another nice example of syntax highlighting even if you aren't interested):

#!/usr/local/bin/python

import sys, os, re, string;

from pygments import highlight
from pygments.lexers import (get_lexer_by_name, get_lexer_for_mimetype)
from pygments.lexers import guess_lexer
from pygments.formatters import (HtmlFormatter, get_formatter_by_name)
from pygments.util import ClassNotFound

print "Content-Type: application/xml"
print ""

query = os.environ["QUERY_STRING"]
code = ""

for line in sys.stdin:
    code = code + line.decode("utf8")

language = re.search("language=([^?&]+)", os.environ["QUERY_STRING"])
if language is None:
    language = ""
else:
    language = language.group(1)

formatname = re.search("formatter=([^?&]+)", os.environ["QUERY_STRING"])
if formatname is None:
    formatname = "html"
else:
    formatname = formatname.group(1)

if language == "":
    lexer = guess_lexer(code)
else:
    try:
        lexer = get_lexer_by_name(language)
    except ClassNotFound:
        lexer = ""

formatter = get_formatter_by_name(formatname)

try:
    result = highlight(code, lexer, formatter)
except AttributeError:
    sys.stderr.write("Failed to highlight code: lexer=" + lexer + " formatter=" + formatname + "\n")
    sys.stderr.write(code.encode("ascii","xmlcharrefreplace") + "\n")
    code = code.replace("&", "&amp;").replace("<", "&lt;").replace(">","&gt;")
    result = "<pre>" + code + "</pre>"

# Put it in the right namespace
result = "<div xmlns='http://www.w3.org/1999/xhtml'>" + result + "</div>"

print result.encode("ascii","xmlcharrefreplace")

I'm quite pleased that I was able to make it work in both Saxon and MarkLogic server! (The web service solution won't work in Saxon because there's no standard way to do an HTTP POST with XSLT, but if you have an extension that will POST, you could craft an extension function that would work in Saxon using the web service. That would let you avoid the overhead of Jython.)