http://norman.walsh.name/2003/11/13/charent 6 110 2003-11-13 $Date: 2005-09-11 10:27:02 -0400 (Sun, 11 Sep 2005) $ Norman Walsh 2003 Norman Walsh After much consideration, I don’t think XML should try to solve the character entity problem. James Baldwin Baldwin James The price one pays for pursuing any profession or calling is an intimate knowledge of its ugly side. I don’t think XML should try to solve the character entity problem. There, I’ve said it. I wonder where I left my flame-retardant underwear? The character entity problem, if you aren’t familiar with it (and if you aren’t, this essay might be a good one to skip) is the fact that entity references, often used to name characters that can’t conveniently be typed, aren’t part of well-formed XML documents that don’t have a DTD. The following document is not well-formed: <?xml version='1.0'?> <doc> <p>Les &eacute;tudiants sont tr&egrave;s s&eacute;rieux.</p> </doc> XML only has five predefined entities (&lt;, &amp;, etc.), you have to define the rest in the DTD (either the internal or external subset). The problem is that well-formed documents don’t have to have a DTD. There are some good reasons that you might want to avoid a DTD (you may be using another schema language, for example, or you may want to use your documents in environments like SOAP which forbid the document type declaration). So what can you do? That, as they say, is the sixty-four dollar question. For a long time, the answer was “not much,” and we’ve been wrestling with this one for years. I think everyone accepts that the best answer is to just use the characters you want. XML is Unicode Unicode , so if you want to say “the students are very serious” in French, say it: “les étudiants sont très sérieux.” But what if you can’t type in those characters? I’ve hacked up pretty good Unicode support in emacs, but I still can’t even pretend to support interesting mathematics. An equation as simple as this one stymies me: One solution is to use numeric character references. These refer to characters by their Unicode codepoint, so no declarations are necessary. They work, but they’re hard to read. Now, in point of fact, emacs-nxml-mode does a pretty decent job of displaying numeric character references, which reinforces my feeling that this is really a user-interface issue. With a better font, that’d be much more readable. It would still be ugly, but I wouldn’t have to remember all those codepoints to read it. Of course, with a better font, I wouldn’t have bothered with the numeric character references, I would have just stuck in the characters.

Tim Bray recently proposed a solution that involved a new encoding. On balance, I think the proposal would introduce so much confusion that it’s probably not the right answer, but it does have a significant benefit over all the other proposals I’ve seen: it’s a lexical hack. The more I think about it, the more I think that a standard solution to this problem should address it at the lexical level. By lexical level, I mean that the solution should not intrude into the XML object model. Whatever solution we arrive at (if we do), it should be the case that “sérieux” spelled with a literal character or the new solution should be indistinguishable in the XML object model. (Just as “s&#233;rieux” is indistinguishable now.) If it isn’t, then we don’t need a standard solution. If the solution involves some sort of transformation from the “input document” to the “real document”, then that transformation can just be another step on the XML processing pipeline and we don’t need to bake it into XML. (We need an XML processing pipeline, but that’s a whole other story.) Suppose, for example, that we use elements to represent these characters. Some folks feel strongly that the named character mechanism should have namespaced names and elements provide that for free. Then you might spell “sérieux” “s<e:eacute/>rieux”. Almost any non-trivial operation that you perform on a document that uses this mechanism is going to have to transform all the named characters into literal characters before it begins (I can’t imagine searching or sorting or any other operation on the text that’s not going to find it hugely inconvenient, if not impossible, to operate on the text with embedded elements). But, if there’s going to be an explicit transformation, nothing special is needed, just code that transformation up in XSLT or Java or Perl or Python or your language of choice and put it in the pipeline. One of the things that motivated this essay was a recent proposal that named characters should be transformed by schema validation. I think that’s a horrible idea. I can understand how the XML Schema WG got there, but I think that making the interpretation of named characters depend on validation is a really bad idea. The proposal I saw From Michael Sperberg-McQueen , alas in W3C member-only space. In the interest of fairness, I stole the example sentence about serious students from Michael’s proposal. suggested that we might spell spell “sérieux” “s{#e:eacute}rieux” and that validation would provide the replacement text. I don’t know if that’s a good way to spell those characters or not, John Cowan offered some good arguments for a slightly different spelling. But my point is that we don’t need anything special to implement this. Suppose we decide to talk about serious French students this way: <?xml version='1.0'?> <doc xmlns:h="http://www.w3.org/1999/xhtml"> <p>Les {#h:eacute}tudiants sont tr{#h:egrave}s s{#h:eacute}rieux.</p> </doc> I can easily construct an XSLT 2.0 stylesheet that transforms the input into actual characters and another that transforms the actual characters back into names.

Nothing. Well, nothing standard at the XML level, anyway. There’s nothing that can practically be done to introduce a new mechanism at the lexical level in XML. I suppose that some future version of XML could decide that all the ISO 8879/9573 entities were predefined, but that seems unlikely to me. And if you aren’t going to solve this at the lexical level, just solve it as part of your processing pipeline. If there really is a best way to spell characters at this level, it’ll win by natural selection and tool vendors will be motivated to make it efficient. If not, then we shouldn’t be standardizing it anyway, right?