http://norman.walsh.name/2009/09/26/sqltoxml 12 31 2009-09-26T13:01:23+01:00 Norman Walsh 2009 Norman Walsh A number of Mac applications store information in SQLite databases. Step one to do something useful with that data is to get it into XML. I hate having my data squirreled away in proprietary or quasi-proprietary ways. If I can't get my information back out of an app, I'd rather not use it. When I started using the Mac, I switched to iCal and AddressBook: both can export data in standard text formats which I can easily convert to XML. But exporting the data is a manual process (though I could probably automate it with some clever AppleScript or something, I've never tried). I build a number of views of my address book and calendar data automatically so manual processes don't fit well into my workflow. It didn't take too long to figure out where iCal stores my appointments or how to pull them together. Having worked out where iCal stores my appointments, I turned my attention to AddressBook. Long story short: the address data is saved in a database in ~/Library/Application Support/AddressBook/AddressBook-v22.abcddb. After installing the sqlite3 application from MacPorts, I was able to extract a text dump. So far so good. Here, for example, is a table definition and the first row of data in that table: CREATE TABLE ZABCDRECORD ( Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER, Z_OPT INTEGER, ZDISPLAYFLAGS INTEGER, ZMODIFICATIONDATEYEAR INTEGER, ZCREATIONDATEYEAR INTEGER, ZADDRESSBOOKSOURCE INTEGER, ZISALL INTEGER, ZME INTEGER, Z19_ME INTEGER, ZINFO INTEGER, ZBIRTHDAYYEAR INTEGER, ZPRIVACYFLAGS INTEGER, ZNOTE INTEGER, ZADDRESSBOOKSOURCE1 INTEGER, ZCONTACTINDEX INTEGER, ZSOURCEWHERECONTACTISME INTEGER, ZVERSION INTEGER, ZSYNCCOUNT INTEGER, ZSHARECOUNT INTEGER, ZADDRESSBOOKSOURCE2 INTEGER, ZMODIFICATIONDATE TIMESTAMP, ZCREATIONDATE TIMESTAMP, ZMODIFICATIONDATEYEARLESS FLOAT, ZCREATIONDATEYEARLESS FLOAT, ZBIRTHDAY TIMESTAMP, ZBIRTHDAYYEARLESS FLOAT, ZUNIQUEID VARCHAR, ZNAME VARCHAR, ZNAMENORMALIZED VARCHAR, ZTMPREMOTELOCATION VARCHAR, ZNAME1 VARCHAR, ZREMOTELOCATION VARCHAR, ZSERIALNUMBER VARCHAR, ZSUFFIX VARCHAR, ZTITLE VARCHAR, ZTMPHOMEPAGE VARCHAR, ZNICKNAME VARCHAR, ZORGANIZATION VARCHAR, ZMAIDENNAME VARCHAR, ZIDENTITYUNIQUEID VARCHAR, ZPHONETICFIRSTNAME VARCHAR, ZDEPARTMENT VARCHAR, ZPHONETICLASTNAME VARCHAR, ZMIDDLENAME VARCHAR, ZFIRSTNAME VARCHAR, ZIMAGEREFERENCE VARCHAR, ZJOBTITLE VARCHAR, ZPHONETICMIDDLENAME VARCHAR, ZLASTNAME VARCHAR, ZSORTINGFIRSTNAME VARCHAR, ZSORTINGLASTNAME VARCHAR, ZCREATEDVERSION VARCHAR, ZLASTDOTMACACCOUNT VARCHAR, ZLASTSAVEDVERSION VARCHAR, ZSYNCANCHOR VARCHAR, ZSEARCHELEMENTDATA BLOB, ZMODIFIEDUNIQUEIDSDATA BLOB ); INSERT INTO "ZABCDRECORD" VALUES(1,18,287,NULL,NULL,2008,NULL,1,NULL,NULL, 3,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL, 239119683.670433,NULL,18281283.670433,NULL,NULL, '93973926-7EF6-40F0-ADBD-8C7BBFC30FA1:ABSubscriptionRecord',NULL, NULL,NULL,NULL,'local','B7303AAD-DA79-46E6-BC7D-91DAD82AEFB8', NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL, NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL); Next, I wrote 150 or so lines of Perl to convert the text into XML. The XML is designed to be a totally straightforward representation of the table structure of the database. From the preceding SQL statements, sqltoxml produces: <table name='ZABCDRECORD'> <columns> <column name='Z_PK' type='INTEGER'/> <column name='Z_ENT' type='INTEGER'/> <column name='Z_OPT' type='INTEGER'/> <column name='ZDISPLAYFLAGS' type='INTEGER'/> <column name='ZMODIFICATIONDATEYEAR' type='INTEGER'/> <column name='ZCREATIONDATEYEAR' type='INTEGER'/> <column name='ZADDRESSBOOKSOURCE' type='INTEGER'/> <column name='ZISALL' type='INTEGER'/> <column name='ZME' type='INTEGER'/> <column name='Z19_ME' type='INTEGER'/> <column name='ZINFO' type='INTEGER'/> <column name='ZBIRTHDAYYEAR' type='INTEGER'/> <column name='ZPRIVACYFLAGS' type='INTEGER'/> <column name='ZNOTE' type='INTEGER'/> <column name='ZADDRESSBOOKSOURCE1' type='INTEGER'/> <column name='ZCONTACTINDEX' type='INTEGER'/> <column name='ZSOURCEWHERECONTACTISME' type='INTEGER'/> <column name='ZVERSION' type='INTEGER'/> <column name='ZSYNCCOUNT' type='INTEGER'/> <column name='ZSHARECOUNT' type='INTEGER'/> <column name='ZADDRESSBOOKSOURCE2' type='INTEGER'/> <column name='ZMODIFICATIONDATE' type='TIMESTAMP'/> <column name='ZCREATIONDATE' type='TIMESTAMP'/> <column name='ZMODIFICATIONDATEYEARLESS' type='FLOAT'/> <column name='ZCREATIONDATEYEARLESS' type='FLOAT'/> <column name='ZBIRTHDAY' type='TIMESTAMP'/> <column name='ZBIRTHDAYYEARLESS' type='FLOAT'/> <column name='ZUNIQUEID' type='VARCHAR'/> <column name='ZNAME' type='VARCHAR'/> <column name='ZNAMENORMALIZED' type='VARCHAR'/> <column name='ZTMPREMOTELOCATION' type='VARCHAR'/> <column name='ZNAME1' type='VARCHAR'/> <column name='ZREMOTELOCATION' type='VARCHAR'/> <column name='ZSERIALNUMBER' type='VARCHAR'/> <column name='ZSUFFIX' type='VARCHAR'/> <column name='ZTITLE' type='VARCHAR'/> <column name='ZTMPHOMEPAGE' type='VARCHAR'/> <column name='ZNICKNAME' type='VARCHAR'/> <column name='ZORGANIZATION' type='VARCHAR'/> <column name='ZMAIDENNAME' type='VARCHAR'/> <column name='ZIDENTITYUNIQUEID' type='VARCHAR'/> <column name='ZPHONETICFIRSTNAME' type='VARCHAR'/> <column name='ZDEPARTMENT' type='VARCHAR'/> <column name='ZPHONETICLASTNAME' type='VARCHAR'/> <column name='ZMIDDLENAME' type='VARCHAR'/> <column name='ZFIRSTNAME' type='VARCHAR'/> <column name='ZIMAGEREFERENCE' type='VARCHAR'/> <column name='ZJOBTITLE' type='VARCHAR'/> <column name='ZPHONETICMIDDLENAME' type='VARCHAR'/> <column name='ZLASTNAME' type='VARCHAR'/> <column name='ZSORTINGFIRSTNAME' type='VARCHAR'/> <column name='ZSORTINGLASTNAME' type='VARCHAR'/> <column name='ZCREATEDVERSION' type='VARCHAR'/> <column name='ZLASTDOTMACACCOUNT' type='VARCHAR'/> <column name='ZLASTSAVEDVERSION' type='VARCHAR'/> <column name='ZSYNCANCHOR' type='VARCHAR'/> <column name='ZSEARCHELEMENTDATA' type='BLOB'/> <column name='ZMODIFIEDUNIQUEIDSDATA' type='BLOB'/> </columns> <rows> <row> <Z_PK>1</Z_PK> <Z_ENT>18</Z_ENT> <Z_OPT>287</Z_OPT> <ZCREATIONDATEYEAR>2008</ZCREATIONDATEYEAR> <ZISALL>1</ZISALL> <ZINFO>3</ZINFO> <ZCREATIONDATE>239119683.670433</ZCREATIONDATE> <ZCREATIONDATEYEARLESS>18281283.670433</ZCREATIONDATEYEARLESS> <ZUNIQUEID>93973926-7EF6-40F0-ADBD-8C7BBFC30FA1:ABSubscriptionRecord</ZUNIQUEID> <ZREMOTELOCATION>local</ZREMOTELOCATION> <ZSERIALNUMBER>B7303AAD-DA79-46E6-BC7D-91DAD82AEFB8</ZSERIALNUMBER> </row> <!-- ... --> </rows> </table> As you can see, I've made no effort to maintain some aspects of the database (like the primary key), I've simply dropped NULL fields, and I'm relying on the field names to be valid XML NCNames. There are clearly other, equally reasonable, design choices that I could have made. The resulting XML is the bare minimum needed to switch to XML tools for subsequent downstream processing (turning address book tables into VCards, for example). But it gets the job done. And maybe it'll come in handy for someone else.