10 81 2007-08-09T12:55:32-04:00 $Date$ NormanWalsh 2007Norman Walsh A tale of woe with bad parts, good parts, and a moral. I've been using Mercurial for a few weeks now. I had no trouble switching; it's similar enough to the other version control systems I've used. And even though I'm usually connected to the net, I'm loving fast, local commits. So much so that I have six or eight repositories up and running now: my home directory, one for each of the web sites I maintain, one for papers I've written, another for presentations, basically most of (eventually, all of) the stuff that isn't in some other version control system.

Then a few days ago, I happened to notice something odd (alas, I don't recall what). Verifying local and remote repositories revealed that my home directory repository was totally borked on the server (it showed 1 revision where locally I had 23) and the repository for this weblog showed a bunch of errors, both locally and remotely. The very worst part is that there had been no error messages to indicate that anything had gone wrong. In fact, running an hg push in my home directory appeared to complete successfully. It's hard to imagine a worse sort of error than that in a version control system. Much digging (with tremendous assistance from MattMackall at selenic) revealed the problem. In a nutshell, my hosting company, where the server I'm pushing to is located, terminates with extreme prejudice any process that uses too much memory or too much CPU. My attempt to push the commit of a 50Mb image caused the Mercurial process on the server to chew up hundreds of megabytes of memory attempting to compute diffs, and when it crossed some threshold, it got SIGKILLd. I think the client should have noticed this and not appeared to complete successfully, but I'm not sure how realistic that is. I'm willing to stipulate that kill -9ing one half of system isn't something you'd expect to happen. In practice, I didn't realize I'd attempted to put a 50Mb image into the system and I don't have any practical reason to do so. Matt maintains that a similar circumstance caused the errors in my other repository. I'm not really convinced, but since there doesn't appear to be any way to tell, I'm not pushing the point. So now I've got two repositories with errors deep in their version history. (The second of 23 transactions in one case, the 1366th of 1420 in the other.)

Luckily, Mercurial has features that allowed me to deal with these issues. They're called “queues”. I don't really grok them, but Matt helped me use them to fix the problems. In each case, I turned all of the commits that had occurred after the point where the error occurred into a queue of patches. Then I rolled back to the error version, fixed the errors, refreshed the patch for that version, reapplied all the patches, and finally deleted the queue. With fixed local repositories in hand, I was able to migrate them back to the server and all seems to be well.

You might think the moral is, don't use Mercurial, but I'm not willing to go that far. A lot of big projects are (or are going to be) using it. I'm confident that the bugs will get worked out. The queuing functionality seems to provide the ability to work around problems. No, the moral is an unverified backup is a worthless backup. Backing up your data (and really, that's what I'm using Mercurial for here) but not checking that the backup is both readable and correct is accepting a risk proportional to the importance of your data. And you're backing it up because it's important, right? Your gun, your bullet, your foot. No more blindly pushing changes and assuming they work for me. ) { chop; next if /^\#/ || /^\s*$/; $section = $1 if /^\[(.*?)\]/; $key = $1 if /^([^\[\s=]+)/; if ($section eq 'paths' && $key eq 'default') { ($host = $_) =~ s/^.+\s//; } } die "No default path?\n" if !defined($host); die "Only prepared for ssh paths; found \"$host\"\n" unless $host =~ /^ssh:\/\/(.*?)\/(.*)$/; my $rsh = $1; my $path = $2; print "Verifying local repository\n"; system("hg verify"); print "Pushing local changes to $host\n"; system("hg push"); print "Verifying remote repository\n"; system("rsh $rsh \"(cd $path; hg verify)\"");]]> Share and enjoy.