Summary

  • Debian launched its snapshot service in April 2010, offering older package collections by date and version for debugging and repeatable upgrades.
  • Hardware trouble that year exposed two different jobs: keeping saved packages available and collecting the next changes before they disappeared upstream.

A time machine sounds like something that should rise above the indignities of a broken disk controller. Debian's had to wait for a replacement.

The project announced snapshot.debian.org on 12 April 2010. Users could fetch earlier packages and source code through an ordinary apt repository. An administrator could rehearse an upgrade against a dated collection, then return to that collection for the real job. A developer hunting a regression could revisit what had changed. The launch announcement put the collection at about 6.5 terabytes, with almost all main and security archive packages back to March 2005.

The workshop diary was already more eventful than the brochure. On 20 February, the team's log recorded a second site serving a replicated copy. The primary machine, stabile, still imported new snapshots; sibelius supplied another web front end. Four days later, a controller failure left half the primary machine's disks inaccessible. The surviving server kept serving the collection. New imports stopped.

The team had planned a secondary importer with its own database. The broken controller gave that plan a vigorous promotion up the to-do list.

Yesterday stayed open; today needed collecting

By 9 March, the primary was back online and snapshots collected on the secondary were being merged. One collection, backports.org, had missed the necessary push to that secondary. On 15 March, the log recorded the primary taking snapshots again and the secondary data merged.

Recovery also meant looking inside the boxes. Files were named by a hash of their contents, while a database linked those names to packages and archive paths. The checks compared names with actual contents and asked whether database references had corresponding files. They found several kinds of damage, including correct files with wrongly calculated names and corrupt package indexes. Some records needed relabelling; some damaged files were removed. Switching the power back on was merely the opening act.

Later that year came another useful, irritating lesson. A 14 December update described hardware trouble around 20 November. The remaining mirror kept presenting its November 20 collection while the primary stopped importing. A private backup system had continued collecting, and its data was subsequently merged. But it had omitted debian-ports. That archive had a gap for the affected period.

The service gave software history a practical address: a dated collection that another machine could actually use. Its repair notes showed the labour behind that address. Yesterday could remain on the shelf while today's delivery failed to arrive. A time machine needed someone watching the loading bay.