Catalogue / tarsnap-2012-ec2-power-outage

A power loss in EC2 corrupts Tarsnap's server disk, and rebuilding state from S3 takes nearly 44 hours

Tarsnap, 2012. Tarsnap central server and S3-backed storage.

Organisation
Tarsnap
Evidence
Incident report
Source
Tarsnap outage

Daemonic Dispatches (Colin Percival's blog), published 2012-07-04

Records

Failure#f1

An EC2 power loss corrupted the metadata disk, and a full rebuild from S3 ran slowly through recovery code only ever tested on small data

Decision#d1

Keep all user data in S3 and only cached metadata on the EC2 server so that all server state can be rebuilt

Decision#d2

Treat the corrupted local state as untrustworthy and rebuild it all from S3 instead of trusting it

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.