Catalogue / gitlab-2017-database-outage

GitLab.com's primary database is wiped during a manual replica rebuild, and the pg_dump backups turn out to be missing, leaving a staging copy to restore from

GitLab, 2017. GitLab.com PostgreSQL database.

Organisation
GitLab
Evidence
Incident report
Source
Postmortem of database outage of January 31

GitLab blog, published 2017-02-10

Records

Failure#f1

Rebuilding a lagging replica by hand ended with the primary's data directory being wiped

Failure#f2

Daily pg_dump backups had been silently failing, and Azure disk snapshots were not enabled on the database servers

Decision#d1

Restore from a staging copy of the six-hour-old manual snapshot, not the 24-hour-old one

Decision#d2

Spend the effort on recoverability and on making the host obvious, not on blocking dangerous commands

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.