Failure#f1
Catalogue / gitlab-2017-database-outage
GitLab.com's primary database is wiped during a manual replica rebuild, and the pg_dump backups turn out to be missing, leaving a staging copy to restore from
GitLab, 2017. GitLab.com PostgreSQL database.
Records
Failure#f2
Daily pg_dump backups had been silently failing, and Azure disk snapshots were not enabled on the database servers
Decision#d1
Restore from a staging copy of the six-hour-old manual snapshot, not the 24-hour-old one
Decision#d2
Spend the effort on recoverability and on making the host obvious, not on blocking dangerous commands
Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.