Catalogue / roblox-2021-consul-outage

Consul streaming lock contention and BoltDB freelist write amplification cause a 73-hour outage at Roblox

Roblox, 2021. Core infrastructure service discovery and coordination (HashiStack / Consul).

Organisation
Roblox
Evidence
Incident report
Source
Roblox Return to Service

Roblox Blog, published 2022-01-20

Records

Failure#f1

Consul streaming channel contention and BoltDB freelist write amplification stall the sole coordination cluster

Failure#f2

Cold-start cache deployment failures prolong recovery after Consul stabilises

Decision#d1

Keep performance- and latency-critical core and edge workloads on-prem while using public cloud for burst, DevOps and analytics

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.