Catalogue / cloudflare-2023-control-plane-outage

Power loss at one Oregon data center exposes hidden dependencies in Cloudflare's active-active control plane and disrupts raw logs

Cloudflare, 2023. Control plane and analytics infrastructure.

Organisation
Cloudflare
Evidence
Incident report
Source
Post mortem on the Cloudflare Control Plane and Analytics Outage

The Cloudflare Blog, published 2023-11-04

Records

Failure#f1

Hidden dependencies on PDX-04-only Kafka and ClickHouse broke HA control-plane services during a full-facility power loss

Failure#f2

Failing over control-plane services to European disaster recovery sites triggered an API thundering herd

Failure#f3

Faulty breakers and a full cold bootstrap of PDX-04 prolonged recovery; log push data from the outage was not recovered

Decision#d1

Run core control plane active-active across three Oregon facilities while excluding logging from the HA cluster

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.