Catalogue / datadog-2023-multiregion-systemd-outage

A systemd security update applied automatically through a legacy update channel deletes Cilium CNI routes in five Datadog regions at the same time

Datadog, 2023. Multi-region telemetry and monitoring platform.

Organisation
Datadog
Evidence
Incident report
Source
2023-03-08 Incident: Infrastructure connectivity issue affecting multiple regions

Datadog (The Monitor), published 2023-05-16

Records

Failure#f1

Unintended automatic systemd update deleted Cilium CNI routes across five independent regions at once

Decision#d1

Separate live telemetry intake and serving systems from historical storage

Decision#d2

Use statically sharded share-nothing stores for primary telemetry and quorum-based stores for metadata

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.