Catalogue / cloudflare-2024-alerts-observability

Cloudflare moves alert analytics from alertmanager2es to Vector and ClickHouse to track all Alertmanager states

Cloudflare, 2024. Alerts observability pipeline (Prometheus, central Alertmanager, Vector, ClickHouse, Grafana).

Organisation
Cloudflare
Evidence
Reported from production
Source
Minimizing on-call burnout through alerts observability

The Cloudflare Blog, published 2024-03-29

Records

Decision#d1

Ingest Alertmanager webhooks and APIs via Vector into ClickHouse with a Map column for dynamic labels

Failure#f1

Alertmanager inhibitions sometimes failed, and the failed ones had circular inhibition rules, found only when users reported being alerted

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.