Catalogue / grafana-2021-hosted-prometheus-outage

A limit-enforcement bug lets one new tenant overload a shared Grafana Cloud metrics cluster, and the failure cascades through the stack

Grafana Labs, 2021. Grafana Cloud Hosted Prometheus (Grafana Enterprise Metrics clusters).

Organisation
Grafana Labs
Evidence
Incident report
Source
How we responded to a 2-hour outage in our Grafana Cloud Hosted Prometheus service

Grafana Labs blog, published 2021-03-27

Records

Failure#f1

A bug in per-tenant limit enforcement let one tenant overload a shared cluster, and error handling made the overload spread

Decision#d1

Share metrics clusters between tenants under per-tenant data and rate limits, and size clusters partly from those limits

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.