Catalogue / hulu-2019-metrics-pipeline

Hulu scales a dual-datacenter InfluxDB and Kafka metrics pipeline past 1 million metrics per second with dynamic filtering

Hulu, 2019. Time series metrics pipeline.

Organisation
Hulu
Evidence
Reported from production
Source
How Hulu Uses InfluxDB and Kafka to Scale to Over 1 Million Metrics a Second

Hulu Tech blog, published 2019-07-25

Records

Decision#d1

Replace shared Graphite and OpenTSDB with dual-datacenter InfluxDB, Telegraf and tag-enforcing relays

Failure#f1

Outage of one InfluxDB cluster dropped metrics from direct relays and left datacenters out of sync

Decision#d2

Buffer metrics in per-datacenter Kafka queues and fan in via local InfluxDB writers

Decision#d3

Poll both InfluxDB clusters every minute with Influx Imposer to kill slow or dangerous queries

Decision#d4

Filter high-cardinality bad metrics dynamically at the InfluxDB writer layer via Barricade

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.