Decision#d1
Catalogue / hulu-2019-metrics-pipeline
Hulu scales a dual-datacenter InfluxDB and Kafka metrics pipeline past 1 million metrics per second with dynamic filtering
Hulu, 2019. Time series metrics pipeline.
Records
Failure#f1
Outage of one InfluxDB cluster dropped metrics from direct relays and left datacenters out of sync
Decision#d2
Buffer metrics in per-datacenter Kafka queues and fan in via local InfluxDB writers
Decision#d3
Poll both InfluxDB clusters every minute with Influx Imposer to kill slow or dangerous queries
Decision#d4
Filter high-cardinality bad metrics dynamically at the InfluxDB writer layer via Barricade
Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.