Decision#d1
Catalogue / slack-2022-kafka-clusters
Slack automates 10 Kafka clusters managing 0.7 PB with Cruise Control, Terraform, and dedicated topic clusters
Slack, 2022. Self-driving Kafka cluster fleet.
Records
Decision#d2
Prevent Kafka broker hot-spotting with broker-multiple partition counts, throttled small-batch rebalances, and Cruise Control
Decision#d3
Give each Kafka cluster its own dedicated ZooKeeper cluster to limit coordination blast radius
Decision#d4
Upgrade Kafka by cutting over to a dark-traffic-validated cluster instead of upgrading in place
Decision#d5
Split three of the largest topics off a saturated 90-broker cluster onto smaller dedicated Nitro d3en clusters
Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.