Catalogue / slack-2022-kafka-clusters

Slack automates 10 Kafka clusters managing 0.7 PB with Cruise Control, Terraform, and dedicated topic clusters

Slack, 2022. Self-driving Kafka cluster fleet.

Organisation
Slack
Evidence
Reported from production
Source
Building Self-driving Kafka clusters using open source components

Slack Engineering, published 2022-02-11

Records

Decision#d1

Automate ZooKeeper replacement with ZooKeeper 3.6, Consul KV node IDs, and Terraform Auto Scaling Groups

Decision#d2

Prevent Kafka broker hot-spotting with broker-multiple partition counts, throttled small-batch rebalances, and Cruise Control

Decision#d3

Give each Kafka cluster its own dedicated ZooKeeper cluster to limit coordination blast radius

Decision#d4

Upgrade Kafka by cutting over to a dark-traffic-validated cluster instead of upgrading in place

Decision#d5

Split three of the largest topics off a saturated 90-broker cluster onto smaller dedicated Nitro d3en clusters

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.