Catalogue / zendesk-2024-kafka-kubernetes-migration

Zendesk migrates 12 Kafka clusters (300 TB, 30B messages/day) from EC2 to Kubernetes via hybrid broker-level replica moves

Zendesk. Apache Kafka event streaming infrastructure.

Organisation
Zendesk
Evidence
Reported from production
Source
Seamless Transition: Migrating Kafka Cluster to Kubernetes

Zendesk Engineering, published 2024-02-13

Records

Decision#d1

Migrate Kafka from EC2 to Kubernetes by joining new brokers into the existing cluster instead of MirrorMaker2

Decision#d2

Extend Kubernetes Pod Disruption Budgets with an Under Replicated Partitions monitor covering EC2 brokers

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.