Catalogue / cloudflare-2022-kafka-messagebus

Cloudflare wraps its inter-service Kafka clusters with a Go client, Protobuf schemas, a connector framework and GitOps lag alerts

Cloudflare, 2022. Inter-service Apache Kafka platform (Messagebus).

Organisation
Cloudflare
Evidence
Reported from production
Source
Using Apache Kafka to process 1 trillion inter-service messages

The Cloudflare Blog, published 2022-07-19

Records

Decision#d1

Wrap Kafka behind an opinionated Go client library and a templated connector framework to lower adoption cost

Decision#d2

Standardize inter-service Kafka messages on Protobuf in a central schema repo with breaking-change checks and team ownership

Decision#d3

Enforce one Protobuf message type per Kafka topic on Messagebus

Decision#d4

Provision Kafka topics via Salt GitOps with automatic high-lag alerts and per-topic Grafana views

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.