Catalogue / linkedin-2022-brooklin-mirror-maker

LinkedIn balances Kafka mirroring across data centers by per-partition throughput using Cruise Control, Samza and Pinot

LinkedIn, 2022. Brooklin Mirror Maker (BMM).

Organisation
LinkedIn
Evidence
Reported from production
Source
Load-balanced Brooklin Mirror Maker: Replicating large-scale Kafka clusters at LinkedIn

LinkedIn Engineering Blog, published 2022-04-11

Records

Decision#d1

Assign Kafka mirroring partitions by measured throughput instead of partition count using manual consumer assignment

Decision#d2

Collect partition throughput via a separate Cruise Control topic, a Samza Avro merger and Apache Pinot

Decision#d3

Drive bounded elastic task scaling and latency-triggered rebalances from a monitoring service

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.