Catalogue / github-2024-push-processing

GitHub splits its monolithic RepositoryPushJob into a Kafka push topic fanning out to isolated per-owner jobs

GitHub, 2024. Repository push processing pipeline.

Organisation
GitHub
Evidence
Reported from production
Source
How we improved push processing on GitHub

The GitHub Blog, published 2024-06-11

Records

Decision#d1

Replace sequential monolithic RepositoryPushJob with a Kafka topic fanning out to isolated per-owner background jobs

Decision#d2

Use consistent per-event feature flagging to cut over push processing without data loss or double processing

Failure#f1

Recurring incidents where errors on the Pushes MySQL cluster early in RepositoryPushJob kept unrelated pull requests from synchronizing

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.