Catalogue / slack-2022-checkpoint-circuit-breakers

Slack adds orchestration-level circuit breakers to its Checkpoint CI/CD job queue to stop cascading failures

Slack, 2022. Checkpoint CI/CD orchestration service.

Organisation
Slack
Evidence
Reported from production
Source
Slowing Down to Speed Up – Circuit Breakers for Slack’s CI/CD

Slack Engineering, published 2022-08-19

Records

Failure#f1

Recurring 2020 CI/CD cascading failures when growth peaks overloaded Git, Checkpoint queues, and CI search

Decision#d1

Put two-state, Trickster-backed circuit breakers with deferral and load shedding in the Checkpoint orchestrator

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.