Catalogue / doordash-2022-iguazu

DoorDash builds Iguazu on Kafka, Flink and S3 to stream hundreds of billions of events per day

DoorDash, 2022. Iguazu, real-time event processing platform.

Organisation
DoorDash
Evidence
Reported from production
Source
Building scalable real time event processing with Kafka and Flink

DoorDash Engineering Blog, published 2022-10-21

Records

Decision#d1

Replace ad-hoc multi-hop legacy pipelines with one open-source Kafka and Flink platform

Decision#d2

Put a customized async Confluent Kafka Rest Proxy in front of Kafka instead of direct clients in every service

Decision#d3

Use replication factor 2, leader-only acknowledgments and 50ms to 100ms linger for non-transactional events

Decision#d4

Register Protobuf event schemas in Confluent Schema Registry at build time in CI/CD instead of at runtime

Decision#d5

Stage Kafka events to S3 Parquet via per-event Flink jobs before loading Snowflake with Snowpipe

Decision#d6

Automate multi-repo Terraform event onboarding with a GitHub App, Cadence workflows and Retool UIs

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.