Catalogue

Cards
165
Records
486
Workloads
19
Corpus version
74847648e25a
Workload
Kind
Setup

26 of 486 records match; the first 20 are shown.

KindRecordOrganisationYearSetup
Decision

Build an S3 and Spark data lake for update-heavy block data while keeping Snowflake and Fivetran for simpler workloads

notion-2024-data-lake#d1

Notion2024Sharded
Decision

Use Debezium-to-Kafka incremental CDC with one topic per table and RDS export-to-S3 only for bootstrapping

notion-2024-data-lake#d2

Notion2024Sharded
Decision

Choose Apache Hudi over Iceberg and Delta Lake and align S3 partitioning with Postgres's 480 shards

notion-2024-data-lake#d3

Notion2024Sharded
Decision

Land raw CDC data in S3 before processing and handle skewed shards differently in Scala Spark

notion-2024-data-lake#d4

Notion2024Sharded
Decision

Replace paginated job lookups with a dedicated Apache Pinot hybrid table for arbitrary-window job tenure counts

uber-2024-job-counting-pinot#d1

Uber2024Sharded
Failure

Full-dataset production traffic acted like full table scans on Pinot, maxing out SSDs and breaching 10-second broker timeouts

uber-2024-job-counting-pinot#f1

Uber2024Sharded
Decision

Prune off-heap MMAP Pinot segments with on-heap Bloom filters, 32 daily segments and a 30-minute consumer cache

uber-2024-job-counting-pinot#d2

Uber2024Sharded
Failure

Periodic 10-second rate-limiter bursts from upstream breached Pinot's 10-second broker timeout

uber-2024-job-counting-pinot#f2

Uber2024Sharded
Decision

Cover upstream trip data up to a week late with dynamic T-7d backfills and T-1d Hive COUNT(*) checks

uber-2024-job-counting-pinot#d3

Uber2024Sharded
Decision

Replace ad-hoc multi-hop legacy pipelines with one open-source Kafka and Flink platform

doordash-2022-iguazu#d1peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Put a customized async Confluent Kafka Rest Proxy in front of Kafka instead of direct clients in every service

doordash-2022-iguazu#d2peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Use replication factor 2, leader-only acknowledgments and 50ms to 100ms linger for non-transactional events

doordash-2022-iguazu#d3peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Register Protobuf event schemas in Confluent Schema Registry at build time in CI/CD instead of at runtime

doordash-2022-iguazu#d4peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Stage Kafka events to S3 Parquet via per-event Flink jobs before loading Snowflake with Snowpipe

doordash-2022-iguazu#d5peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Automate multi-repo Terraform event onboarding with a GitHub App, Cadence workflows and Retool UIs

doordash-2022-iguazu#d6peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Automate ZooKeeper replacement with ZooKeeper 3.6, Consul KV node IDs, and Terraform Auto Scaling Groups

slack-2022-kafka-clusters#d1peak of 100,000 requests/s or more

Slack2022Sharded
Decision

Prevent Kafka broker hot-spotting with broker-multiple partition counts, throttled small-batch rebalances, and Cruise Control

slack-2022-kafka-clusters#d2peak of 100,000 requests/s or more

Slack2022Sharded
Decision

Give each Kafka cluster its own dedicated ZooKeeper cluster to limit coordination blast radius

slack-2022-kafka-clusters#d3peak of 100,000 requests/s or more

Slack2022Sharded
Decision

Upgrade Kafka by cutting over to a dark-traffic-validated cluster instead of upgrading in place

slack-2022-kafka-clusters#d4peak of 100,000 requests/s or more

Slack2022Sharded
Decision

Split three of the largest topics off a saturated 90-broker cluster onto smaller dedicated Nitro d3en clusters

slack-2022-kafka-clusters#d5peak of 100,000 requests/s or more

Slack2022Sharded

6 more records. Sign in to see all 26.