Catalogue

Cards
165
Records
486
Workloads
19
Corpus version
74847648e25a
Workload
Kind
Setup

19 of 486 records match.

KindRecordOrganisationYearSetup
Decision

Build an S3 and Spark data lake for update-heavy block data while keeping Snowflake and Fivetran for simpler workloads

notion-2024-data-lake#d1

Notion2024Sharded
Decision

Use Debezium-to-Kafka incremental CDC with one topic per table and RDS export-to-S3 only for bootstrapping

notion-2024-data-lake#d2

Notion2024Sharded
Decision

Choose Apache Hudi over Iceberg and Delta Lake and align S3 partitioning with Postgres's 480 shards

notion-2024-data-lake#d3

Notion2024Sharded
Decision

Land raw CDC data in S3 before processing and handle skewed shards differently in Scala Spark

notion-2024-data-lake#d4

Notion2024Sharded
Decision

Replace ad-hoc multi-hop legacy pipelines with one open-source Kafka and Flink platform

doordash-2022-iguazu#d1peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Put a customized async Confluent Kafka Rest Proxy in front of Kafka instead of direct clients in every service

doordash-2022-iguazu#d2peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Use replication factor 2, leader-only acknowledgments and 50ms to 100ms linger for non-transactional events

doordash-2022-iguazu#d3peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Register Protobuf event schemas in Confluent Schema Registry at build time in CI/CD instead of at runtime

doordash-2022-iguazu#d4peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Stage Kafka events to S3 Parquet via per-event Flink jobs before loading Snowflake with Snowpipe

doordash-2022-iguazu#d5peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Automate multi-repo Terraform event onboarding with a GitHub App, Cadence workflows and Retool UIs

doordash-2022-iguazu#d6peak of 100,000 requests/s or more

DoorDash2022Sharded
Decision

Automate ZooKeeper replacement with ZooKeeper 3.6, Consul KV node IDs, and Terraform Auto Scaling Groups

slack-2022-kafka-clusters#d1peak of 100,000 requests/s or more

Slack2022Sharded
Decision

Prevent Kafka broker hot-spotting with broker-multiple partition counts, throttled small-batch rebalances, and Cruise Control

slack-2022-kafka-clusters#d2peak of 100,000 requests/s or more

Slack2022Sharded
Decision

Give each Kafka cluster its own dedicated ZooKeeper cluster to limit coordination blast radius

slack-2022-kafka-clusters#d3peak of 100,000 requests/s or more

Slack2022Sharded
Decision

Upgrade Kafka by cutting over to a dark-traffic-validated cluster instead of upgrading in place

slack-2022-kafka-clusters#d4peak of 100,000 requests/s or more

Slack2022Sharded
Decision

Split three of the largest topics off a saturated 90-broker cluster onto smaller dedicated Nitro d3en clusters

slack-2022-kafka-clusters#d5peak of 100,000 requests/s or more

Slack2022Sharded
Decision

Migrate chat history from a single MongoDB replica set to Cassandra once data and index outgrew RAM

discord-2017-message-storage#d1

Discord2017Sharded
Decision

Bound Cassandra partition sizes with 10-day time buckets and Snowflake clustering keys

discord-2017-message-storage#d2

Discord2017Sharded
Decision

Delete corrupt edit/delete race rows when author_id is null and omit null columns on Cassandra writes

discord-2017-message-storage#d3

Discord2017Sharded
Failure

Millions of message tombstones in one channel triggered 10-second Cassandra stop-the-world GC pauses

discord-2017-message-storage#f1

Discord2017Sharded