Decision#d1
Catalogue / notion-2024-data-lake
Notion builds an S3, Debezium, Kafka, Hudi, and Spark data lake for update-heavy sharded Postgres data
Notion, 2024. In-house data lake.
Records
Decision#d2
Use Debezium-to-Kafka incremental CDC with one topic per table and RDS export-to-S3 only for bootstrapping
Decision#d3
Choose Apache Hudi over Iceberg and Delta Lake and align S3 partitioning with Postgres's 480 shards
Decision#d4
Land raw CDC data in S3 before processing and handle skewed shards differently in Scala Spark
Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.