Catalogue / notion-2024-data-lake

Notion builds an S3, Debezium, Kafka, Hudi, and Spark data lake for update-heavy sharded Postgres data

Notion, 2024. In-house data lake.

Organisation
Notion
Evidence
Reported from production
Source
Building and scaling Notion's data lake

Notion Blog, published 2024-07-01

Records

Decision#d1

Build an S3 and Spark data lake for update-heavy block data while keeping Snowflake and Fivetran for simpler workloads

Decision#d2

Use Debezium-to-Kafka incremental CDC with one topic per table and RDS export-to-S3 only for bootstrapping

Decision#d3

Choose Apache Hudi over Iceberg and Delta Lake and align S3 partitioning with Postgres's 480 shards

Decision#d4

Land raw CDC data in S3 before processing and handle skewed shards differently in Scala Spark

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.