Catalogue / notion-2021-postgres-sharding

Notion shards its Postgres monolith by workspace ID into 480 logical shards

Notion, 2021. Postgres block store.

Organisation
Notion
Evidence
Reported from production
System
Postgres block store
Setup
Sharded

Near: a fleet of 32 physical databases, each containing

As of
2021
Source
Herding elephants: Lessons learned from sharding Postgres at Notion

Notion, published 2021-10-06

Architecture

Notion moved its block data, and every table related to it, from one Postgres monolith to 480 logical shards, each a Postgres schema, spread evenly over 32 physical databases; the application routes each query by workspace ID.

Section: Decision 3: Capacity planning | Section: Decision 2: Partition block data by workspace ID | Section: Decision 1: Shard all data transitively related to block

Figures, as the source states them

FigureAs statedNoteWhere
IOPS needed to meet existing demand60,000 IOPSAt least; disk throughput, not request rateSection: Decision 3: Capacity planning
Upper bound per table500 GBDesign limit chosen for the shard layout, not a stored sizeSection: Decision 3: Capacity planning
Upper bound per physical database10 TBDesign limit chosen for the shard layout, not a stored sizeSection: Decision 3: Capacity planning
Logical shards480 shardsChosen layoutSection: Decision 3: Capacity planning
Physical databases32 databasesChosen layoutSection: Decision 3: Capacity planning

Records

Decision#d1

Shard the Postgres monolith in application code once autovacuum stalled, rather than scale up or adopt a packaged sharding product

notion-2021-postgres-sharding#d1 at corpus version 74847648e25a
Chosen

Notion wrote its own partitioning scheme and routes queries from application logic across several Postgres databases, instead of adopting a packaged sharding product.

Section: Application-level sharding

Rejected

Packaged sharding or clustering such as Citus or Vitess. They appeal for simplicity and built-in cross-shard tooling, but their clustering logic is opaque and the team wanted control over how its data is distributed.

Section: Application-level sharding

Switch to another database system such as DynamoDB. Judged too risky for Notion's workload.

Near: switching to another database system such as DynamoDB

Run Postgres on bare-metal NVMe-heavy instances for more disk throughput. Rejected because of the maintenance cost of backups and replication.

Near: running Postgres on bare-metal NVMe heavy instances

What forced it

By mid-2020 growth was set to exceed what the monolith could do. Engineers on call often woke up to database CPU spikes, simple catalog-only migrations had become unsafe and uncertain, and VACUUM stalled consistently, raising the threat of transaction ID wraparound, at which Postgres stops accepting writes.

Section: Deciding when to shard

Reasoning

The team wanted to control how its data is distributed across databases itself. Resizing the RDS instance was, in its words, not a viable long-term strategy, because query performance and upkeep processes tend to degrade well before a table reaches the hardware's size limit, as the stalling autovacuum showed.

Section: Application-level sharding | Section: Designing a sharding scheme

What worked before

The one Postgres database had handled five years of growth, spanning four orders of magnitude.

Section: Deciding when to shard

Cost

The team waited until the monolith was heavily strained, so migrations had to be very frugal about adding load. Shard earlier heads the list of things the authors say they would reconsider.

Section: Difficult lessons learned

Outcome

The switch-over was done in five minutes of scheduled maintenance, after which the authors say users quickly noticed the improvement, quoting one user's post that the product felt much faster. The authors' own verdict is that the sharding was a tremendous success.

Near: we took Notion down for five minutes | Near: users quickly began to notice the improvement | Section: Difficult lessons learned

Reviewer's notes

The post gives no request rate or stored-data total for the monolith, so the point where VACUUM stalled cannot be tied to a size.

The reviewer's reading, not the source's claim

Decision#d2

Partition by workspace ID and shard every table reachable from the block table

notion-2021-postgres-sharding#d2 at corpus version 74847648e25a
Chosen

Block data and its related rows are partitioned by workspace ID. Each workspace receives a UUID when created, and the UUID space is split into even buckets.

Section: Decision 2: Partition block data by workspace ID

Reasoning

Every table reachable from block by foreign key is sharded with it, because a record left on another datastore from its block could become inconsistent (a block deletion could succeed while its comment update failed), since transactions only apply within one datastore. Notion is team-based, every sharded row is a block or belongs to one, and each block sits in exactly one workspace; since users mostly query within one workspace, this key avoids most cross-shard joins.

Section: Decision 1: Shard all data transitively related to block | Section: Decision 2: Partition block data by workspace ID

Cost

Sharded rows carry both the old primary key and the space ID as partition key, so the application must pass space IDs around. The authors say one combined key would have avoided this.

Section: Difficult lessons learned

Reviewer's notes

The first footnote's warning about a shard key that stops matching the product model is a general remark, not something Notion reports having hit; the card records the workspace key only as Notion's own choice.

The reviewer's reading, not the source's claim

Decision#d3

Move to shards with an audit log and catch-up script, not logical replication, because the strained monolith left little room for extra load

notion-2021-postgres-sharding#d3 at corpus version 74847648e25a
Chosen

Writes to the tables being moved were recorded in an audit log table, and a catch-up process replayed them onto the shards, changing rows where needed. A reverse audit log and script were prepared and tested in case of a fall back to the monolith.

Section: Double-writing with an audit log

Rejected

Write directly to both databases. A failure of either write can quickly put the two databases out of step, too unreliable for datastores on the critical path.

Section: Double-writing with an audit log

Postgres logical replication. Struggled to keep up with the block table's write volume during the initial snapshot step, and has limited ability to modify data of rows on the way to the target.

Section: Double-writing with an audit log

What forced it

The monolith was already under heavy strain, so the migration could add very little load, which ruled out logical replication. The workspace ID column was empty there and filling it would have added load, so it was set on each row as it was written to the shards.

Section: Difficult lessons learned

Cost

Double-write throughput was the main bottleneck in the final switch-over, because the server was taken down and had to wait for the catch-up script to finish propagating writes to the shards. The authors think another week of tuning, to under 30 seconds of catch-up, might have allowed a switch with no downtime.

Section: Difficult lessons learned

Outcome

The production backfill of existing data, run on one m5.24xlarge instance using all 96 of its CPUs, took around three days. The reverse audit log and script were not needed, since the team did not have to revert to the monolith.

Section: Backfilling old data | Section: Double-writing with an audit log

Review

Extracted by
claude-sonnet-5-5
Human review
Not started. Status recorded on 2026-11-16.