Notion moved its block data, and every table related to it, from one Postgres monolith to 480 logical shards, each a Postgres schema, spread evenly over 32 physical databases; the application routes each query by workspace ID.
Section: Decision 3: Capacity planning | Section: Decision 2: Partition block data by workspace ID | Section: Decision 1: Shard all data transitively related to block
Figures, as the source states them
Figure
As stated
Note
Where
IOPS needed to meet existing demand
60,000 IOPS
At least; disk throughput, not request rate
Section: Decision 3: Capacity planning
Upper bound per table
500 GB
Design limit chosen for the shard layout, not a stored size
Section: Decision 3: Capacity planning
Upper bound per physical database
10 TB
Design limit chosen for the shard layout, not a stored size
Shard the Postgres monolith in application code once autovacuum stalled, rather than scale up or adopt a packaged sharding product
notion-2021-postgres-sharding#d1at corpus version 74847648e25a
Chosen
Notion wrote its own partitioning scheme and routes queries from application logic across several Postgres databases, instead of adopting a packaged sharding product.
Section: Application-level sharding
Rejected
Packaged sharding or clustering such as Citus or Vitess. They appeal for simplicity and built-in cross-shard tooling, but their clustering logic is opaque and the team wanted control over how its data is distributed.
Section: Application-level sharding
Switch to another database system such as DynamoDB. Judged too risky for Notion's workload.
Near: switching to another database system such as DynamoDB
Run Postgres on bare-metal NVMe-heavy instances for more disk throughput. Rejected because of the maintenance cost of backups and replication.
Near: running Postgres on bare-metal NVMe heavy instances
What forced it
By mid-2020 growth was set to exceed what the monolith could do. Engineers on call often woke up to database CPU spikes, simple catalog-only migrations had become unsafe and uncertain, and VACUUM stalled consistently, raising the threat of transaction ID wraparound, at which Postgres stops accepting writes.
Section: Deciding when to shard
Reasoning
The team wanted to control how its data is distributed across databases itself. Resizing the RDS instance was, in its words, not a viable long-term strategy, because query performance and upkeep processes tend to degrade well before a table reaches the hardware's size limit, as the stalling autovacuum showed.
Section: Application-level sharding | Section: Designing a sharding scheme
What worked before
The one Postgres database had handled five years of growth, spanning four orders of magnitude.
Section: Deciding when to shard
Cost
The team waited until the monolith was heavily strained, so migrations had to be very frugal about adding load. Shard earlier heads the list of things the authors say they would reconsider.
Section: Difficult lessons learned
Outcome
The switch-over was done in five minutes of scheduled maintenance, after which the authors say users quickly noticed the improvement, quoting one user's post that the product felt much faster. The authors' own verdict is that the sharding was a tremendous success.
Near: we took Notion down for five minutes | Near: users quickly began to notice the improvement | Section: Difficult lessons learned
Reviewer's notes
The post gives no request rate or stored-data total for the monolith, so the point where VACUUM stalled cannot be tied to a size.
Partition by workspace ID and shard every table reachable from the block table
notion-2021-postgres-sharding#d2at corpus version 74847648e25a
Chosen
Block data and its related rows are partitioned by workspace ID. Each workspace receives a UUID when created, and the UUID space is split into even buckets.
Section: Decision 2: Partition block data by workspace ID
Reasoning
Every table reachable from block by foreign key is sharded with it, because a record left on another datastore from its block could become inconsistent (a block deletion could succeed while its comment update failed), since transactions only apply within one datastore. Notion is team-based, every sharded row is a block or belongs to one, and each block sits in exactly one workspace; since users mostly query within one workspace, this key avoids most cross-shard joins.
Section: Decision 1: Shard all data transitively related to block | Section: Decision 2: Partition block data by workspace ID
Cost
Sharded rows carry both the old primary key and the space ID as partition key, so the application must pass space IDs around. The authors say one combined key would have avoided this.
Section: Difficult lessons learned
Reviewer's notes
The first footnote's warning about a shard key that stops matching the product model is a general remark, not something Notion reports having hit; the card records the workspace key only as Notion's own choice.
Move to shards with an audit log and catch-up script, not logical replication, because the strained monolith left little room for extra load
notion-2021-postgres-sharding#d3at corpus version 74847648e25a
Chosen
Writes to the tables being moved were recorded in an audit log table, and a catch-up process replayed them onto the shards, changing rows where needed. A reverse audit log and script were prepared and tested in case of a fall back to the monolith.
Section: Double-writing with an audit log
Rejected
Write directly to both databases. A failure of either write can quickly put the two databases out of step, too unreliable for datastores on the critical path.
Section: Double-writing with an audit log
Postgres logical replication. Struggled to keep up with the block table's write volume during the initial snapshot step, and has limited ability to modify data of rows on the way to the target.
Section: Double-writing with an audit log
What forced it
The monolith was already under heavy strain, so the migration could add very little load, which ruled out logical replication. The workspace ID column was empty there and filling it would have added load, so it was set on each row as it was written to the shards.
Section: Difficult lessons learned
Cost
Double-write throughput was the main bottleneck in the final switch-over, because the server was taken down and had to wait for the catch-up script to finish propagating writes to the shards. The authors think another week of tuning, to under 30 seconds of catch-up, might have allowed a switch with no downtime.
Section: Difficult lessons learned
Outcome
The production backfill of existing data, run on one m5.24xlarge instance using all 96 of its CPUs, took around three days. The reverse audit log and script were not needed, since the team did not have to revert to the monolith.
Section: Backfilling old data | Section: Double-writing with an audit log