Catalogue / meta-2026-data-ingestion-migration

Meta migrates tens of thousands of MySQL-to-warehouse CDC ingestion jobs via automated shadow and reverse-shadow phases

Meta, 2026. Social graph MySQL-to-data-warehouse CDC ingestion system.

Organisation
Meta
Evidence
Reported from production
Source
Migrating Data Ingestion Systems at Meta Scale

Engineering at Meta, published 2026-05-12

Records

Decision#d1

Replace customer-owned social-graph ingestion pipelines with a self-managed data warehouse ingestion service

Decision#d2

Migrate jobs through three gated phases—shadow, reverse shadow and cleanup—verified on row counts, checksums, latency and resources

Decision#d3

Contain CDC bad-data propagation via paired reverse-shadow backfills and metadata bad-partition flags

Decision#d4

Batch shadow migrations under limited capacity, holding back jobs with known bugs and reusing legacy snapshots

Decision#d5

Automate lifecycle promotion from Scuba signals and run hourly mismatch row-level debugging

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.