Catalogue / linkedin-2022-venice

LinkedIn runs Venice across 4,000 hosts for asynchronous derived-data serving, Read Compute, and Da Vinci eager caching

LinkedIn, 2022. Venice derived data platform.

Organisation
LinkedIn
Evidence
Reported from production
Source
Open Sourcing Venice - LinkedIn's Derived Data Platform

LinkedIn Engineering Blog, published 2022-09-26

Records

Decision#d1

Restrict Venice to asynchronous batch and stream writes with background version swaps, server-side partial updates, and hybrid replay

Decision#d2

Push vector math onto Venice servers with Read Compute and embed Da Vinci as an eager local-state client library

Decision#d3

Swap BDB-JE for pluggable RocksDB storage formats and operator-driven Zstandard shared-dictionary compression

Decision#d4

Run active-active multi-region replication and rack-aware Apache Helix self-healing reusing the Full Push rebalance path

Decision#d5

Enforce per-dataset storage and read quotas on multi-tenant clusters and group over-replicated partitions to bound fanout

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.