Catalogue / uber-2024-job-counting-pinot

Uber moves job-tenure counting to an Apache Pinot hybrid table tuned with sorted segments, Bloom filters and 32 daily segments

Uber, 2024. User Tenure Store (job-counting service on Apache Pinot).

Organisation
Uber
Evidence
Reported from production
Source
How Uber Accomplishes Job Counting At Scale

Uber Blog, published 2024-05-22

Records

Decision#d1

Replace paginated job lookups with a dedicated Apache Pinot hybrid table for arbitrary-window job tenure counts

Failure#f1

Full-dataset production traffic acted like full table scans on Pinot, maxing out SSDs and breaching 10-second broker timeouts

Decision#d2

Prune off-heap MMAP Pinot segments with on-heap Bloom filters, 32 daily segments and a 30-minute consumer cache

Failure#f2

Periodic 10-second rate-limiter bursts from upstream breached Pinot's 10-second broker timeout

Decision#d3

Cover upstream trip data up to a week late with dynamic T-7d backfills and T-1d Hive COUNT(*) checks

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.