Catalogue / uber-2022-presto-alluxio-cache

Uber adds selective Alluxio local NVMe caching, timestamped keys, and consistent hashing to Presto

Uber. Presto analytics clusters with Alluxio local NVMe cache.

Organisation
Uber
Evidence
Reported from production
Source
Speed Up Presto at Uber with Alluxio Local Cache

Uber Engineering Blog, published 2022-10-26

Records

Decision#d1

Embed Alluxio local NVMe cache in Presto workers with a static Cache Filter for selective tables

Decision#d2

Include HDFS file latest modification time in cache keys to handle real-time Hudi upserts

Decision#d3

Replace modulo-based Soft Affinity Scheduling with node-ID consistent hashing and replication

Decision#d4

Persist file-level protobuf cache metadata on disk with timestamped version directories

Decision#d5

Propagate HiveFileContext and PrestoCacheContext to Alluxio with callbacks into RuntimeStats

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.