Catalogue / uber-2022-spark-clp-logging

Uber splits CLP compression into a streaming Log4j appender to compress 5.38 PB of Spark logs 169x

Uber. Spark big-data platform logging pipeline (CLP Log4j integration).

Organisation
Uber
Evidence
Reported from production
Source
Reducing Logging Cost by Two Orders of Magnitude using CLP

Uber Engineering, published 2022-09-29

Records

Decision#d1

Split CLP into a streaming Phase 1 Log4j appender and offline Phase 2 aggregation

Decision#d2

Replace IEEE-754 float encoding in CLP with a custom lossless 32-bit decimal encoding

Failure#f1

Uncompressed streaming Spark log writes caused SSD write-amplification and premature SSD burnout

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.