Decision#d1
Catalogue / uber-2022-spark-clp-logging
Uber splits CLP compression into a streaming Log4j appender to compress 5.38 PB of Spark logs 169x
Uber. Spark big-data platform logging pipeline (CLP Log4j integration).
Records
Decision#d2
Replace IEEE-754 float encoding in CLP with a custom lossless 32-bit decimal encoding
Failure#f1
Uncompressed streaming Spark log writes caused SSD write-amplification and premature SSD burnout
Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.