Catalogue / lyft-2023-model-inference-profiling

Lyft profiles model inference logs across models using Spark, Fugue, and mergeable Whylogs blobs

Lyft. LyftLearn distributed model inference profiling pipeline.

Organisation
Lyft
Evidence
Reported from production
Source
Building a large scale unsupervised model anomaly detection system — Part 1

Lyft Engineering, published 2023-04-21

Records

Decision#d1

Summarize heterogeneous model inference logs into mergeable single-pass Whylogs profiles

Decision#d2

Distribute hourly profile generation and daily merges across Spark on Kubernetes via Fugue

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.