Catalogue / lyft-2023-clickhouse-analytics

Lyft consolidates sub-second analytics on ClickHouse and sunsets Apache Druid

Lyft. Sub-second near real-time analytics platform (ClickHouse).

Organisation
Lyft
Evidence
Reported from production
Source
Druid Deprecation and ClickHouse Adoption at Lyft

Lyft Engineering, published 2023-11-29

Records

Decision#d1

Sunset Apache Druid and consolidate sub-second analytics onto unsharded HA ClickHouse on Kubernetes

Decision#d2

Validate and cut over Druid queries to ClickHouse via dynamic query transpilation and tiered traffic shifting

Decision#d3

Eliminate ClickHouse full table scans with time-series sorting keys, minmax skip indices, and materialized projections

Failure#f1

ZooKeeper conflict resolution marks ClickHouse tables read-only and causes Flink push-ingestion failures

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.