Catalogue / lyft-2023-verity-data-quality

Lyft builds Verity, an SQS- and DynamoDB-backed asynchronous platform running 140,000 weekly Hive data quality checks

Lyft, 2023. Verity data quality platform for Hive Data Warehouse.

Organisation
Lyft
Evidence
Reported from production
Source
From Big Data to Better Data: Ensuring Data Quality with Verity

Lyft Engineering, published 2023-10-03

Records

Decision#d1

Split data quality checking into four loosely coupled backend services connected by SQS queues

Decision#d2

Isolate traffic classes with dedicated SQS Check Queue and Executor pairs

Decision#d3

Store Check Results in DynamoDB for key-and-time lookups without database maintenance

Decision#d4

Run scheduled checks in Verity's own scheduler and gate Airflow/Flyte ETLs with thin SDK-backed blocking operators

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.