Catalogue / shopify-2022-flink-bigquery-clustering

Shopify clusters a 1-billion-row BigQuery table on WHERE columns to cut a projected $949k/month query to $1.37k

Shopify. Merchant marketing tool data pipeline (Apache Flink and BigQuery).

Organisation
Shopify
Evidence
Reported from production
Source
Reducing BigQuery Costs: How We Fixed A $1 Million Query

Shopify Engineering, published 2022-11-03

Records

Decision#d1

Explore offloading a 1-billion-row dataset from Flink RocksDB internal state to BigQuery for general availability

Decision#d2

Cluster the BigQuery table on two WHERE-clause columns to cut per-query bytes scanned

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.