Catalogue / instacart-2023-ray-distributed-ml

Instacart replaces shared Celery ML task queues with ephemeral KubeRay clusters on AWS EKS

Instacart, 2023. Distributed ML training platform on Griffin (Ray on Kubernetes).

Organisation
Instacart
Evidence
Reported from production
Source
Distributed Machine Learning at Instacart

tech-at-instacart, published 2023-03-17

Records

Decision#d1

Move zone-level parallel ML training from a shared long-running Celery queue to per-application Ray clusters on EKS

Decision#d2

Isolate Python environments per Ray cluster behind a unified Griffin launcher API for dev and Airflow runs

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.