Catalogue / ebay-2023-nugraph-export

eBay exports 15-billion-vertex NuGraph graphs to HDFS from a FoundationDB DR cluster via Spark and JVM fixes

eBay, 2023. NuGraph graph export ETL pipeline.

Organisation
eBay
Evidence
Reported from production
Source
How We Export Billion-Scale Graphs on Transactional Graph Databases

eBay Innovation, published 2023-11-15

Records

Decision#d1

Scan a continuously synchronized single-datacenter FoundationDB DR cluster for full-graph exports instead of restoring backups

Decision#d2

Build a Spark plugin with resumable 5-second iterators, vertex-aligned splits and supernode partition grouping

Decision#d3

Replace JanusGraph's Guava LRU VertexCache with Caffeine to eliminate over-10% Spark GC overhead

Decision#d4

Remove unnecessary Spark shuffles in TinkerPop CloneVertexProgram and shrink ObjectWritable.toString() to stop OOMs

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.