Decision#d1
Catalogue / github-2023-code-search
GitHub builds Blackbird, its own Rust code search engine, with a 25 TB index sharded by Git blob ID
GitHub, 2023. Code search (Blackbird).
Records
Decision#d2
Shard the search index by Git blob ID and store it as a delta-encoded tree to remove duplicate content
Decision#d3
Decouple crawling from indexing with Kafka, one partition per shard, to get commit-consistent queries
Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.