Catalogue / github-2023-code-search

GitHub builds Blackbird, its own Rust code search engine, with a 25 TB index sharded by Git blob ID

GitHub, 2023. Code search (Blackbird).

Organisation
GitHub
Evidence
Reported from production
Source
The technology behind GitHub’s new code search

The GitHub Blog, published 2023-02-06

Records

Decision#d1

Build a custom code search engine in Rust instead of general text search, because indexing was slow and costly

Decision#d2

Shard the search index by Git blob ID and store it as a delta-encoded tree to remove duplicate content

Decision#d3

Decouple crawling from indexing with Kafka, one partition per shard, to get commit-consistent queries

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.