SourceHut runs on 10 dedicated servers and about 30 virtual machines, all provisioned by hand, with no Kubernetes, Docker or CloudFormation. Its application is split into independent miniservices that can fail separately, and many pages make only one or two database round-trips.
Near: SourceHut has 10 dedicated servers and about 30 | Near: any of our services to fail independently | Near: designed to make only one or two round-trips
Components
Component
Role
Where
Dedicated servers and virtual machines
The platform's hosts, provisioned by hand. The team prefers Alpine Linux for them, and the average production server has only 20 to 30 packages installed, so no moving parts go unexplained.
Near: Alpine Linux because it's lightweight and simple
Miniservices
A distributed set of small services, designed so that any one can fail without a total outage.
Near: any of our services to fail independently
Database access
Many pages are built to need just one or two trips to the database, so the queries behind them can be tuned with joins and indexes made for the purpose.
Near: designed to make only one or two round-trips
Email-based collaboration
Because users collaborate by email, the whole service could be offline without stopping their work, and servers catch up on redelivery when they return.
Near: our entire service could be offline without affecting
Monitoring
Monitoring alerts the team to problems before they are serious, and the raw data is exposed through a public Prometheus instance.
Near: monitoring which alerts us to problems before | Near: from our public Prometheus
Figures, as the source states them
Figure
As stated
Note
Where
Dedicated servers
10 servers
All provisioned by hand
Near: SourceHut has 10 dedicated servers and about 30
Virtual machines
30 virtual machines
About; all provisioned by hand
Near: SourceHut has 10 dedicated servers and about 30
Employees
2 employees
Plus an intern; the whole company, in the first quarter of 2020
Provision plain servers by hand on minimal Alpine Linux and avoid Kubernetes, Docker and CloudFormation
sourcehut-2020-simple-infrastructure#d1at corpus version 74847648e25a
Chosen
Production uses hand-provisioned dedicated servers and virtual machines, preferably on Alpine Linux, with few packages each, and no Kubernetes, Docker, CloudFormation or monorepos.
Near: no Kubernetes here, no Docker, no CloudFormation | Near: We prefer Alpine Linux because it's lightweight | Near: only 20 to 30 packages installed
Reasoning
Part of a culture of simplicity: servers that do not break unexpectedly and no moving parts the team does not understand, which the author ties to reliability.
Near: This culture of simplicity extends to our backend | Near: simple servers which don't break unexpectedly | Near: no moving parts that we don't understand
Outcome
The author reports two outages so far in 2020, both planned and announced a week in advance.
Near: SourceHut has had two outages in 2020
Reviewer's notes
The post contrasts its outage count with GitHub's and GitLab's and does not state its own traffic, so the reliability claim is the author's, over the first months of 2020 only.
Split the forge into independent miniservices and use email so an outage does not stop users working
sourcehut-2020-simple-infrastructure#d2at corpus version 74847648e25a
Chosen
The application is a distributed set of miniservices, so any one can fail without a total outage, and collaboration runs over email, so users can keep working when the service is offline and servers catch up afterwards.
Near: any of our services to fail independently | Near: because we embrace email for collaboration | Near: servers automatically get caught up
Reasoning
Reliability on several levels: simple servers, monitoring that alerts early, and a design that works even during an outage.
Near: We achieve reliability on several levels
Outcome
The author reports that, of the two outages in 2020 so far, only one affected more than one subsystem.
Near: SourceHut has had two outages in 2020 | Near: only one of which affected more