One multi-tenant ASP.NET application on IIS serves every Q&A site. SQL Server is the single source of truth, with Redis and Elasticsearch fed from it, and all servers and services are duplicated in a second data center.
Section: Web Tier (IIS 8.5, ASP.Net MVC 5.2.3, and .Net 4.6.1) | Near: of truth. All data in Elastic and Redis | Near: doubly redundant via another data center
Components
Component
Role
Where
HAProxy load balancers
Terminate TLS and route to backends mostly by host header; also apply rate limiting and record per-request performance metrics from headers sent by the web tier. They hold plenty of memory to cache TLS sessions.
Section: Load Balancers (HAProxy)
Web tier
Nine primary IIS servers run Stack Overflow, Careers and the Stack Exchange sites, with two more for dev and the meta sites. The Q&A application is multi-tenant, so one application pool on one server could run the whole Q&A network.
Section: Web Tier (IIS 8.5, ASP.Net MVC 5.2.3, and .Net 4.6.1)
Service tier
Runs internal services behind the web tier, mainly the tag engine (Stack Server) and the Providence API, plus Elasticsearch indexing jobs. Three servers are enough because loading changed posts every 2 minutes should not be repeated by all nine web servers.
Section: Service Tier (IIS, ASP.Net MVC 5.2.3, .Net 4.6.1, and HTTP.SYS)
Redis
Two main servers (master and replica) back an L2 cache and a publish and subscribe channel used to clear L1 caches and to feed websockets. Two further servers hold a machine learning instance for recommendations.
Section: Cache & Pub/Sub (Redis)
Websockets
A thin raw-socket layer on the web tier pushes notifications, vote counts and new answers or comments to browsers.
Near: we use websockets to push real-time updates | Near: raw sockets running on the web tier
Elasticsearch
Serves site search, related questions and suggestions when asking. Each data center has a 3-node cluster with one index per site, kept current by the service tier.
Section: Search (Elasticsearch)
SQL Server
The single source of truth for all data in Elasticsearch and Redis. Two clusters, each with one master and replicas in New York and Colorado, hold the Stack Overflow, Sites, PRIZM and Mobile databases in one and everything else, including every other Q&A site, in the other.
Section: Databases (SQL Server)
Second data center
Colorado holds a redundant copy of all servers and services for disaster recovery, linked to New York by a 10Gbps MPLS used for data replication.
Section: Ground Rules | Near: 10Gbps MPLS between our 2 data centers | Near: in Colorado (our DR data center)
Figures, as the source states them
Figure
As stated
Note
Where
HTTP requests to the load balancer in one day
209,420,973 requests per day
A single day, February 9th 2016; requests to the load balancers
Near: 209,420,973 (+61,336,090) HTTP requests to our load balancer
Page loads in one day
66,294,789 page loads per day
A single day, February 9th 2016
Near: 66,294,789 (+30,199,477) of those were page loads
SQL queries from HTTP requests in one day
504,816,843 queries per day
A single day, February 9th 2016; from HTTP requests alone
Serve every Q&A site from one multi-tenant application and keep web servers overprovisioned
stackoverflow-2016-architecture#d1at corpus version 74847648e25a
Chosen
A single application serves all the Q&A sites, while Careers, API v2 and the Mobile API are separate applications. Nine primary web servers are run although fewer are needed.
Section: Web Tier (IIS 8.5, ASP.Net MVC 5.2.3, and .Net 4.6.1) | Near: we're down to needing only 1 web server
Reasoning
The author says the large overprovisioning in the web tier is for rolling builds, headroom and redundancy, and promises a fuller explanation in a later post.
Section: Web Tier (IIS 8.5, ASP.Net MVC 5.2.3, and .Net 4.6.1)
Outcome
After a hardware upgrade in early 2015 and a lot of tuning, only 1 web server is needed to run Stack Overflow. The team has run on one server by accident a few times and it held. The author says it works but does not call it a good idea.
Near: both a hardware upgrade in early 2015 | Near: we're down to needing only 1 web server | Near: unintentionally tested this, successfully
Reviewer's notes
The post does not say why Q&A is multi-tenant in one application, only that it is. The figures are for 2016, not the earlier design.
Use Elasticsearch for search instead of SQL Server full-text search, because SQL CPUs are expensive
stackoverflow-2016-architecture#d2at corpus version 74847648e25a
Chosen
Search runs on Elasticsearch: a 3-node cluster per data center, one index per site, with the service tier indexing items changed since a stored last-position marker, using SQL Server's ROWVERSION as the sequence.
Section: Search (Elasticsearch)
Rejected
SQL full-text search. CPU on the SQL Server machines costs much more than Elasticsearch, which also offers more features.
Section: Search (Elasticsearch)
Solr. It could not search across many indexes at once, which the network needs, at the time of the decision.
Section: Search (Elasticsearch)
Reasoning
Scalability and a better allocation of money.
Section: Search (Elasticsearch)
Cost
The cluster stays on Elasticsearch 1.4 because moving to 2.x means a major change to types and reindexing everything, and the author lacks time for the migration plan.
Cache in two layers, a web-server HTTP cache and Redis, and clear the first layer through Redis pub/sub
stackoverflow-2016-architecture#d3at corpus version 74847648e25a
Chosen
L1 is the HTTP cache on each web server and L2 is Redis. On a miss in both, a server loads from the source and fills both. When one web server removes a value, a Redis pub/sub message clears the other servers' L1 copies.
Section: Cache & Pub/Sub (Redis)
Reasoning
A value one server missed may be found in Redis by the next server, saving a database query or API call.
Section: Cache & Pub/Sub (Redis)
Outcome
Across everything the team uses Redis for, not only this cache, it handles about 160 billion operations a month with every instance below 2% CPU, usually much lower.
Push updates to browsers over websockets, which are more efficient than polling, at the cost of load balancer port exhaustion
stackoverflow-2016-architecture#d4at corpus version 74847648e25a
Chosen
Real-time updates such as notifications, vote counts and new answers reach browsers over websockets served by raw sockets on the web tier, using the team's NetGain library.
Near: we use websockets to push real-time updates | Near: raw sockets running on the web tier
Reasoning
At the team's scale websockets are far more efficient than polling: more data can be pushed with fewer resources, and users see it sooner.
Near: Why websockets? They're tremendously more efficient than polling
Cost
Ephemeral port and file handle exhaustion on the load balancer are among the issues websockets bring.
Near: ephemeral port and file handle exhaustion
Outcome
About 500,000 concurrent websocket connections are open at peak, and some browsers have kept theirs open for over 18 months.
Near: about 500,000 concurrent websocket connections open