Rate limiting is layered to stop ping floods overloading the database: HAProxy on the load balancers applies a lax first limit, and NGINX on the application servers applies a stricter limit and geo-blocking before requests reach the application.
Section: Tarpit
Figures, as the source states them
Figure
As stated
Note
Where
Ping requests for the single offending check UUID
300 requests per second
Sustained; up from 10 requests per second; one client only
Near: increase from 10 requests/second to sustained 300 requests/second
Tarpit rate limiting held sockets open until the load balancers ran out of connection slots and rejected real traffic
healthchecks-2025-tarpit-outage#f1at corpus version 74847648e25a
Trigger
A misconfigured client's ping requests for one specific UUID rose from 10 requests per second to a sustained 300 requests per second, starting at 23:24 UTC on November 1.
Section: Timeline | Near: flood of ping requests from a misconfigured client
How it failed
HAProxy answered rate-limited clients with the tarpit method, which holds a network socket during its wait before returning 429. The wait was 1 second, but it did not slow this client, and its request rate was high enough that the HAProxy servers began hitting their connection slot limits. At 23:39 UTC one load balancer started rejecting connections and was automatically removed from DNS rotation, which raised the load on the remaining balancers and started a chain reaction.
Section: Tarpit | Section: Timeline
Why defences did not hold
The tarpit was meant to slow the client down, and its 1-second wait was chosen to lower the chance of running short of connection slots. The wait did not slow this client.
Section: Tarpit
Impact
From 23:30 UTC on November 1 to 1:10 UTC on November 2, both the website and the ping endpoints had intermittent connectivity problems, with some requests taking several seconds or timing out.
Near: had intermittent connectivity issues in the form
Fix
The author identified the offending client (one IPv4 address and a /64 IPv6 subnet) by 01:06 UTC and blocked it with an immediate reject, and things went back to normal. At 08:40 UTC the rate limiting was changed to return HTTP 429 immediately, not tarpit. By November 5 came a 5-second HTTP request timeout, a higher maxconn, rate limits on hchk.io as well as hc-ping.com, a Netdata alert on frontend sessions and a new log format.
Section: Timeline | Section: The Fix | Section: Update on November 5, 2025
The stated aim was to avoid similar issues in the future. The author describes Deny as the most polite of the four methods but also the most expensive during an attack, since the server does a full TLS handshake, parses the HTTP request and sends a response for every request; a tarpit, by contrast, can slow clients that send requests one after another but can overwhelm the load balancer when clients make huge numbers of requests in parallel, because each wait ties up a socket.
Section: Timeline | Section: Tarpit
What worked before
HAProxy used a tarpit with a 1-second wait to rate-limit particularly spammy clients, the wait being set short to reduce the risk of running out of connection slots. It did not slow the client in the November 1 flood.