Catalogue / healthchecks-2025-tarpit-outage

Healthchecks.io's HAProxy tarpit rate limit used up load balancer connection slots during a ping flood from one client

Healthchecks.io, 2025. Load balancer rate limiting for the ping endpoints.

Organisation
Healthchecks.io
Evidence
Incident report
System
Load balancer rate limiting for the ping endpoints
As of
2025-11
Source
Post-mortem: Outage on November 1

Healthchecks.io blog, published 2025-11-03

Architecture

Rate limiting is layered to stop ping floods overloading the database: HAProxy on the load balancers applies a lax first limit, and NGINX on the application servers applies a stricter limit and geo-blocking before requests reach the application.

Section: Tarpit

Figures, as the source states them

FigureAs statedNoteWhere
Ping requests for the single offending check UUID300 requests per secondSustained; up from 10 requests per second; one client onlyNear: increase from 10 requests/second to sustained 300 requests/second

Records

Failure#f1

Tarpit rate limiting held sockets open until the load balancers ran out of connection slots and rejected real traffic

healthchecks-2025-tarpit-outage#f1 at corpus version 74847648e25a
Trigger

A misconfigured client's ping requests for one specific UUID rose from 10 requests per second to a sustained 300 requests per second, starting at 23:24 UTC on November 1.

Section: Timeline | Near: flood of ping requests from a misconfigured client

How it failed

HAProxy answered rate-limited clients with the tarpit method, which holds a network socket during its wait before returning 429. The wait was 1 second, but it did not slow this client, and its request rate was high enough that the HAProxy servers began hitting their connection slot limits. At 23:39 UTC one load balancer started rejecting connections and was automatically removed from DNS rotation, which raised the load on the remaining balancers and started a chain reaction.

Section: Tarpit | Section: Timeline

Why defences did not hold

The tarpit was meant to slow the client down, and its 1-second wait was chosen to lower the chance of running short of connection slots. The wait did not slow this client.

Section: Tarpit

Impact

From 23:30 UTC on November 1 to 1:10 UTC on November 2, both the website and the ping endpoints had intermittent connectivity problems, with some requests taking several seconds or timing out.

Near: had intermittent connectivity issues in the form

Fix

The author identified the offending client (one IPv4 address and a /64 IPv6 subnet) by 01:06 UTC and blocked it with an immediate reject, and things went back to normal. At 08:40 UTC the rate limiting was changed to return HTTP 429 immediately, not tarpit. By November 5 came a 5-second HTTP request timeout, a higher maxconn, rate limits on hchk.io as well as hc-ping.com, a Netdata alert on frontend sessions and a new log format.

Section: Timeline | Section: The Fix | Section: Update on November 5, 2025

Decision#d1

Answer rate-limited requests with an immediate HTTP 429 instead of tarpitting them

healthchecks-2025-tarpit-outage#d1 at corpus version 74847648e25a
Chosen

After the outage, rate-limited requests are denied immediately with HTTP 429 instead of being tarpitted.

Section: The Fix

What forced it

The November 1 outage, where the load balancers hit their connection count limits under the tarpit and rejected legitimate requests.

Near: rejecting legitimate requests | Section: Tarpit

Reasoning

The stated aim was to avoid similar issues in the future. The author describes Deny as the most polite of the four methods but also the most expensive during an attack, since the server does a full TLS handshake, parses the HTTP request and sends a response for every request; a tarpit, by contrast, can slow clients that send requests one after another but can overwhelm the load balancer when clients make huge numbers of requests in parallel, because each wait ties up a socket.

Section: Timeline | Section: Tarpit

What worked before

HAProxy used a tarpit with a 1-second wait to rate-limit particularly spammy clients, the wait being set short to reduce the risk of running out of connection slots. It did not slow the client in the November 1 flood.

Section: Tarpit

Review

Extracted by
claude-sonnet-5-5
Human review
Not started. Status recorded on 2026-11-16.