Catalogue / fathom-2020-viral-customer-backlog

One viral Fathom customer backlogs the shared pageview queue, and a slow database disk and unindexed tables prolong it

Fathom Analytics, 2020. Pageview queue and MySQL database.

Organisation
Fathom Analytics
Evidence
Incident report
Source
What happened when a customer got over 10m pageviews in a few hours?

Fathom Analytics blog, published 2020-08-19

Records

Failure#f1

A viral customer's pageviews backlog the shared queue for all sites while the database runs out of disk IOPS

Failure#f2

Unindexed backlog tables and a text-typed site ID kept the database slow after the disk was fixed

Decision#d1

Cap concurrent queue workers on purpose to protect the database, accepting a bigger backlog

Decision#d2

Limit pageviews processed per site per minute so one viral site cannot block the rest, then drop it

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.