Catalogue / nylas-2018-webhooks

Nylas rebuilds its webhook sender after per-customer database readers stopped scaling

Nylas, 2018. Webhook notifications for account changes.

Organisation
Nylas
Evidence
Reported from production
Source
Speeding up our Webhooks System 60x

Nylas (posted on DEV Community), published 2018-06-20

Records

Decision#d1

One reader per shard replaces one reader thread per customer webhook after delays reached 10 minutes

Decision#d2

Kinesis chosen over the MySQL binlog and Kafka as the ordered source of changes, mainly for operational simplicity

Decision#d3

New sender ran two months in a simulated mode that sent nothing, compared against the legacy sender, before cutover

Failure#f1

Resuming from the last sent MySQL auto-increment id made the new webhook sender skip transactions in testing

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.