Catalogue / pusher-2020-webhooks-kinesis

Pusher moves Channels webhooks off self-hosted beanstalkd and a Ruby packager to Kinesis, SQS and a Go consumer, with a one-step cutover

Pusher. Channels webhooks.

Organisation
Pusher
Evidence
Reported from production
Source
Migrating Channels webhooks from Beanstalkd to Kinesis/SQS

Pusher blog, published 2020-08-06

Records

Decision#d1

Replace self-hosted beanstalkd and a Ruby packager with Kinesis, SQS and a Go consumer before the system gave severe problems

Decision#d2

Run the new pipeline alongside the old in production without sending webhooks, then cut over in one configuration change

Decision#d3

Divide the event stream into locked logical shards so that one packager sees all of a channel's events and can debounce them

Sign in to read what was chosen, what was rejected and what it cost, each with its place in the source. An account is free.