Catalogue / 37signals-2023-solid-queue

HEY moves part of its job processing from Resque on Redis to a database-backed queue

37signals, 2023. HEY background jobs (Active Job, Resque and Solid Queue).

Organisation
37signals
Evidence
Reported from production
System
HEY background jobs (Active Job, Resque and Solid Queue)
As of
2023-12
Source
Introducing Solid Queue

37signals Dev, published 2023-12-18

Architecture

HEY runs its Active Job work on two backends at once: Resque on Redis, and the new database-backed Solid Queue, which at publication carried about a third of a stated total of roughly 18 million jobs a day, with more being moved.

Near: about 1/3 of our roughly 18 million jobs

Figures, as the source states them

FigureAs statedNoteWhere
Total jobs per day that the Solid Queue share is a third of18,000,000 jobs per dayRoughly; the post does not say whether this is HEY alone or all of 37signalsNear: of our roughly 18 million jobs per day
Jobs executed daily on Solid Queue5,600,000 jobs per dayAround; at publicationNear: around 5.6 million jobs executed daily
Polling queries on the queue database1,300 queries per secondAround; current rate at publication, not stated as a peakNear: we perform around 1,300 polling queries per
Average polling query time110 microsecondsAverageNear: average query time of 110
Rows examined per polling query0.02 rowsAverageNear: examination of 0.02 rows per query

Records

Decision#d1

Moved jobs from Resque on Redis to a database-backed queue, starting with the most critical ones

37signals-2023-solid-queue#d1 at corpus version 74847648e25a
Chosen

Built Solid Queue, a database-backed Active Job backend, and ran HEY's most critical jobs on it first, with more to follow until HEY runs only on it. Resque on Redis stayed in use meanwhile.

Near: we use in HEY to run about | Near: until we run HEY exclusively using Solid Queue | Near: using Solid Queue for our most critical jobs

Rejected

GoodJob, the existing database-backed Active Job backend. It is built only for PostgreSQL, which 37signals does not use, and a backend that could aspire to become the new Rails default needed to support MySQL, PostgreSQL and SQLite at a minimum.

Near: GoodJob stood out, but unfortunately, it is built | Near: We don’t use PostgreSQL at 37signals

Reasoning

The goal was that a Rails developer could get background jobs from nothing more than a database, without the seven gems and extra systems 37signals needed; experience from building Solid Cache suggested going back to a database queue.

Near: experience we gained from Solid Cache | Near: Our goal was to enable developers to install

What worked before

37signals ran Resque and Redis for many years and the author says it served them well, though running it at scale brought unspecified difficulties, and gaps were filled with plugins and in-house libraries, seven gems in all for Basecamp and HEY.

Near: running Resque and Redis for many years | Near: still need seven different gems

Outcome

After a few weeks running HEY's most critical jobs, the author reports, as the team's own assessment, that the main benefit was simplicity and ease of operation: with jobs held in a relational database and reached through Active Record, debugging job problems was significantly easier than with Resque.

Near: main benefit we've already reaped has been simplicity | Near: Having everything stored in a relational DB

Reviewer's notes

The benefit is reported as simpler operation. The post gives no before-and-after figures for throughput, latency or incidents.

The reviewer's reading, not the source's claim

The difficulties with Resque and Redis are named but never described or measured, so they are kept in worked_before rather than treated as a forcing condition.

The reviewer's reading, not the source's claim

This is a partial migration reported after a few weeks, not a completed one.

The reviewer's reading, not the source's claim

Decision#d2

Database queue design that keeps the polled table small and claims jobs with SKIP LOCKED

37signals-2023-solid-queue#d2 at corpus version 74847648e25a
Chosen

Workers claim jobs with SELECT ... FOR UPDATE SKIP LOCKED so they do not block each other. Jobs that are running, scheduled for later, failed or blocked by concurrency limits are kept apart from jobs ready to run, which keeps the polled table small, and a covering index serves both selection and ordering.

Near: fetch and lock jobs without locking other workers | Near: keep the table that workers poll as small | Near: makes use of a covering index

Reasoning

The author notes that speed is the usual reason to run jobs on Redis, and holds that a performant database queue is possible provided the system is designed appropriately.

Near: Performance is typically the primary reason for relying | Near: possible to develop a performant queuing system

Outcome

At around 5.6 million jobs executed daily, polling ran at around 1,300 queries per second, averaging 110 microseconds and examining 0.02 rows per query.

Near: around 5.6 million jobs executed daily, we perform

Reviewer's notes

The figures are an average query time and row count at one moment. The post gives no peak load, job latency or queue-lag measurement and no database size.

The reviewer's reading, not the source's claim

Review

Extracted by
claude-sonnet-5-5
Human review
Not started. Status recorded on 2026-11-16.