Decision#d1
Moved jobs from Resque on Redis to a database-backed queue, starting with the most critical ones
- Chosen
Built Solid Queue, a database-backed Active Job backend, and ran HEY's most critical jobs on it first, with more to follow until HEY runs only on it. Resque on Redis stayed in use meanwhile.
Near: we use in HEY to run about | Near: until we run HEY exclusively using Solid Queue | Near: using Solid Queue for our most critical jobs
- Rejected
GoodJob, the existing database-backed Active Job backend. It is built only for PostgreSQL, which 37signals does not use, and a backend that could aspire to become the new Rails default needed to support MySQL, PostgreSQL and SQLite at a minimum.
Near: GoodJob stood out, but unfortunately, it is built | Near: We don’t use PostgreSQL at 37signals
- Reasoning
The goal was that a Rails developer could get background jobs from nothing more than a database, without the seven gems and extra systems 37signals needed; experience from building Solid Cache suggested going back to a database queue.
Near: experience we gained from Solid Cache | Near: Our goal was to enable developers to install
- What worked before
37signals ran Resque and Redis for many years and the author says it served them well, though running it at scale brought unspecified difficulties, and gaps were filled with plugins and in-house libraries, seven gems in all for Basecamp and HEY.
Near: running Resque and Redis for many years | Near: still need seven different gems
- Outcome
After a few weeks running HEY's most critical jobs, the author reports, as the team's own assessment, that the main benefit was simplicity and ease of operation: with jobs held in a relational database and reached through Active Record, debugging job problems was significantly easier than with Resque.
Near: main benefit we've already reaped has been simplicity | Near: Having everything stored in a relational DB
- Reviewer's notes
The benefit is reported as simpler operation. The post gives no before-and-after figures for throughput, latency or incidents.
The reviewer's reading, not the source's claim
The difficulties with Resque and Redis are named but never described or measured, so they are kept in worked_before rather than treated as a forcing condition.
The reviewer's reading, not the source's claim
This is a partial migration reported after a few weeks, not a completed one.
The reviewer's reading, not the source's claim