Moving Background Jobs to a Queue Without Downtime

Most small teams do not have a background job problem until they do. A cron entry that emails nightly invoices is fine at 200 customers; at 4,000 it runs long enough to overlap the next run, and nobody notices until a client forwards the same GST invoice twice. The fix — a queue with a dedicated worker process — is not hard to build. The hard part is getting there without a bad Tuesday afternoon. Here is a migration path that keeps the old system running until the new one has earned trust.

Why does a queue migration break production?

Most queue migrations break production because teams change the scheduler and the executor in the same deploy. You are really changing two things at once: what decides a job is due, and what actually runs it. If both flip and invoices go out twice, you cannot tell which half caused it. The second failure mode is the enqueue point — if you push a job to Redis inside an HTTP request and Redis is down, the customer's action now fails, which is worse than a slow email. The third is losing the old path: once the crontab is deleted and the script is gone, rollback means reconstructing something nobody fully remembers.

Start with a jobs table in your existing database

Durability matters more than throughput at small scale, so write jobs into the database you already run before you touch a broker. A jobs table with columns for id, type, payload, run_at, attempts, status and idempotency_key gives you a queue that survives restarts, a replayable audit trail, and zero new infrastructure. Claim work with SELECT ... FOR UPDATE SKIP LOCKED — available in Postgres 9.5 and later — and three workers can safely share the table. Run the worker as a separate process on the same box to begin with; a second service on your existing servers is enough until volume justifies splitting it out. Teams pushing past a few hundred jobs a second should move to Redis, SQS or BullMQ, but most SMB workloads never get close.

Make every job safe to run twice

Every job you migrate must be idempotent before it goes near a worker, because retries are not an edge case. Add a unique constraint or a sent_at marker so a second attempt cannot double-send an invoice, double-charge a card or double-credit a wallet. For email, store the provider's message ID; for payments, pass the gateway's idempotency key; for exports, write to a deterministic filename and overwrite. During the overlap period the old cron and the new worker may both execute the same job — that is the entire point of the overlap, and it is only safe if the second execution is a no-op. If a job genuinely cannot be made idempotent, do not migrate it in the first phase.

Run the new worker in shadow mode for a week

Shadow mode means the worker picks up real jobs and does everything except the irreversible part. Leave the cron job alive and sending; let the worker log what it would have done, then compare the two. The usual pattern:

• Emails: render the full HTML, write it to a log or a local Mailhog inbox, send nothing
• Webhooks: POST to an internal echo endpoint instead of the partner URL
• Payment reconciliation: compute the mismatch set and alert only
• Exports: write to a staging bucket, not the client's SFTP folder

After seven days, diff the counts. If the worker processed 4,211 emails and cron sent 4,190, you have 21 discrepancies to explain — a conversation you want to have now rather than after cutover.

Cut over one job type at a time

Cut over in slices, smallest blast radius first, behind a flag that reverts to the old path in a single deploy. Start with something recoverable — weekly digests, internal reports — and leave anything touching money until last. A realistic three-week sequence looks like this: week one, jobs table plus shadow worker for digests and reports; week two, flip digests to the worker and comment out that crontab line, keeping the flag for a fortnight; week three, migrate the nightly invoice run. In one such run, 4,200 PDFs took 38 minutes on cron and collided with the 04:00 backup; on the queue it ran across ~40 parallel workers and finished in 6 minutes. Write down which flag controls which path — a short architecture note mapping each job type to its old and new route pays for itself the first time you roll back at 2am.

What to monitor in the first two weeks

Queue depth and the age of the oldest pending job are the only two metrics you truly need at the start; everything else is detail. Alert when the oldest unprocessed job is more than five minutes old during business hours, and route exhausted retries into a dead-letter table that someone actually opens. Track a duplicate counter — how often the idempotency guard fires — because a sudden spike usually means the old path is still running alongside the new one. Watch worker memory: an export job that loads 200MB of rows into memory should stream instead. Keep the commented-out crontab file in version control for one full billing cycle before deleting it.

Where to start

Pick one low-risk job — a digest, a report, a webhook retry — and take it through the whole path: jobs table, idempotency guard, shadow run, flag flip. One job done end to end teaches your team more than a design document, and it hands you the pattern the other eleven will follow. If you want a second pair of eyes on the cutover plan before the first flag goes live, that is the sort of review Maysan Labs does — usually two sessions, one before week two and one after the first week of monitoring.