Zero-Downtime Migrations: The First 90 Days

Every team claims to deploy continuously right up to the moment the first schema change lands. Then a Tuesday evening release becomes a forty-minute maintenance window, the ops lead emails customers about degraded service, and the 2 a.m. deploy habit quietly returns. It hurts B2B teams most: a 60-person company whose finance lead reconciles UPI settlements at 11 p.m., or whose filing-week traffic spikes around the 20th, cannot absorb a maintenance window on a random weekday.

Why is the first month slower, not faster?

The first month is slower because every structural change now ships as two or three releases instead of one, and that overhead is real before it becomes routine. In our experience a team moving off maintenance-window deploys adds roughly 20 to 40 percent to release overhead for the first three to five schema changes, then settles down. The work is mostly coordination, not database trivia: the old application code has to tolerate the new schema, and the new code has to tolerate the old schema, in the same rolling deploy. What is usually missing is a convention for who writes the backfill, who verifies row counts, and who approves the drop step two weeks later. Write that down in week one.

What does expand-contract look like on a real table?

Expand-contract means adding the new shape alongside the old one, migrating data in batches, and removing the old shape only once nothing reads it. Take a MySQL 8 orders table with 4.2 million rows, where amount must become amount_inr and pick up two GST fields at the same time. One ALTER TABLE that renames and adds columns holds a metadata lock long enough to stall writes, so you split it.

• Release 1: add amount_inr as nullable, dual-write both columns from application code, and backfill in batches of 5,000 rows with a short sleep between batches.
• Verify: amount IS NOT NULL AND amount_inr IS NULL must return zero rows before you go further.
• Release 2: switch reads to the new column behind a feature flag, keep writing both.
• Release 3: stop writing to amount, wait a fortnight, then drop it.

The batched backfill on 4.2 million rows typically finishes in 25 to 45 minutes with no write stall, versus a lock duration you cannot reliably predict on a busy table. For index rebuilds, use gh-ost or pt-online-schema-change on MySQL and CREATE INDEX CONCURRENTLY with pg_repack on Postgres. Our architecture notes cover the conventions we apply to schema work like this.

Which changes still need a maintenance window?

A small, honest set of changes still needs a window, and pretending otherwise is how teams end up restoring backups at midnight. Any full table rebuild — converting utf8 to utf8mb4, changing a primary key from INT to BIGINT on a heavily written table, or restructuring partitions — should be scheduled, announced, and rehearsed against a copy. Postgres versions before 11 also rewrite the table when you add a column with a default, which is easy to miss in code review. The practical rule: if you cannot state in one sentence how the previous release behaves against the new schema, take the window. A planned two-hour pause costs far less than a rolled-back deploy during a customer's filing deadline.

What should you measure by day ninety?

Four numbers tell you whether zero-downtime migration has become a habit rather than a heroic one-off. Track them monthly, not per deploy:

• Longest DDL lock on your largest table, in seconds — it should trend down and stay under your statement timeout.
• Migration duration as a share of total deploy duration; above roughly 30 percent means you are bundling too much into one change.
• Backfill throughput and read-replica lag during a backfill, so you know your actual headroom.
• Deploys that required a schema rollback, which should be zero after month two.

Keep an owner and a one-line rollback note on every migration file. Teams often add read replicas for reporting and then find a backfill saturates them; sizing that properly is a cloud infrastructure decision as much as a database one.

Where to start

Pick the single largest table you write to and write down exactly how you would change one column on it without locking — including who approves the drop step. Do it once, imperfectly, on staging. If you would rather have someone sit with your team through the first three or four migrations and then leave you to it, that is the kind of work Maysan Labs takes on. Take one thing from this even if you never call us: the two-release habit.