The quarterly K8s migration is now a Tuesday night

The thesis

Production K8s migrations (the real ones, with stateful data and live customer traffic) are evening-tier work in 2026, not quarter-tier projects. Most engineering orgs haven't updated their mental model. They're still scheduling "infrastructure modernization initiatives" with kickoff decks, weekly syncs, and a target completion of "end of next quarter." Then they wonder why nothing actually moves.

Tonight we moved 38 pods, ~150 GB of state, an entire ES cluster (including its master quorum), and an ingress's worth of traffic across two AZs onto storage that didn't exist on those nodes 36 hours ago. With 13 minutes of outage. From a French ISP-grade home setup, with a CTO who said "go faster, I'm taking the risk" and a Claude Code window.

This is what tooling has done for us. People should know.

What we actually did

Evening of 2026-05-05 → morning of 2026-05-06:

• Stage 2, burn-in environment cutover to the new nodes. 9 pods. Validated the storage path and the compute uplift before touching prod.
• Stage 3, production application (the API, the queues, redis, the FTP intake). 23 pods. 4 phases (stateless → redis → import-ftp → re-enable scheduled imports). 3 minutes of customer outage during the redis cutover, the only window where we couldn't roll.
• Stage 4, Elasticsearch. 6 nodes restructured (legacy 3-data + 2-master + 1-transform on Longhorn → 2-data + 2-master + 1-voting_only + 1-transform on LINSTOR with daily S3 snapshots). 10 minutes of customer outage from a stale env-var oversight.

Replicas, replicated storage, S3 backups, anti-affinity, voting-only tiebreaker on a tiny VM in another AZ. The whole shape of "how a real prod cluster looks." Done in an evening. Walked back into the room the next morning, dashboards green, channel quiet.

The compute claim

Mid-migration, on a real workload (a 63 MB XML import that fans out into 24,000 child jobs across 6 Horizon workers), the new nodes processed it 5.7× faster per job than the equivalent on legacy. 58 jobs/sec vs 10 jobs/sec, same code, same Redis, same Postgres. Just newer silicon and proper local-NVMe-backed DRBD instead of whatever the legacy storage was doing.

That number alone justifies the migration. Doing it on a Tuesday night justifies stopping the meeting.

What made it possible

It wasn't the execution. The execution was just typing the script. That's the whole point.

It was:

1. Two months of paranoid planning. Every stage, patch file, and cutover sequence on paper. The outage windows pre-budgeted. The rollback steps documented. By the time we ran a command tonight, we'd already played the move in our heads four or five times. Planning is the bottleneck, not execution. Always was. Tooling just made execution cheap enough that the planning effort actually pays off.
2. The runbook structure. Every destructive step lived in a script under infra/tmp/ (one-shot) or infra/clusters/prod/ (canonical / reproducible). Idempotent. Re-runnable. When the first containerd-move script aborted halfway because k8s-agent.service didn't exist (it was kubelet.service), I just edited the failed step and re-ran. Recovery time: 30 seconds. Not "schedule a follow-up call."
3. The cutover tricks. The cleanest one tonight was for the redis migration. Standard playbook says: BGSAVE, dump.rdb out, delete the old PVC, recreate on new storage, copy dump.rdb in, start redis. The race is the gap between kubectl apply (which scales redis up) and your manual scale --replicas=0 (which stops it before it overwrites your dump.rdb with empty data). My CTO suggested overriding the redis container's command to sleep 2d for the duration of the data restore. Pod runs, mounts PVC, doesn't run redis, can't write to disk. We kubectl cp the data in, then re-apply with the real command. Race eliminated. The kind of move you can only see if you understand both the system AND the orchestrator's idempotency model.
4. The CTO who took the risk. Originally this was supposed to be a 48-72-hour burn-in followed by a careful prod cutover spread over a week. He looked at the data after 4 hours and said "go faster, I'm taking responsibility." That is the rarest thing in engineering management: someone who reads the signals and writes "go" in the chat. Without that, this is still a quarterly project. With that, it's tonight.

What surprised us

The surprises tonight, none of which broke us:

• DEV1-M comes with a 9 GB root disk, not the 50 GB I'd assumed during planning. Containerd images alone trip kubelet's DiskPressure threshold under default eviction policy. The voting-only ES master couldn't schedule on the tiebreaker node. Fix: attach a 50 GB SBS volume, restructure as /mnt/sbs with bind mounts to /var/lib/containerd and /var/log. Discovered, fixed, documented in a runbook, committed. 90 minutes door-to-door. Three years ago this would have been a Slack thread that died after two days because nobody could agree on the right approach.
• Bitnami pulled bitnami/redis:8.0 from Docker Hub silently sometime in August 2025, as part of their licensing changes. Our prod redis was running on a cached image. The moment it ever restarted, it was dead. We caught this in Stage 2 burn-in (where the image had to pull fresh on a new node) and switched to bitnamilegacy/redis:8.0. It's dumb luck that this didn't blow up production three months ago.
• ECK auto-deletes per-nodeSet headless services when you remove a nodeSet. Our prod API was hardcoded to search-es-es-data.elasticsearch.svc, the headless service of the legacy data nodeSet. When Stage 4 removed that nodeSet, ECK deleted the service. The public site's /acheter/* routes started returning 500. 10 minutes of customer outage. Switched to the cluster-wide LB service search-es-es-http.elasticsearch.svc, rolled the API pods, recovered. Never reference per-nodeSet services from app config. Now permanent in my head, and in the recap doc.

These are the kind of things you only learn by doing it. The kind of things people who've never done a real migration don't know they don't know. The kind of things that would eat a quarter, if you let them.

The ceiling moved

Most engineering orgs are still in the "infrastructure modernization is a quarterly initiative" mental model. They have a Confluence page. They have a Q3 OKR. They have a Slack channel called #infra-modernization-2026 that gets one message a week.

The actual ceiling moved. Container Storage Interface drivers are stable. Operators (ECK, Piraeus, etc.) handle the orchestration that used to need humans (shard drains, voting-config exclusions, PVC binding races). S3 snapshots are free real estate. Kustomize gives you patches that compose. kubectl cp works through a tar pipe and is fast enough for any data volume that fits in your dump.rdb.

If you've been running production K8s for three years, you have all the components. You just need the planning discipline and the runbook structure, plus a CTO who reads the signals. Then it's an evening.

Mine took 13 minutes of outage. Yours might take 30. It's still a Tuesday.