Executive Playbooks
The Cutover Playbook: Embracing the Delta
"Don't hope D-Day will be Blue Skies... Be paranoid in the lead-up in a good way. That's how you earn a good night's sleep and wake up confident and full of energy on D-Day."
Production Reality: Testing will never fully replicate production. That is not a failure of capability — it is a structural reality. No matter how mature the test environment or how seasoned the QA team, certain failure modes only become visible for the first time after go-live. The real exposure often sits below the application layer — system-level settings, runtime parameters, and environmental conditions that no test can faithfully replicate.
Effective risk management begins by acknowledging the delta rather than attempting to eliminate it. The objective is to identify where failure is plausible, and design containment before impact becomes material.
The Cutover Failure Equation
Failure Impact = Detection Delay × Decision Delay × Execution Delay
Minimising any single delay factor reduces overall failure impact exponentially
Delay
Delay
Delay
IMPACT
- Failure mode understood?
- Are we making it worse?
- What are the downstream dependencies?
- Is there a time-sensitive business deadline?
- What signals confirm the scope of impact?
- Do we have enough to stop, isolate, or contain?
- Is rollback still possible?
What do we do right now to stop the bleeding? Options depend on context — there is rarely one right answer.
- Throttle traffic
- Fail over regions
- Switch to read-only
- Isolate impacted customers
- Stop background jobs
- Disable integrations
- Pause queues & processors
- Prevent bad data spread
- Reduce functionality temporarily
Containment has bought you time. Now choose the safest path forward with the information available.
- Rollback or fix forward?
- Partial service or full outage?
- Risk profile of each path?
- Can we avoid making it worse?
- Critical business deadlines?
- Impacted dependencies?
- Interim fix to restore service?
- What assurance can we give the business?
- Was data lost or duplicated?
- Was data corrupted?
- Was processing out of order?
- Were downstream systems affected?
- Are audit logs intact?
- Replay events / messages
- Restore backups
- Reconcile databases
- Re-run failed jobs
- Recover missing transactions
- Reconciliation reports
- Checksum / count validation
- Customer verification
- Financial reconciliation
- Business & data owner sign-off
The Two Critical Risk Parameters
Impact Latency Window (ILW)
The time between when a defect occurs and when a customer or regulator becomes aware of it. Maximising this window gives the team room to detect and contain before exposure becomes material.
Impact Amplification Window (IAW)
The point at which an isolated defect begins to compound into widespread operational, financial, or reputational damage. Containment strategies must activate before this threshold is crossed.
The Five Principles of Cutover
1. Focus on the Grey Areas
Start with what cannot be evaluated faithfully in QA. Where production conditions diverge, documentation becomes less dependable and experience becomes critical. These are the zones where surprises emerge. Test coverage metrics alone are not a proxy for safety.2. Prioritise Risk with Objective Scoring
Intuition has limits, especially under deadline pressure. A structured framework such as Failure Mode and Effects Analysis (FMEA) imposes discipline: assess severity, likelihood, and detectability, then rank exposures accordingly. Prioritisation is a strategy.3. Reduce Customer Exposure
Containment begins with volume control. The critical question is simple: must every transaction file be processed immediately, or can exposure be deliberately constrained? Reducing exposure converts potential crises into manageable operational issues.4. Maximise the Impact Latency Window
Timing decisions are often treated as logistical details. They are strategic controls. If a cutover occurs just after account statements are issued, teams may have weeks to detect and correct issues before customers notice. Time is the most underappreciated risk control in large transformations.5. Align the Right Team for Cutover Night
The team that builds the system is not the team that safely runs the cutover. Cutover night requires people who have lived through production crises — who carry historical memory, recognise failure patterns, know containment tactics, and can make calculated decisions when only 80% of the information is available.Part 2 — The War Room Reality
When Systems Fail at 2AM
At 2AM, when systems fail and 19 million customers are impacted, flawless designs matter less than having the right people in the room.
Transformation programs are designed in boardrooms. Cutover nights are survived in war rooms.
The skills required to design and test a complex transformation are fundamentally different from the skills required to run the cutover when things go wrong. At that moment, you don't need perfect designers or functional testers. You need people who have lived through a production crisis — accountable for live systems, able to recognise failure patterns, know containment tactics, and make calculated decisions with only 80% of the information available. BAU teams bring something project teams rarely have: historical memory, production instincts, and reusable knowledge — scripts, workarounds, and operational muscle memory that never appear in project documentation.
A Habit From Level 1 Support — Still Applied Today
One to two weeks before any major cutover, I sit with the support team and ask: what can go wrong? Do you have scripts ready for data integrity checks, reconciliation, impact analysis, and recovery validation?
It sounds simple. But when a script runs against a production dataset at volume, it takes 20–30 minutes to return results. That window is breathing space. You are not scrambling under pressure — you are already calm, already analysing. By the time results come back, you have done half your thinking. The last thing you want during a production crisis is to start writing scripts after things have already failed.
The Cutover Response Model
1. Rapid Detection
Know within minutes, not hours, that something has gone wrong and what it's touching.
2. Empowered Decision-Making
The room has the authority to act — no waiting on approvals that can't be reached at 2AM.
3. Immediate Production Access
The people diagnosing the issue can also act on it, without a second team as a bottleneck.
4. Operational Workarounds
Pre-built scripts and containment options ready before the crisis, not written during it.
In the best case: 60 minutes to understand the issue, 20–30 to decide on containment, 60 to determine rollback or fix-forward. Sometimes the right call is an interim fix — get the lights back on, recover the critical data, then immediately work a clear plan for the remaining 20%.
Someone Whose Sole Job Is Managing the Room
Not the technical lead. Not the program manager who has been running on empty for 18 hours. Someone trusted and calm — focused on containing the noise and giving stakeholders the right update at the right time. Too little communication and the business panics. Too much and it becomes unproductive. That balance is a skill in itself.
A 12-hour cutover can easily stretch to 24 or 36 hours when things go wrong. The final lap of the relay doesn't care how fast you ran the first three legs — only how you finish matters.
And here is what most people forget: if you have the right team and navigate a cutover problem well, it stops being a negative experience. The team learns together, confidence builds, and that knowledge becomes part of the organisation's institutional memory. What you learn from a live production issue sits in your head in a way no training course ever replicates.
The best cutover teams are not the ones that never had problems. They are the ones who faced problems — and came out stronger.