Field Runbook
Multi-Cloud Disaster Recovery
DOC-DR-2026 · V2
Operational Runbook · Updated July 2026

Multi-Cloud
Disaster Recovery

// "A backup with no tested restore path is just stored hope."

Operational standards for failover, backup restoration, recovery coordination, and validation across AWS, Azure, and GCP for systems that must recover predictably under pressure — rebuilt for a year in which regulators, boards, and Forrester itself all expect multi-day hyperscaler outages to happen again.

Recovery Standards Runbook Discipline RTO / RPO Focus Backup Posture DORA / NIS2 Recovery-as-Code
✦ What's New in v2.0
  • New §03 — The 2025–26 Wake-Up Call: the AWS US-EAST-1 and Azure Front Door outages, and what Forrester's 2026 hyperscaler prediction means for planning.
  • New §08 — Backup Posture Management (CBPM): continuous discovery and drift detection replaces point-in-time backup audits.
  • New §10 — Recovery-as-Code: recovery steps expressed as versioned, auditable workflows instead of tribal-knowledge scripts.
  • New §12 — DORA and NIS2 compliance mapping for regulated entities operating across clouds.
  • Rewritten §13 — Cloud-specific service mappings refreshed for 2026 service names and capabilities across AWS, Azure, and GCP.
  • New §15–16 — DR maturity model and a 30/60/90-day rollout plan, mirroring the structure used across the handbook library.
  • Converted to a fully self-contained, standalone file — no external stylesheets, no CDN dependencies, dark/light theme built in.
02

Objectives & Scope

// WHY THIS RUNBOOK EXISTS

This handbook exists to turn disaster recovery from a slide in a compliance deck into a rehearsed, auditable capability. It applies to any organization running production workloads across two or more of AWS, Azure, and GCP — whether that multi-cloud footprint is a deliberate resilience strategy or the accumulated result of acquisitions, team preferences, and vendor negotiations.

  • Reduce ambiguity during major service disruption — every role, decision, and threshold is defined before the incident, not during it.
  • Standardize recovery language across engineering, leadership, and regulators so "we're recovering" means the same thing to everyone in the room.
  • Turn recovery from theory into rehearsal with runbooks that are tested on a cadence, not written once and forgotten.
  • Separate recovery, validation, and failback into explicit, independently-verifiable stages.
  • Prove backup posture continuously rather than assuming coverage because a policy exists somewhere.
Scope note: this handbook covers cross-cloud application and data recovery. It does not replace a full business-continuity plan (BCP), which also covers people, facilities, and third-party vendor dependencies — but it feeds directly into one, and the runbook requirements in §07 are written to satisfy BCP audit evidence needs.
03

The 2025–26 Wake-Up Call

// WHY "IT WON'T HAPPEN TO US" STOPPED BEING A PLAN

For years, multi-region and multi-cloud DR was treated as an expensive insurance policy that mature organizations bought "eventually." Two incidents ten days apart changed the conversation from theoretical to operational, and Forrester's 2026 cloud predictions turned it into a planning assumption.

October 20, 2025
AWS US-EAST-1 — DNS & DynamoDB Cascade
A fault in the internal subsystem monitoring network load balancer health in EC2 triggered a surge of DNS resolution errors for the DynamoDB API. Applications could not locate or connect to their databases. The outage ran roughly 15 hours, disrupted major banks, airlines, and SaaS tools such as Slack and Atlassian, and demonstrated that a control-plane failure in one heavily-used region can cascade globally — even for services architected for multi-AZ resilience within that region.
October 29, 2025
Azure Front Door — Global Identity & Routing Disruption
Nine days after the AWS incident, Microsoft Azure customers faced widespread access issues traced to Azure Front Door — the platform's global CDN and load-balancing layer — compounded by identity-management disruption. The proximity of the two incidents made a single point clear to boards worldwide: this was not a one-off, and single-provider dependency is a standing business risk, not a hypothetical one.
Late 2025
The Common Pattern Emerges
Post-incident analysis across AWS, Azure, and a separate Cloudflare disruption found a shared root-cause shape: small, seemingly innocuous changes — a race condition, a configuration that looked healthy for minutes before failing, a routine update — triggered systemic failures that exposed fragile assumptions. Teams that had tested failover before the incidents recovered measurably faster than teams discovering their runbooks were stale in real time.
2026 →
Forrester's Prediction: More of the Same
Forrester's Predictions 2026: Cloud Computing report forecasts at least two more major multi-day hyperscaler outages in 2026, attributing the risk to AWS, Azure, and Google Cloud prioritizing AI data-center capacity build-out over legacy infrastructure hardening. The practical implication for architects: treat hyperscaler regional or global disruption as a planning input with a non-trivial annual probability, not a black-swan event.
The hidden-dependency problem. Map not just where your application runs, but where your monitoring, alerting, authentication, and deployment pipeline run. If they all trace back to the same cloud provider as your production workload, you have a single point of failure with extra steps — your DR plan can be technically sound and still fail because the tooling you need to execute it is down too.
04

RTO & RPO Standards

// THE TWO NUMBERS EVERYTHING ELSE IS BUILT AROUND

RTO (Recovery Time Objective) defines the maximum acceptable duration between an outage and restored service. RPO (Recovery Point Objective) defines the maximum acceptable data-loss window, measured back from the moment of failure. Both must be defined per application tier and backed by demonstrated recovery capability — not optimistic assumptions written into a spreadsheet once and never re-tested.

🔴
Tier 0 — Critical
RTO < 15 min · RPO < 1 min. Active-active or synchronous replication required.
🟠
Tier 1 — High
RTO < 1 hr · RPO < 15 min. Warm standby with automated failover.
🔵
Tier 2 — Standard
RTO < 4 hrs · RPO < 1 hr. Pilot light with scripted rebuild.
🟢
Tier 3 — Deferred
RTO < 24 hrs · RPO < 4 hrs. Backup and restore is sufficient.
RTO/RPO must be validated, not declared. A Tier 0 classification with an untested restore path is a Tier 3 system wearing a Tier 0 label. Every declared RTO/RPO pair in §07's runbook requirements must be backed by evidence from a drill in §14 within the last testing cycle.
05

Recovery Patterns

// COST VS. RECOVERY SPEED TRADE-OFF

Four patterns cover nearly every production system. The right choice is the cheapest pattern that still meets the tier's RTO/RPO from §04 — over-provisioning resilience wastes budget that could harden a genuinely critical system instead.

Backup & Restore Tier 3

Lowest steady-state cost, slowest recovery — hours, sometimes longer. Data is backed up regularly and restored on demand. Suitable for lower-criticality systems where extended downtime is tolerable. The single biggest risk: an untested restore path silently rotting until the day you need it.

Pilot Light Tier 2

A minimal version of the environment — core data stores replicated, compute scaled to zero or near-zero — stays running at low cost. On failover, the "pilot light" is scaled up to full production capacity. Faster than backup/restore, cheaper than warm standby.

Warm Standby Tier 1

A reduced-capacity secondary environment runs continuously and can scale up on failover. Traffic can shift with a DNS or load-balancer change rather than a full rebuild. Higher steady-state cost, materially faster recovery, and the pattern most teams underestimate the ongoing cost of.

Active-Active Tier 0

Full production capacity runs simultaneously in two or more clouds or regions, actively serving traffic. Highest resilience, highest complexity, and highest cost — data consistency and conflict resolution become first-class engineering problems. Reserve for the handful of services where minutes of downtime have material business or regulatory consequence.

06

Anatomy of a Recovery Flow

// THE SIX STAGES OF A CONTROLLED RECOVERY

Every well-run recovery — whether it's an automated failover or a human-led incident response — passes through the same six stages. Skipping validation to "recover faster" is the single most common cause of a second, self-inflicted outage during failback.

🚨
Detect
Detect Incident
📋
Classify
Declare Severity & DR Mode
🛡️
Contain
Stabilize & Contain
🔀
Act
Failover / Restore
Verify
Validate App & Data
📢
Inform
Stakeholder Comms
↩ Planned failback once primary is verified healthy

RTO/RPO Standards in Practice

The declared RTO/RPO for each tier drives every decision downstream: which recovery pattern is affordable (§05), what the runbook must document (§07), what backup posture must prove (§08), and how often a drill must exercise the path (§14). Treat these values as engineering requirements with the same rigor as a latency SLO — because functionally, that's what they are.

07

Runbook Requirements

// WHAT "DOCUMENTED" ACTUALLY MEANS
  • Every production system has a declared RTO and RPO mapped to one of the four tiers in §04.
  • Recovery owner and incident commander are named — by role, with a current on-call rotation, not a person's name that goes stale when they change teams.
  • Backup location and restore steps are documented at the level of detail an engineer unfamiliar with the system could execute under pressure.
  • Validation steps cover application, data, and external integrations — a restored database that can't reach its payment processor is not a recovered system.
  • Communication templates exist before the incident starts — drafting a customer-facing status update during an active outage costs minutes you don't have.
  • Runbook test date and evidence are recorded, with a defined maximum staleness (see §14) before the runbook is considered unverified.
Runbook Header — Required Fields
# Every service-level DR runbook must declare this block at the top service: payments-api tier: 0 # Tier 0-3, see §04 rto: 15m rpo: 1m recovery_pattern: active-active primary_region: aws:us-east-1 secondary_region: gcp:us-central1 recovery_owner: on-call:payments-sre incident_commander: on-call:incident-lead last_drill_date: 2026-06-02 last_drill_result: pass # pass | pass-with-findings | fail next_drill_due: 2026-09-02 compliance_scope: [dora, pci-dss]
08

Backup Posture Management

// THE LAYER MOST DR PLANS SKIP

The most common cause of a failed multi-cloud recovery isn't a bad runbook — it's a resource that was never covered by a backup policy in the first place, or a policy that drifted out of sync as the environment changed. Most teams cannot currently prove, on demand, that every critical resource across every cloud account has a current, recoverable backup. Cloud Backup Posture Management (CBPM) is the operational discipline that closes that gap continuously rather than at quarterly-audit time.

Continuous Discovery Always On

Automatically discover new and changed resources across AWS, Azure, and GCP accounts as they're created — not on a scan cadence. A resource created outside the standard IaC pipeline is exactly the kind of gap that surfaces only during an incident.

Classification & Coverage By Sensitivity

Classify data by type and sensitivity, then map classification to the required backup policy and tier. A production customer database and a disposable dev-environment scratch volume should never share a policy by default.

Drift Detection Before Incident

Surface coverage gaps and policy drift in real time — a lifecycle rule that quietly expired, a snapshot schedule someone disabled to save cost, a new account that was never onboarded to the central backup policy.

Granular, Not Full-Environment Realistic Recovery

Most real recovery events are a single corrupted database, a deleted object, or a compromised storage bucket — not a full-environment rebuild. Design and test for granular, targeted restores as the common case, with full-environment failover as the rare, higher-severity case.

A backup policy that exists on paper is not backup coverage. The only credible evidence of coverage is a recent, successful restore test against that exact resource class. Treat "we have a backup policy" and "we can prove this resource is currently recoverable" as two different, independently-audited claims.
09

Data & Replication Standards

// MEASURED, NOT ASSUMED

Replication lag must be measured continuously, not assumed from a dashboard that says "healthy." Backups are only considered valid after a successful restore test — an unverified backup is a liability that looks like an asset. Failback must be treated as its own controlled event with explicit validation and rollback criteria, not the automatic reverse of a failover.

Replication ModeConsistencyTypical RPOCost Profile
Synchronous, cross-regionStrong~0 (seconds)Highest — latency-sensitive writes, premium networking
Asynchronous, continuousEventualSeconds–minutesModerate — most common pattern for Tier 1
Scheduled snapshot / log-shippingPoint-in-timeMinutes–hoursLow — standard for Tier 2–3
Periodic full backupPoint-in-timeHours–daysLowest — Tier 3 only
Recovery that is never exercised is not recovery capability. Schedule restore tests on the same cadence as the drills in §14 — a backup you have never restored from is a hypothesis, not a control.
10

Recovery-as-Code

// FROM TRIBAL SCRIPTS TO VERSIONED WORKFLOWS

Manual, ad-hoc recovery across multiple cloud providers introduces exactly the kind of cognitive load that produces mistakes at the worst possible time. The 2026 standard is to express the full restoration sequence as reusable, versioned, auditable workflows — pipelines and orchestrators, not custom shell scripts that live on one engineer's laptop and are difficult to audit or fork safely.

Recovery Workflow — Control Contract
# A recovery-as-code workflow definition — versioned, reviewed, auditable # Stored in the same repo discipline as application infrastructure code trigger: type: manual | automated condition: "health-check.status == DOWN for 5m" pre_checks: - verify_secondary_region_healthy - verify_backup_recency < rpo_threshold - notify: incident-commander execution: steps: - promote_replica: target=secondary, mode=async-to-primary - update_dns: provider=route53|traffic-manager|cloud-dns - scale_secondary: to=production-capacity - run_smoke_tests: suite=critical-path verification: - checklist: app_health, data_integrity, external_integrations - require: human_signoff=true # for Tier 0/1 rollback: - condition: "verification.failed == true" - action: revert_dns, demote_secondary, page_incident_commander audit: - log_all_steps_to: immutable_ledger - record: actor, timestamp, region, outcome
Recovery-as-code is not the same as "fully automatic." For Tier 0/1 systems, keep a human-signoff gate before the workflow completes — the goal is removing ambiguity and manual error from execution, not removing judgment from the decision to fail over.
11

Communications & Escalation

// PARALLEL, NOT SEQUENTIAL

Major incidents require parallel technical recovery and stakeholder communication — not one after the other. Define, before the incident, who communicates to leadership, who communicates to customers, and who owns engineering coordination, so the recovery team is never interrupted to answer the same "what's the status" question twice.

Engineering Coordination

Incident commander owns the technical recovery timeline and runbook execution. Single source of truth for status; all decisions logged in the incident channel.

Leadership Updates

A designated liaison — not the incident commander — briefs leadership on a fixed cadence (e.g. every 30 minutes for Tier 0) so leadership questions don't interrupt recovery work.

Customer Communication

Pre-approved status-page and customer-comms templates, filled in and published by a designated owner, following the severity classification declared in §06.

12

DORA & NIS2 Compliance

// WHEN DR EVIDENCE BECOMES A REGULATORY REQUIREMENT

For EU financial entities, the Digital Operational Resilience Act (DORA) — in force since January 17, 2025 — turns much of this handbook from best practice into a legal obligation. DORA requires ICT business continuity policies, disaster recovery plans, and backup policies that are tested regularly and updated based on lessons learned from disruptions, resilience testing, and audit findings. Non-compliance penalties reach into the millions of euros or a percentage of global annual turnover.

DORA PillarWhat It RequiresWhere This Handbook Covers It
ICT Risk ManagementDocumented framework covering identification, protection, detection, response, recovery§07 Runbook Requirements, §08 Backup Posture
Incident ReportingStandardized classification and reporting of major ICT incidents to regulators within strict timeframes§06 Recovery Flow (Classify stage), §11 Communications
Resilience TestingRegular recovery drills, vulnerability assessments; Threat-Led Penetration Testing (TLPT) for significant entities§14 Testing & Chaos Drills
Third-Party ICT RiskRegister of Information for critical ICT third-party arrangements, including cloud providers§13 Cloud Mappings, §08 Backup Posture

Outside financial services, the EU's NIS2 Directive extends comparable ICT risk-management and incident-reporting obligations to a broader set of "essential" and "important" entities across energy, healthcare, transport, and digital infrastructure sectors. Even organizations outside direct DORA/NIS2 scope increasingly see these frameworks referenced in customer security questionnaires and vendor due-diligence — treat the runbook evidence this handbook requires as reusable compliance artifact, not overhead.

2026 practical shift: regulators have moved from reviewing paperwork to demanding real-time, defensible evidence — automated reporting and demonstrable, continuous control over ICT risk, rather than a point-in-time attestation. This is the same shift §08's continuous backup posture management is designed to support.
13

Cloud-Specific Mappings

// AWS · AZURE · GCP — 2026 SERVICE NAMES
CapabilityAWSAzureGCP
Global routing / failoverRoute 53 / Global AcceleratorAzure Front Door / Traffic ManagerGlobal Cloud Load Balancing / Cloud DNS
VM / workload backupAWS BackupAzure BackupBackup and DR Service
Server-level DR / replicationAWS Elastic Disaster RecoveryAzure Site RecoveryBackup and DR Service (replication)
Database replicationAurora Global Database / DynamoDB Global TablesCosmos DB multi-region / Azure SQL geo-replicationCloud Spanner / Cloud SQL cross-region replicas
Runbook automationSystems Manager Automation / Step FunctionsAzure Automation / Logic Apps / Durable FunctionsWorkflows / Cloud Functions / Cloud Composer
ObservabilityCloudWatch / X-RayAzure Monitor / Application InsightsCloud Monitoring / Cloud Logging / Cloud Trace
Backup posture / complianceAWS Backup Audit Manager / ConfigAzure Policy / Microsoft Defender for CloudSecurity Command Center / Policy Intelligence
Incident & status communicationAWS Health Dashboard, EventBridge alertsAzure Service Health, Resource Health alertsPersonalized Service Health, Cloud Monitoring alerts
Names change; verify before you build. Cloud providers rename and re-scope DR-adjacent services frequently. Treat this table as a starting map for a design conversation, and confirm current service names, regional availability, and SLAs against each provider's documentation before committing a runbook to a specific service.
14

Testing & Chaos Drills

// NON-DISRUPTIVE VALIDATION AS A HABIT

Regular testing is what separates a disaster recovery plan from a disaster recovery capability. Non-disruptive testing techniques let teams validate failover procedures and confirm RTO/RPO targets without impacting production workloads — there is no longer a good excuse for "we can't test because it's too risky."

  • Recovery roles are current — verify the on-call rotation and named owners in every runbook header still resolve to real people.
  • Failover triggers and decision rules are understood by everyone on the rotation, not just the person who wrote the runbook.
  • Restore procedures have been tested recently, against the specific staleness threshold declared per tier (Tier 0: monthly, Tier 1: quarterly, Tier 2: semi-annually, Tier 3: annually).
  • Runbooks match the live architecture — a drill that surfaces a stale runbook is a successful drill; a drill skipped because "the runbook is probably fine" is not.
  • Failback is rehearsed, not improvised — treat failback as its own drill, not an assumed inverse of failover.
  • Chaos scenarios include hidden dependencies — deliberately fail the monitoring or auth system a recovery workflow depends on, per §03's lesson.
Drill TypeWhat It ValidatesDisruption Level
Tabletop exerciseRoles, decision rules, communication templatesNone — discussion only
Restore test (isolated)Backup integrity and restore procedure, in a sandboxed environmentNone — production untouched
Game-day failoverEnd-to-end failover of a live but non-critical path, or during a defined low-traffic windowLow — scoped and monitored
Full regional failoverComplete RTO/RPO validation for a Tier 0 system under real conditionsPlanned, scheduled, with rollback ready
15

DR Maturity Model

// WHERE ARE YOU TODAY

Most organizations are at Stage 1 or early Stage 2 for the majority of their services, with only their most critical systems reaching Stage 3 or 4. Advance tier by tier — a Stage 2 capability with honest, tested RTO/RPO numbers is worth more than a Stage 4 architecture diagram that has never been drilled.

Stage 01
Reactive
  • Backups exist, untested
  • No declared RTO/RPO
  • Recovery is improvised
  • Single-cloud, single-region
  • No named recovery owner
Stage 02
Documented
  • RTO/RPO declared per tier
  • Runbooks written, rarely tested
  • Backup posture spot-checked
  • Manual failover procedures
  • Named owners, informal rotation
Stage 03
Rehearsed
  • Drills on a fixed cadence
  • Recovery-as-code for Tier 0/1
  • Continuous backup posture (CBPM)
  • Communication templates ready
  • Compliance evidence tracked
Stage 04
Resilient
  • Active-active for Tier 0
  • Automated chaos drills
  • Hidden-dependency mapping
  • Cross-cloud cost-of-failover modeled
  • Continuous DORA/NIS2 evidence
16

Implementation Roadmap

// 90-DAY ONRAMP FROM REACTIVE TO REHEARSED

Do not attempt to bring every service to Stage 3 simultaneously. Start with the systems where a failure would be most damaging, prove the pattern works, then extend it service by service.

90-Day Disaster Recovery Rollout
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ DAYS 1-30 — CLASSIFY & DISCOVER ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ✦ Inventory production systems, assign a tier (§04) to each ✦ Map hidden dependencies: monitoring, auth, CI/CD, DNS — not just app + DB ✦ Run a backup posture discovery pass across every cloud account (§08) ✦ Identify coverage gaps and policy drift; triage by tier ✦ Draft the runbook header block (§07) for every Tier 0/1 system ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ DAYS 31-60 — VALIDATE & DOCUMENT ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ✦ Run isolated restore tests for every Tier 0/1 backup ✦ Complete full runbooks: recovery steps, validation checklist, comms templates ✦ Assign recovery owner and incident commander roles to on-call rotations ✦ Run tabletop exercises for Tier 0 systems ✦ Stand up continuous backup posture monitoring (CBPM) for Tier 0/1 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ DAYS 61-90 — REHEARSE & AUTOMATE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ✦ Run the first game-day failover for the highest-priority Tier 0 system ✦ Convert manual recovery steps to recovery-as-code (§10) where it reduces risk ✦ Rehearse failback as an independent drill, not an assumed reversal ✦ Map DORA/NIS2 evidence requirements against current runbook artifacts ✦ Schedule the next drill cycle and publish the maturity self-assessment (§15)
Common Rollout Mistakes Avoid
  • Declaring RTO/RPO without a validated restore test to back it
  • Treating backup existence as equivalent to backup coverage
  • Skipping the hidden-dependency map — monitoring and auth outages block recovery too
  • Automating failover before a human-signoff gate exists for Tier 0/1
  • Running drills only for the systems everyone already trusts
Success Indicators Measure These
  • 100% of Tier 0/1 systems have a drill result less than one testing cycle old
  • Backup posture gaps trend toward zero and stay there
  • Mean time to recover trending down release over release
  • Failback treated and drilled as its own event, every time
  • Compliance evidence generated as a byproduct of drills, not a separate scramble
Reference sources: Forrester's Predictions 2026: Cloud Computing report · post-incident analyses of the October 2025 AWS US-EAST-1 and Azure Front Door outages · EU Regulation 2022/2554 (DORA) and the NIS2 Directive · CNCF and cloud-provider disaster recovery documentation for AWS, Azure, and GCP.
Back to Handbooks