Region failover

Failover is a rehearsed control, not a region you pick in the incident channel. If the standby was never built, this runbook cannot invent it. Context: cross-region disaster recovery.

Symptoms

Probes show a region-level failure: API errors, storage timeouts, or a zonal event that exhausted remaining capacity. Route 53, Traffic Manager, Front Door, Cloud DNS or GTM health checks go unhealthy for the primary. Users in one geography fail while another still works (that can also be DNS or CDN). A single AZ failure is not automatically a region failover. Stay in-region if the design says multi-AZ.

Business impact

Failover changes RPO and often write location. Clients keep cached DNS answers until TTL expires. Both regions accepting writes is split-brain. Standby traffic also incurs cross-region traffic cost. Failing over without a current replica is data loss.

Immediate checks

1. Open the application DR document. Note RTO, RPO, the DNS or anycast object, and who is allowed to flip it.
2. Confirm the failed dependency (compute, database, identity, a vendor API). A bad deploy can look like a region event.
3. Confirm the standby region: compute healthy, replica lag within RPO, secrets and certificates present, quota enough to absorb traffic.
4. Confirm you are looking at documented failover (health-checked DNS or Front Door origins), not an opinion that “we can just point Route 53 at the other ALB.”
5. Snapshot or record primary state if the provider still allows reads. Do not start a long backup if RTO is minutes.

Provider commands and console paths

AWS (documented DNS failover). Route 53, health checks and failover records. Behaviour: Configuring DNS failover.

aws route53 get-health-check-status --health-check-id ID
aws route53 get-health-check --health-check-id ID
aws route53 list-resource-record-sets --hosted-zone-id ZONE

Azure (documented). Traffic Manager: priority (or another method you already chose) plus endpoint monitoring (Traffic Manager). Front Door: origin health and origin groups (Front Door). These are different products. Traffic Manager is DNS. Front Door is an anycast HTTP proxy.

az network traffic-manager profile show --resource-group RG --name PROFILE
az network traffic-manager endpoint list --resource-group RG --profile-name PROFILE
az afd origin list --resource-group RG --profile-name PROFILE --origin-group-name GROUP

If az afd is missing, use the Front Door portal origin-health blade.

Google Cloud (documented). Cloud DNS routing policies with health checks (routing policies). Use Cloud Load Balancing failover only if it is in the design.

gcloud dns record-sets list --zone=ZONE
gcloud dns record-sets describe RECORD --type=TYPE --zone=ZONE

Alibaba Cloud (documented). Global Traffic Manager attached to Alibaba Cloud DNS, with the health checks you configured. Start from GTM 3.0 and the GTM console. Do not invent aliyun GTM flags here.

Opinion, not a provider failover product. Copying AMIs, changing an app config to another region’s URL, or updating a spreadsheet of IPs is a manual move. It can work. It is not Route 53 failover.

Interpretation

Health check failed, standby healthy, replica inside RPO: invoke the documented flip.

Health check failed, standby lag exceeds RPO: failover is a data-loss decision. That belongs to the product owner, not the on-call default.

Health check failed, primary compute is fine: the check is wrong (path, security group, or TLS). Fix the check. Do not fail away from a healthy region.

Front Door origin unhealthy, Traffic Manager still pointing at the same region: you may have two layers. Fail over the layer that actually receives clients.

Safe mitigation

Follow the application’s DR steps: promote the data store if the design says promote, then shift traffic with the DNS or Front Door object already in the design. If failover is automatic, stop forcing the primary record. If it is manual, change the failover record or disable the primary endpoint through the same API you used in the last drill. Keep the primary from accepting writes if the design requires that. Traffic shift without data promote splits the dataset.

Risky actions to avoid

Do not create a new hosted zone, a new Traffic Manager profile, or a new Front Door profile during the outage.

Do not lower TTL to “zero” as a first action unless the design already uses a short TTL and you understand resolver behaviour. Providers let you set TTL; they do not promise every resolver honours an emergency change instantly. See the DNS runbook for lookup diagnosis: DNS routing failure.

Do not fail over because a status page is yellow if your own dependencies are healthy.

Do not run a full production cutover as the first test of a design that has never been drilled.

Validation

dig against several public resolvers should return the standby endpoints after caches expire. Front Door metrics should show the standby origin taking requests. Application reads and writes must hit the promoted data store. Record failback criteria.

Point external probes, including Vigiles infrastructure monitoring, at the client-facing name, not only at the primary region’s internal name.

Prevention

1. Write RTO and RPO per service, including who declares failover.
2. Keep the standby current: IAM, certificates, container images, and the quotas a failover will consume. See quota exhaustion.
3. Drill failover on a schedule. An untested DNS or origin change is a design, not a recovery capability.
4. Prefer documented products (Route 53 health checks, Azure Traffic Manager or Front Door, Cloud DNS routing policies, Alibaba GTM) over a runbook that says “update the A record.”
5. If a step is not on the provider pages linked below, label it as your design, not as a vendor feature.