Built and measured in August 2026 across us-east-1 and us-west-2, run as a public demo until 30 October 2026, then torn down. The infrastructure no longer exists — the code and full write-up are in the repo.
The architecture
Primary path (top): Route 53 primary record → us-east-1 API Gateway → Lambda → DynamoDB replica. Failover path (bottom): the primary health check fails twice → secondary record → the identical us-west-2 stack. One global table, replicated between the regions.
One table, not two that sync. A global table is a single logical table with a replica in each region; DynamoDB owns the replication. It’s the only piece of the stack that isn’t duplicated, and it’s the reason data survives losing a region at all.
What was measured
The primary was broken for real, not simulated: Lambda reserved concurrency set to 0, so every invocation throttles and the API returns 503. An actual compute failure, and fully reversible.
| Break | Restore | |
|---|---|---|
| Users switched region | 90 s | 120 s |
| Alarm changed state | 270 s | 300 s |
Two separate things worth separating.
Users moved in 90 seconds. Route 53’s health check runs at a 30-second interval with a failure threshold of 2, so ~60 seconds is the detection floor before it even begins serving the secondary, and resolver caching adds the rest. That’s the real recovery time, and it matches the 60–80 seconds measured on the original build.
The alarm arrived three minutes after that — and that’s correct, not a bug. Route 53 acts on its own health checks directly; the CloudWatch metric is a downstream publication with its own lag, plus two 60-second evaluation periods on top. The alarm’s job is to tell me to look, not to perform the failover. By the time it fires, users have been served by the other region for minutes.
Confusing those two numbers is how people end up quoting an RTO that’s three times worse than reality — or, worse, wiring an alarm into an automated response and discovering it acts long after the thing it’s reacting to has resolved itself.
Failback works too, which is a separate claim needing its own test: plenty of setups fail over correctly and never come home.
The trap that would have broken it
Never point a health check at the record it controls.
If the primary’s health check resolved api.lab.andruxa.dev, then: the primary fails → DNS moves to the secondary → the primary’s check now resolves to the secondary → reports healthy → traffic flaps back to a region that’s still broken.
So each check targets its own region’s execute-api hostname directly. That also matters because a health check sends the FQDN it’s configured with as the Host header, and the custom domain would reject an unexpected one.
A DR setup that can’t fail back is worse than one that can’t fail over, because it looks fine.
Details worth knowing
Alias records pointing at API Gateway can’t use EvaluateTargetHealth, so each failover record needs an explicit health check attached.
Regional API Gateway needs its certificate in its own region — this is not the CloudFront us-east-1 rule. Two certs, one per region, both validating against the same DNS name.
servedBy in every response is what made this debuggable. Having each region stamp its identity into the payload turned “did failover happen?” from a guess into a string comparison. The same trick appears in the deployment measurements and the routing project.
Running it in public — and why it stopped
It ran as a public demo for eleven weeks, guarded with a 5 req/s throttle on both regions’ APIs and a 24-hour TTL on every write so the table couldn’t grow without bound. The TTL attribute is epoch seconds; milliseconds would mean the year 56000 and nothing would ever be deleted. DynamoDB replicates the expiry deletions to the other region like any other write.
I priced it at $1 a month. It cost about $6. Everything is pay-per-request, so I assumed idle meant free and counted only the two health checks. But a health check is traffic: roughly fifteen checkers each probing every 30 seconds came to about 45,000 requests a day per region, every one billed by API Gateway, Lambda and DynamoDB — visitors or not. The HTTPS option on the checks added another $2. A cost check across the whole account caught it, and a live endpoint wasn’t worth three times my monthly budget when the measurements above already make the case.
The lesson generalises: the monitor is a client. Price the traffic your health checks generate, not just the checks.
How this compares
It’s one of three failover mechanisms I built and timed against each other — the full comparison is here, including anycast, which is faster and more expensive.
Links
- Source and full write-up
- Terraform version — the same stack as provider aliases