Updated recently
Last updated:
Quick Answer: AWS architecture patterns high availability work is mostly unglamorous: spread every stateful component across at least two AZs, decouple services with queues, bound failure with retries, circuit breakers and bulkheads, degrade gracefully instead of erroring, and test the whole thing with failure drills. Multi-AZ infrastructure without those behaviours is a diagram, not availability.
Why “Multi-AZ” Does Not Mean Available
Every architecture review I have been part of included a slide with three AZ boxes and arrows between them. Almost none of those systems would have survived an AZ event, because the diagram covered infrastructure and the outage lived in the assumptions: a single NAT gateway, a database in one AZ, a cache with no failover path, an SDK timeout of thirty seconds that turned a five-second blip into a thread-pool exhaustion cascade.
The gap exists because availability is not a property of where you place servers. It is a property of what your system does when a dependency answers slowly, answers wrongly, or does not answer at all. That is the shift these AWS architecture patterns are built around — designing for the failure you have not had yet, rather than documenting the redundancy you have already bought.
It is also worth being honest about scope. Two AZs cover the failure mode that hits most frequently and costs least to defend against. Regional failure is rarer and much more expensive to design for, so treat multi-region as a business decision with a price tag, not a technical checkbox. Most teams should get two-AZ plus graceful degradation genuinely right before spending anything on the second region.
The AWS Architecture Patterns High Availability Actually Rests On
| Pattern | Failure it survives | What it costs you |
|---|---|---|
| Multi-AZ stateful services | Single AZ impairment | Some replica lag and extra hourly cost |
| Async decoupling (SQS, EventBridge) | Downstream slowdown or outage | Eventual consistency, idempotency work |
| Retry with backoff and jitter | Transient errors, throttling | Latency when things are genuinely down |
| Circuit breaker | Dependency failure cascading into yours | Serving degraded responses on purpose |
| Bulkhead / pool isolation | One workload starving another | Lower theoretical utilisation |
| Graceful degradation | Partial feature loss under stress | Two code paths to maintain |
| Idempotency keys | Duplicate delivery after retries | Design discipline on every write path |
| Health-checked failover | Silent corruption of one node | Real probes that test dependencies |
Read the third column before you adopt anything. Every pattern trades peak efficiency for survival, and the engineering argument you need to win internally is that trade — not whether the pattern is “correct.” It is.
Multi-AZ Done Properly, Not Neatly
The placement rules are simple to state and consistently violated:
- Load balancer subnets in two AZs, targets spread across both by default.
- RDS with Multi-AZ failover and a read replica in the second AZ for workloads that tolerate read lag.
- NAT gateways per AZ — one gateway with every private subnet pointing at it is the single most common single point of failure in an otherwise “multi-AZ” VPC. This exact mistake appears in our VPC breakdown and it survives every design review I have seen.
- ElastiCache, OpenSearch and MSK clusters spread across AZs with topology awareness enabled.
- Autoscaling groups with
balancedzone selection so a scale-out event does not pile every new instance into one AZ.
Then verify rather than assume. Configuration drift is how a correct design becomes an incorrect reality six months later — one Terraform apply against a modified console state is usually enough.
# Does the database actually fail over, or just claim to?
aws rds describe-db-instances --db-instance-identifier orders-db
--query 'DBInstance.[AvailabilityZone,MultiAZ,BackupRetentionPeriod,DeletionProtection]'
# Is every AZ in the VPC carrying private subnets with their own route table?
aws ec2 describe-subnets --filters Name=vpc-id,Values=vpc-0abc123def456
--query 'Subnets[].[AvailabilityZoneId,MapPublicIpOnLaunch,CidrBlock]'
# Are the NAT gateways distributed, or all in one AZ?
aws ec2 describe-nat-gateways --filter Name=vpc-id,Values=vpc-0abc123def456
--query 'NatGateways[].[NatGatewayId,SubnetId,State]'
Decouple, Then Make Writes Idempotent
Queues are the most effective availability pattern available to you, for one reason: they convert a synchronous dependency failure into a backlog. When the invoicing service is down, the order service keeps accepting orders and the invoicing service catches up afterwards. Without a queue, the order service either fails or waits — and waiting is how thread pools die.
The price is exactly-once delivery becoming at-least-once delivery, which means every consumer must be idempotent. Deduplicate on a message ID stored with a short TTL, make the write conditional, or use an idempotency key on the API. Teams that skip this discover duplicates in production: the shipment that went out twice, the customer charged twice. In my experience, the second incident is what finally gets idempotency scheduled.
Bound the Failure: Retries, Breakers, Bulkheads
Retries are a reflex and the reflex is often wrong. Retrying immediately amplifies an outage — you replace one failing call with fifty. Retrying a non-idempotent operation creates duplicates. Retrying forever turns a dependency problem into a thread-pool problem in your own service.
The defaults that hold up: exponential backoff with full jitter, a small retry budget (a percentage of traffic, not a count), and a hard deadline across the whole call chain that is shorter than the caller’s timeout. Add a circuit breaker so that once a dependency fails repeatedly, you stop calling it for a cooling-off period and serve a fallback immediately. Add bulkheads — separate connection pools or queues per workload — so the reports endpoint cannot consume the connections the checkout path needs.
Graceful Degradation Beats Clever Failover
The best availability behaviour is usually invisible: the system serves a worse version of itself instead of an error page. Recommendations fall back to a cached or popular-items list. The product image service is down, so thumbnails come from CloudFront’s cached copy. Search is degraded, so the category browse still works. None of these require multi-region infrastructure; they require deciding in advance which features are optional.
Write that list down. The teams that handle outages well made the “what can we lose?” decision on a calm afternoon, and the teams that handle them badly make it live, on a bridge call, while executives ask questions.
The AZ Outage That Exposed a Single NAT Gateway
The pattern I keep seeing validated the hard way: a payments-adjacent service in ap-south-1 was properly multi-AZ — ALB across two AZs, instances in an ASG across two AZs, RDS Multi-AZ. During a regional event that degraded one AZ for about forty minutes, every private-subnet instance in the healthy AZ started failing outbound calls. All of them routed through one NAT gateway sitting in the impaired AZ.
What actually broke downstream was worse than the networking fault: the SDK retries on outbound calls to the payment gateway were unbounded with no jitter, so thread pools filled, the ALB health checks started timing out, and instances were terminated and replaced repeatedly — which, in an Auto Scaling group, produced the appearance of a total collapse while the fix was a single route-table change. Total customer impact ran far past the actual network impairment.
The remediation list is the checklist for this entire article: NAT gateway per AZ (plus gateway endpoints for S3 and DynamoDB, which removed most of the NAT traffic anyway), bounded retries with jitter everywhere, a circuit breaker on the payment-gateway client, health checks that fail on dependency state rather than process state, and — the part that actually prevented recurrence — a quarterly failure drill that deliberately terminates the NAT gateway in one AZ and watches who notices first.
How to Prove It Works: Failure Drills
- Start with the cheap drills. Terminate an instance, block an AZ’s NAT route, revoke a task role. Observe whether alerts fire before customers notice.
- Measure time to detect, not time to fix. If detection takes longer than impact, your observability is the weak pillar, not your architecture.
- Run a restore drill. Restore the database to a point in time, every quarter. A backup you have never restored is a hypothesis.
- Write the timeline afterwards. What degraded, what did not, what surprised everyone. Feed it back into your Well-Architected review, where every risk already has an owner and a date attached to it.
- Codify every fix. Hand-fixed infrastructure drifts back. Put the guardrails in Terraform and policy checks — see Terraform vs CDK vs CloudFormation for where each fits.
One expectation to reset: none of this makes a system unbreakable. It makes failure cheap, visible and survivable — which is the only version of availability an architecture can honestly promise.
Training Resources
Design and then break these patterns deliberately in SkilBrill AWS Advanced Architecture Training — resilience patterns, multi-AZ failure drills, cost and reliability trade-offs. The foundations underneath, including VPC and networking, are covered in AWS Cloud Architecture Training.
FAQ
What does high availability mean on AWS?
It means the system continues serving users when components fail — typically achieved with multi-AZ deployment, load balancing with real health checks, queues that absorb downstream outages, and degradation behaviour that serves reduced functionality instead of errors.
Which AWS architecture patterns prevent most outages?
Distributing NAT gateways and stateful services across AZs, bounded retries with jitter, circuit breakers on external dependencies, idempotent consumers behind queues, and health checks that validate dependency state. Those five would have prevented most incidents I have seen.
Is multi-region architecture worth it?
Only if your RTO genuinely requires it and leadership accepts the cost — active-active doubles infrastructure, complicates data consistency and needs real testing. For most workloads, two AZs plus tested backups and graceful degradation is the proportionate answer.
How do I test if my architecture survives an AZ failure?
Run controlled failure drills: remove one AZ’s NAT route, terminate instances, fail over the database manually, and measure time to detect versus time to recover. Fix what the drill exposes before a real event does it for you.
Why do my retries make an outage worse?
Because immediate, unbounded retries amplify load on an already failing dependency and hold your threads hostage. Use exponential backoff with jitter, a small retry budget, and a global deadline shorter than the caller’s timeout.
