Designing Resilient Systems on GCP: Load Balancing, MIGs and Health Checks

Quick Answer: A GCP resilient architecture is a regional external Application Load Balancer in front of a regional managed instance group spread across two or more zones, with an HTTP health check that tests only the process itself and autohealing tuned to tolerate slow boots. The load balancer gives you anycast VIP failover, the MIG gives you zone independence, and the health check decides whether any of it actually works.

What “Resilient” Actually Means, and What It Does Not

Resilience is a number, not a quality. Before you pick components, write down two: RTO (how long an outage may last) and RPO (how much data you may lose). A marketing site with an RTO of one hour and a payment service with an RTO of five minutes need different designs, and pretending one architecture fits both is how teams waste budget on redundancy they will never exercise.

Also worth stating plainly: redundancy without testing is decoration. Every design below only counts if you have run a failure injection and watched what happened — the zone drain, the failed health check, the bad deploy. In my experience the most common GCP outage in mid-sized Indian product teams is not a zone failure. It is a deploy that health checks themselves got wrong.

The GCP Resilient Architecture Reference Topology

For a stateless web service, this is the baseline I would defend in a design review. It is deliberately boring.

Two deliberate choices in there. The load balancer is regional rather than global because the primary audience is Indian and you avoid the latency of an extra hop — global is the right answer when you have users in multiple continents. And the MIG is regional rather than zonal because a zonal MIG gives you no automatic capacity replacement when the zone itself has a problem.

Health Checks: The Component That Silently Destroys Everything

If you take one idea from this article, make it this: the health check is a contract, not a formality. It decides which instances receive traffic, when autohealing fires, and — during a rolling deploy — when new instances are allowed to come up. A wrong health check does not produce a warning. It produces an outage that looks like a capacity problem.

The rules that hold up in production:

• Check the process, not its dependencies. If /healthz calls your database, then a brief database failover fails every instance at once and autohealing will start destroying them. Test that the application can serve; check dependencies separately and report them, but do not gate traffic on them.
• Be specific. A 200 from / often comes from a reverse proxy before the app is even loaded. Point at an endpoint you own that returns 200 only when the app is genuinely ready.
• Set the initial delay honestly. If your app takes 90 seconds to warm caches, an initial delay of 30 seconds guarantees a flap on every deploy. Three times your observed boot time is a reasonable starting point.
• Make the threshold wider than a blip. Two unhealthy probes with a 10-second interval means 20 seconds of protection. Transient blips should not trigger healing.
• Separate liveness from readiness if your platform lets you. Liveness restarts the process; readiness only removes it from the pool. Conflating them turns a slow dependency into a crash loop.

MIGs, Regional MIGs and GKE: Choosing the Compute Layer

Honest note: Cloud Run removes most of the MIG machinery described here — it does health checks, scaling and zone distribution for you. If your service is a stateless HTTP handler with no long-lived connections, it is the lower-effort answer and you should say so in the design review rather than defaulting to VMs because they feel more serious.

Managed instance groups earn their complexity when you need custom machine types, GPU, long-lived processes, sidecar agents, or licensing that does not fit a container.

Load Balancer Choices, Briefly

• Regional external Application Load Balancer — the default for HTTP/HTTPS APIs in one region. Anycast VIP, cross-zone by design, integrates with backend services and health checks directly.
• Global external Application Load Balancer — when your users span continents. Adds Google’s edge network and Premium Tier routing.
• Internal Application Load Balancer — for service-to-service traffic inside a VPC, replacing hand-rolled DNS round-robin.
• Network Load Balancer (L4) — when you need raw TCP/UDP performance and will handle TLS yourself. It does not terminate HTTP and does not understand HTTP health checks the same way.

If you are coming from AWS, the mental model transfers more cleanly than the product names suggest — an AWS ALB is the close cousin of GCP’s regional external Application Load Balancer, and the VPC primitives underneath behave almost identically. Our AWS VPC explainer walks the same concepts on the other side, and AWS vs GCP in India covers how much that cross-cloud fluency is actually worth when you change employers.

Choosing the L4 balancer because it “sounds more resilient” is a common error. It is faster at the transport layer and dumber at the application layer — you lose the HTTP health checking, cookie affinity and header handling that most web services quietly depend on.

Build It: Health Check, Regional MIG and Backend

# 1. Health check — tests the app, not the database behind it
gcloud compute health-checks create http app-hc 
  --port=8080 
  --request-path=/healthz 
  --check-interval=10s 
  --timeout=5s 
  --healthy-threshold=2 
  --unhealthy-threshold=3

# 2. Template: what an instance should look like
gcloud compute instance-templates create app-templ 
  --machine-type=e2-medium 
  --image-family=debian-12 
  --image-project=debian-cloud 
  --metadata=startup-script='#!/bin/bash
    apt-get update -qq
    /opt/app/bin/start-server' 
  --tags=http-server --scopes=cloud-platform

# 3. Regional MIG spread across zones, with a fixed floor of 3
gcloud compute instance-groups managed create app-mig 
  --region=asia-south1 
  --template=app-templ 
  --size=3 
  --distribution-policy-zones=asia-south1-a,asia-south1-b,asia-south1-c

gcloud compute instance-groups managed set-autoscaling app-mig 
  --region=asia-south1 
  --min-num-replicas=3 
  --max-num-replicas=12 
  --target-cpu-utilization=0.65 
  --cool-down-period=120

# 4. Backend service that actually uses the health check
gcloud compute backend-services create app-bes 
  --global 
  --health-checks=app-hc 
  --session-affinity=NONE 
  --load-balancing-scheme=EXTERNAL_MANAGED

gcloud compute backend-services add-backend app-bes 
  --global 
  --instance-group=app-mig 
  --instance-group-region=asia-south1 
  --balancing-mode=UTILIZATION

Autohealing is configured on the MIG against the same health check, with an initial delay long enough to cover your real boot time. The number to get right is that delay — too short and autohealing fights your deploy; too long and genuinely dead instances sit in the pool stealing traffic. Also check the MIG’s maximum-unhealthy allowance: if you set it to 100%, a systematic problem can take the whole group down at once, which is precisely the failure the next section describes.

The Deploy That Took Down All Six Instances

A logistics platform in Pune ran a regional MIG of six instances behind an internal Application Load Balancer. The team had done the right thing structurally — multi-zone, autoscaling, health checks. What they had not done was look at what their health check actually returned.

The endpoint was /health, implemented as a handler that verified a connection to the primary database and returned 503 if the query failed. During a routine Cloud SQL failover that lasted about ninety seconds, every instance failed its health check simultaneously. Autohealing kicked in, judged the group unhealthy, and began recreating instances. Each new instance ran a startup script that performed a database migration on boot — and the migration could not complete against a database in failover.

The database recovered in ninety seconds. The instance group took another forty minutes to stabilise, because new instances kept failing the same check that had failed in the first place. Customers saw an outage of roughly forty-one minutes caused by a ninety-second database blip.

Three changes fixed it permanently. The health check was rewritten to return 200 based only on the process being ready to serve, with database connectivity reported in a separate detail field that did not gate traffic. Migrations were moved out of the startup path into an explicit release step. And the autohealing policy was given an initial delay plus a maximum-unhealthy bound so a shared dependency failure could not empty the group.

The wider lesson: a health check that reaches into a shared dependency converts a partial failure into a total one. That single line of code did more damage than the infrastructure’s lack of redundancy ever would have.

Failure Modes to Test Before You Call It Done

1. Drain one zone. Stop the instance group in a single zone and confirm capacity and traffic redistribute without a user-visible error.
2. Break the health check deliberately. Point it at a path that returns 503 and watch what autohealing does. If the answer is “destroys everything”, your thresholds are wrong.
3. Ship a bad deploy. Push a version that fails to boot, confirm the rolling update halts rather than completing, and verify rollback time.
4. Kill the database connection. Simulate a failover and confirm the application reports degraded but keeps serving static and cached content.
5. Check the bill after each test. Repeated recreation leaves orphaned disks and addresses. Resilience work has a cost line that should be reviewed, not ignored.

Run these as a documented exercise and you will know your real RTO instead of guessing it. That number is worth having when a customer, an auditor or an investor asks — and being the engineer who can walk a panel through a tested failure mode is one of the clearest senior-level signals there is, which our GCP engineer salary guide reflects in the pay bands.

Training Resources

Design and test this topology in SkilBrill Google Cloud Architecture Training — load balancing, regional MIGs, health checks and failure-injection exercises built as working labs. For the networking foundations underneath it, see Google Cloud Fundamentals Training.

FAQ

What makes a GCP architecture resilient?

Regional components spread across zones, a load balancer that reroutes around failures, health checks that detect real problems, autoscaling with a sensible floor, and — most importantly — tested failure modes. Redundancy that has never been exercised is an assumption, not a design.

Should I use a zonal or regional managed instance group?

Regional for anything serving traffic. A zonal MIG has no capacity replacement when its zone has an issue, and the cost difference is negligible compared to the outage risk. Use zonal only for non-critical batch or internal tools.

What path should a GCP health check use?

An endpoint you own that returns 200 only when the process is genuinely ready to serve — never one that queries the database or a downstream API. Dependency failures should be reported, not used to gate traffic, or a single shared dependency takes down every instance at once.

Regional or global load balancer on GCP?

Regional external Application Load Balancer when your users are concentrated in one region and latency matters. Global when you serve multiple continents and want Premium Tier edge routing. Both provide anycast VIP and cross-zone handling.

When should I choose Cloud Run instead of a managed instance group?

When the service is stateless HTTP with no long-lived connections or special runtime requirements. Cloud Run handles health checking, scaling and zone distribution for you, so you skip most of the operational surface described here. MIGs earn their complexity with custom machine types, GPUs or long-running processes.

Written by SkilBrill Training Institute

October 15, 2026 · 10 min read