Load Balancing
Building Resilient Systems with Health Checks and Failover
Implement active and passive health monitoring to automatically remove failing nodes from your traffic pool. Understand how automated failover ensures 99.9% availability for production-grade applications.
In this article
The Core Problem of Static Infrastructure
In a distributed system, the reliability of your application is only as strong as your ability to detect failure. Many developers initially view a load balancer as a simple traffic distributor that sends requests to a static list of IP addresses. This mental model works perfectly in a laboratory setting but fails immediately in a production environment where networks are unreliable and hardware eventually dies.
Static distribution assumes that every backend server is equally healthy and capable of processing every request it receives. In reality, a server might be running but trapped in a deadlock, or it might have a full disk that prevents it from writing logs or temporary files. When a load balancer is unaware of these internal states, it continues to route users to broken instances, leading to a cascade of failed requests and lost revenue.
Health monitoring transforms a load balancer from a passive router into an intelligent orchestrator. It provides the system with the telemetry needed to differentiate between a server that is merely powered on and a server that is actually ready to do work. Without this distinction, your high-availability strategy is essentially based on hope rather than engineering data.
Implementing robust monitoring allows for the automated removal of failing nodes from the traffic pool before they can impact a significant number of users. This process is the foundation of self-healing infrastructure. By automatically isolating problematic nodes, the system maintains a high success rate even when individual components are experiencing critical issues.
Distinguishing Between Liveness and Readiness
It is vital to understand that a process can be alive without being ready to handle traffic. A service might have started successfully and be listening on a port, but it could still be in the process of loading a large cache or establishing connections to a database. If the load balancer sends traffic during this warm-up period, the application will likely return errors.
Liveness checks determine if the application process is running at all, while readiness checks determine if the application is prepared to fulfill requests. Effective health monitoring strategies utilize both to ensure that traffic is only routed to fully operational instances. This prevents the thundering herd problem where a newly restarted service is immediately overwhelmed by a backlog of requests it cannot yet process.
Proactive Reliability through Active Health Checks
Active health monitoring involves the load balancer periodically sending synthetic probes to the backend servers. These probes are independent of actual user traffic and serve as a heartbeat mechanism. If a server fails to respond to a set number of consecutive probes, the load balancer marks it as unhealthy and stops routing traffic to it.
The most common form of active check is the HTTP probe, where the load balancer makes a GET or HEAD request to a specific endpoint like /health or /status. The application logic at this endpoint should perform a quick internal audit of its critical dependencies. If the database is unreachable or a required microservice is down, the endpoint should return a non-200 status code.
Beyond simple HTTP checks, TCP probes can be used for non-web services. A TCP probe simply attempts to open a socket connection to a specific port. While faster and less resource-intensive, TCP probes are shallow because they only confirm the network stack is responding, not that the application logic is functioning correctly.
1from fastapi import FastAPI, Response, status
2import httpx
3
4app = FastAPI()
5
6async def check_database_connection():
7 # Simulate a database ping to verify connectivity
8 return True
9
10async def check_upstream_service():
11 # Check if a critical external API is reachable
12 try:
13 async with httpx.AsyncClient() as client:
14 response = await client.get("https://api.example.com/ping", timeout=1.0)
15 return response.status_code == 200
16 except Exception:
17 return False
18
19@app.get("/health/ready")
20async def readiness_probe():
21 db_healthy = await check_database_connection()
22 upstream_healthy = await check_upstream_service()
23
24 if db_healthy and upstream_healthy:
25 return {"status": "ready", "components": {"db": "ok", "upstream": "ok"}}
26
27 # Returning 503 tells the load balancer to stop sending traffic
28 return Response(
29 content="Service Unavailable",
30 status_code=status.HTTP_503_SERVICE_UNAVAILABLE
31 )The frequency of these checks must be carefully balanced. Probing too frequently can put unnecessary load on your servers and flood your logs with health check entries. Conversely, probing too infrequently increases the mean time to detection, allowing a failed server to continue receiving and failing user requests for several seconds or even minutes.
Configuring Thresholds and Timeouts
A single failed probe should rarely trigger a failover event. Network glitches are common, and a momentary timeout does not always mean a server is permanently broken. Most load balancers use an unhealthy threshold, which requires a specific number of consecutive failures before the node is removed.
Similarly, a healthy threshold defines how many successful probes are required before a previously failed node is reintroduced to the pool. This prevents flapping, a situation where a marginally stable server keeps entering and leaving the pool, which can lead to unpredictable latency spikes and connection resets for users.
Observing Behavior via Passive Health Monitoring
While active checks are proactive, passive health monitoring is reactive and relies on real user data. This approach, often called outlier detection, involves the load balancer watching the actual responses coming back from backend servers. If a specific server starts returning a high percentage of 5xx errors or experiences extreme latency compared to its peers, it is flagged.
Passive monitoring is powerful because it catches issues that synthetic probes might miss. A health endpoint might report success because the basic dependencies are up, but a specific code path used by actual customers might be triggering a memory leak or a race condition. Passive monitoring observes the true customer experience in real-time.
One of the main challenges of passive monitoring is defining what constitutes a failure. In a busy system, occasional errors are expected due to client-side issues or transient network noise. Engineers must define statistical thresholds, such as an error rate exceeding five percent over a sliding one-minute window, to trigger an eviction.
- Active Checks: Predictable resource usage, identifies failures before users arrive, but can be shallow.
- Passive Checks: Identifies failures based on real traffic, captures complex edge cases, but requires existing user pain to trigger.
- Hybrid Approach: Most production systems use both to provide a comprehensive safety net.
When a node is evicted via passive monitoring, it is typically placed in a quarantine period. During this time, the load balancer stops sending it new requests but might still perform active health checks to see if the server recovers. If the active checks pass after the quarantine expires, the server is gradually reintroduced into the rotation.
Handling Partial Failures
Not all failures are binary. A server might be slow but functional, or it might be failing for one specific customer segment while working for others. Passive monitoring allows for more nuanced traffic management, such as reducing the weight of a slow server rather than removing it entirely.
This concept of graceful degradation ensures that the system remains available even under duress. By shifting the bulk of traffic away from struggling nodes, you prevent a single slow instance from dragging down the average latency of the entire cluster, effectively isolating the performance regression.
Engineering Robust Automated Failover Transitions
Failover is the automated process of rerouting traffic to redundant components when a failure is detected. Achieving 99.9% availability requires this process to be entirely hands-off. In the time it takes for a human operator to receive an alert and log in to a console, thousands of users could have already been impacted.
The transition must be seamless from the perspective of the client. Modern load balancers achieve this by utilizing connection draining or graceful shutdown periods. When a server is marked for removal, the load balancer stops sending new connections but allows existing requests a few seconds to complete their work before fully severing the link.
Effective failover architecture also requires sufficient headroom in the remaining healthy nodes. If you have four servers running at 80% capacity and one fails, the remaining three servers must absorb the extra 25% traffic. If they cannot handle the surge, they may fail in a chain reaction known as a cascading failure.
1/*
2 * Conceptual configuration for an upstream cluster
3 * demonstrating health check and failover logic
4 */
5const upstreamCluster = {
6 name: "api_service_cluster",
7 lb_policy: "ROUND_ROBIN",
8 hosts: [
9 { address: "10.0.1.5", port: 8080 },
10 { address: "10.0.1.6", port: 8080 },
11 { address: "10.0.1.7", port: 8080 }
12 ],
13 health_check: {
14 timeout_ms: 2000,
15 interval_ms: 5000,
16 unhealthy_threshold: 3,
17 healthy_threshold: 2,
18 path: "/health/live"
19 },
20 outlier_detection: {
21 consecutive_5xx: 5,
22 base_ejection_time_ms: 30000,
23 max_ejection_percent: 50
24 }
25};In this configuration, we see both active checks and outlier detection working in tandem. The max_ejection_percent parameter is a critical safety valve. It prevents the load balancer from removing all servers if a global issue occurs, which would otherwise result in a complete outage where no traffic can flow at all.
The Dangers of the Thundering Herd
When a node returns to a healthy state and is reintroduced to the pool, it can be overwhelmed by a sudden influx of traffic. This is known as the thundering herd effect. To mitigate this, load balancers often implement a slow-start or warm-up period, where the weight of the new node is gradually increased over several minutes.
Gradual reintroduction gives the application time to warm its local caches and JIT-compile hot code paths without being crushed by the full production load. This ensures that the recovery of one node doesn't immediately cause it to fail again, creating a cycle of instability that is difficult to debug.
Handling Edge Cases and Operational Pitfalls
One of the most dangerous edge cases in health monitoring is the split-brain scenario. This occurs when the load balancer loses connectivity to the backend nodes, but the backend nodes are actually healthy. If the load balancer incorrectly identifies every node as unhealthy, it might fail open or fail closed depending on the configuration.
Failing open means the load balancer continues to route traffic to all nodes regardless of their health status when it detects that the entire pool is failing. This is often the preferred behavior because if everything is broken, trying to send traffic to the last known state is better than dropping every single connection at the edge.
Another common pitfall is the shallow health check. If your health check only returns a static string without checking the database, a database outage will not trigger a failover. However, if your health check is too deep and performs complex queries, the check itself can become a source of performance degradation and database load.
A health check that is too simple is a lie; a health check that is too complex is a vulnerability. The goal is to verify the critical path of the request without becoming a burden on the system resources.
Finally, always ensure that your health check endpoints are protected from external access. You do not want malicious actors to be able to trigger failovers or gain insights into your internal infrastructure health by probing your status endpoints. Restrict access to these paths to only the internal IP addresses of your load balancing layer.
Testing Your Failover Logic
You cannot trust a failover mechanism that has never been tested under load. Many engineering teams use chaos engineering practices to intentionally terminate instances or inject network latency in production to verify that the health checks and failover logic respond as expected.
Regularly simulating these failures ensures that your timeouts and thresholds are tuned correctly for your specific workload. It also builds confidence in the team that the system can maintain its 99.9% availability target even during the inevitable hardware failures of a modern cloud environment.