A customer opens a mobile app, taps “Place order,” and sees an error. Meanwhile, the operations dashboard says the application is healthy. CPU is low, memory is stable, and the main process is still running.
This is one of the most confusing forms of failure in software: a backend can be alive while its API is effectively unavailable. To users, it makes little difference whether a server crashed or merely stopped answering correctly. Their request still failed.
The confusion comes from treating “the backend” as one thing. In practice, an API request travels through a chain of networks, gateways, services, dependencies, policies, and data stores before a useful response reaches the caller.
Understanding that chain changes how teams design monitoring, investigate incidents, and communicate status. It also helps developers avoid a damaging assumption: a green server process does not prove a working API.
🧩 An API Is a Request Path, Not Just a Process
An application programming interface, or API, is a contract for exchanging requests and responses. A client sends a request to a defined endpoint, such as POST /orders, and expects a response with a particular status, shape, and meaning.
For that exchange to work, much more than application code must be running. The request must reach the correct destination, pass security checks, find available capacity, complete its internal work, and return before the caller gives up.
A useful mental model is a delivery route. Seeing a warehouse open does not prove a parcel can be delivered: roads may be closed, labels may be rejected, loading bays may be full, or the receiving building may be inaccessible.
🟢 What “Still Running” Actually Tells You
A running process usually means the operating system has not terminated the program. A container platform may similarly report that a container exists and its main command has not exited.
That is valuable information, but it is a narrow signal. It does not establish that the process is accepting connections, that its request handlers are responsive, or that its dependencies are usable.
Even a successful deployment can leave a running but broken instance behind. A configuration error may only affect a particular route, tenant, region, or authentication flow, while a basic process check remains green.
🩺 Liveness and Readiness Answer Different Questions
Modern deployment platforms often distinguish between liveness and readiness. A liveness check asks whether an instance appears stuck or dead enough to restart. A readiness check asks whether it should receive production traffic now.
Confusing them creates avoidable outages. An application can be alive enough that restarting it repeatedly would not help, yet unready because it has not loaded required configuration, established safe dependency connections, or completed initialization.
A readiness endpoint should reflect the service’s ability to handle the traffic it is advertised to handle. It should not claim success merely because a web framework has started listening on a port.
🌐 DNS Can Send Clients Somewhere Unexpected
Before many clients can call an API, they must translate a hostname into an address through the Domain Name System, or DNS. If that lookup fails, returns stale information, or points at an incorrect target, the backend may never see the request.
DNS changes are especially tricky because caches exist in browsers, operating systems, network resolvers, and client libraries. Different users can receive different answers for a period of time.
Suppose a team moves api.example.test to a new load balancer but misses one record or target. The new backend may be healthy, while some callers continue attempting to reach an old, unavailable destination.
🚪 Load Balancers Can Have a Different View of Health
A load balancer stands between clients and backend instances. It distributes traffic, often performs health checks, and may remove an instance that fails those checks. Its assessment can differ sharply from the platform’s assessment.
If a health check uses the wrong path, host header, port, or expected status code, every healthy application instance might be marked unusable. Clients then receive gateway errors despite the applications running normally.
The opposite is also possible. A shallow load-balancer check can keep sending traffic to instances whose real request paths are failing. Health checks must be deliberately designed, not treated as incidental plumbing.
🔒 TLS Failures Happen Before Application Logic
HTTPS depends on Transport Layer Security, or TLS, to establish an encrypted connection. Certificate expiration, an incomplete certificate chain, an unsupported protocol setting, or a hostname mismatch can prevent that connection from forming.
When TLS fails at an edge proxy or gateway, the application may receive no trace of the request. Its logs can look quiet while users report a complete outage.
Certificate monitoring and controlled renewal processes matter because an otherwise healthy backend cannot compensate for clients that refuse to establish a secure connection.
🛡️ Authentication and Authorization Can Block Good Requests
An API can be reachable and still reject every relevant caller. Authentication verifies who or what made a request; authorization decides whether that identity may perform the requested action.
Expired signing keys, clock skew, a changed token audience, incorrect permissions, or a failed identity-provider call can turn valid user activity into waves of 401 or 403 responses.
These are not necessarily backend crashes. They are contract or policy failures. Yet if an app requires authenticated calls, widespread authorization failure is still an API availability incident from the user’s perspective.
🚦 Rate Limits Can Look Like Random Downtime
Rate limiting protects services from overload by restricting how many requests a caller, account, address, or token can make within a period. It is often essential, but a poorly chosen rule can block legitimate traffic.
A deployment that causes clients to retry aggressively may multiply request volume and trigger limits. Shared network addresses can also make unrelated users appear to be one high-volume caller.
Good API responses make rate-limit failures recognizable, usually with an appropriate status code and retry guidance. Silent connection drops make a policy decision look like unexplained instability.
📦 An API Gateway May Fail Independently
API gateways commonly handle routing, authentication, request transformation, caching, quotas, and observability. This central role makes them useful, but it also means they are an independent component in the request path.
A malformed routing rule can direct requests to a nonexistent upstream. A plugin update can reject headers that clients previously sent. A gateway’s own capacity or configuration store can become the limiting factor.
When diagnosing an outage, ask where the response was generated. A gateway-generated error tells a different story from an error emitted by application code.
🧭 Routing Rules Can Miss the Intended Service
Routing looks simple until versions, path prefixes, hostnames, methods, and regions enter the picture. A rule may match /v1/orders but not /v1/orders/, or allow GET while accidentally excluding POST.
Path rewriting introduces another risk. The gateway might remove a prefix that the application expects, causing a perfectly healthy service to return 404 because it sees the wrong path.
Test routing from the caller’s actual entry point. Testing an internal service address alone cannot prove that public routing, transformations, and policy layers are correct.
🧱 Firewalls and Network Policies Can Quietly Deny Traffic
Firewalls, security groups, and container-network policies decide which systems may communicate. A restrictive rule can be intentional protection, but it can also be an accidental deployment regression.
A common pattern is that the edge can reach the API but the API cannot reach a database, queue, or identity service after a network change. The API process remains up, but requests fail or hang once they need that dependency.
Network denials are often hard to distinguish from a slow dependency unless logs, flow records, or connection-error metrics expose the failed connection attempt.
⏳ Timeouts Are Contracts Between Layers
Every networked layer tends to have a timeout: client, CDN, proxy, gateway, application server, database driver, and downstream service. These limits define how long each participant is willing to wait.
Problems appear when the values conflict. If a gateway gives up after ten seconds but the application continues working for thirty, the client receives an error while the backend may still create an order or charge a card.
Timeouts should be designed as a budget across the request path. Downstream calls need shorter limits than the caller-facing deadline, leaving time to handle failure and return a meaningful result.
🔁 Retries Can Turn a Small Slowdown Into an Outage
A retry is sensible when a failure is temporary, but it adds load precisely when a dependency may already be struggling. Multiple layers retrying independently can amplify traffic dramatically.
For example, a client retries a request, the gateway retries it, and the service retries a database call. One original action can produce many concurrent attempts, consuming connection pools and making recovery harder.
Use bounded retries, exponential backoff, and jitter, which adds small randomness to retry timing. Most importantly, retry only operations that are safe to repeat or protected by an idempotency mechanism.
🧵 Thread and Connection Pool Exhaustion Stops Progress
Servers have finite resources for concurrent work. Thread pools run blocking tasks, connection pools manage access to databases or other services, and file descriptors represent open network connections and files.
An application can remain fully alive while all useful workers wait on slow downstream calls. New requests may queue until they time out, even though CPU usage looks surprisingly low.
Pool exhaustion is a classic reason that “the server is up” and “the API works” diverge. Queue depth, active connections, wait time, and rejected work are often more revealing than a process count.
🗄️ Database Availability Is Not the Same as Database Usability
A database may be running but unable to serve the workload an API needs. It can be overloaded, locked by a long transaction, out of available connections, unable to satisfy a replica-read requirement, or returning an unexpected schema error.
Consider a checkout API whose database accepts simple health queries but stalls on a particular inventory query. A generic database ping remains green, while customers cannot complete purchases.
Dependency checks should be proportionate. Deep checks catch more realistic failures, but running expensive business queries on every health probe can itself add load. Use representative monitoring separately from lightweight instance checks.
📨 Queues and Event Systems Create Delayed Failures
Many APIs acknowledge a request after publishing work to a queue or event stream. The caller may receive a success response even when workers later fail to process the event.
In other designs, the API waits for a broker acknowledgement and fails immediately if the broker is unreachable or permissions have changed. Both patterns can break without terminating the API process.
Track the outcome that matters to users, not only the initial HTTP response. A request accepted for asynchronous processing is not the same as the requested business operation being completed.
🧾 Contract Changes Break Clients Without Breaking Servers
An API contract includes endpoint paths, fields, data types, validation rules, error formats, and semantic expectations. A server can answer every request successfully by its own definition while breaking clients that rely on the previous contract.
Changing a field from a number to a string, making an optional field required, or altering pagination behavior may cause client parsing failures or incorrect behavior. The backend’s error rate might remain near zero.
Backward compatibility is a product decision as well as an engineering decision. Versioning, deprecation periods, consumer contract tests, and clear change communication reduce this category of breakage.
🧠 A Successful Status Code Can Still Mean Failure
HTTP status codes describe the protocol-level outcome, not automatically the business outcome. An API can return 200 OK with an empty result caused by a faulty filter, stale cache, or incorrectly applied access rule.
Likewise, a service might return a structured error inside a successful response for legacy reasons. Clients must understand the contract, and monitoring must look beyond status-code counts where business correctness matters.
This is sometimes called a semantic failure: the transport worked, but the answer was wrong, incomplete, or misleading. It is often harder to detect than a clear outage.
🧊 Caches Can Serve Old, Missing, or Incorrect Answers
Caches improve speed and reduce backend load by reusing prior results. They also introduce another stateful system whose keys, expiration rules, invalidation behavior, and permissions must be correct.
A cache key that omits the user or locale can leak the wrong response across contexts. A failed invalidation can make updated data appear missing. An overly aggressive cache rule can store an error response.
When investigating API behavior, determine whether the response came from the application or a cache. Headers, tracing, and targeted cache-bypass tests can help distinguish the two.
🕰️ Clock Skew Creates Surprisingly Real API Failures
Distributed systems rely on time for token expiration, request signatures, certificate validation, scheduled jobs, cache expiration, and event ordering. If clocks differ significantly across machines, valid requests can appear expired or not yet valid.
Clock problems are easy to overlook because each server may look healthy in isolation. They tend to surface as intermittent authentication or signature errors across only some nodes.
Reliable time synchronization is operational hygiene, not a minor detail. When failures mention timestamps, expiration, or signatures, compare clocks across the affected components.
🌍 Regional and Dependency Failures Produce Partial Outages
Large systems are often deployed across regions, availability zones, or separate clusters. A global status indicator can conceal a serious local problem if only one region is failing.
Users may be routed by geography, network conditions, account placement, or a session affinity rule. Consequently, one support report can be accurate even when an engineer’s test succeeds from another location.
Break down telemetry by region, version, endpoint, customer segment, and dependency. Aggregate averages are useful for trends, but they can hide the exact slice experiencing failure.
📊 Metrics Need to Describe the User Journey
Host metrics such as CPU, memory, and restarts answer whether infrastructure appears stressed. API metrics such as request rate, latency, error rate, and saturation reveal more about service behavior.
Neither group alone is enough. A service may return fast errors with excellent latency, or slowly degrade while error rates remain low until callers hit their deadlines.
| Signal | Useful question | Blind spot |
|---|---|---|
| Process alive | Has the program exited? | Whether it can do useful work |
| Readiness | Should this instance receive traffic? | Whether real user flows succeed |
| HTTP success rate | Are requests receiving expected status classes? | Incorrect but successful responses |
| Journey check | Can a client complete a key action? | May cover only selected paths |
The goal is not a single perfect metric. It is a set of signals that represent infrastructure health, request-path health, and meaningful user outcomes.
🔎 Logs, Metrics, and Traces Solve Different Parts of the Puzzle
Metrics show patterns: rising latency, an increase in rejected requests, or a saturated connection pool. Logs provide detailed events and error messages. Distributed traces connect one request across gateways, services, and dependencies.
A trace is especially helpful when the API is running but slow. It can show that most of a request’s time was spent waiting for a particular database query or downstream HTTP call.
Correlation IDs make this investigation practical. Pass one request identifier through each layer so a report from a client can be matched to gateway records, application logs, and dependency calls.
🧪 Synthetic Checks Test What Internal Health Checks Miss
A synthetic check is an automated request that behaves like an external client. It can resolve the public hostname, establish TLS, authenticate with a test identity, call an endpoint, and validate a small expected result.
These checks catch failures at the edge of the system: DNS, certificates, routing, and broken public authentication. They are complementary to internal probes, not replacements for them.
Keep synthetic transactions safe and inexpensive. A test that creates real orders or repeatedly modifies production data without careful cleanup creates a new operational risk.
🧭 Start Incident Triage at the Boundary
During an API incident, begin with the caller-visible symptom. What endpoint, method, region, identity type, response code, and approximate time are affected? A concrete failing request is more useful than “the API is down.”
Then move inward through the path: DNS and TLS, edge or gateway, load balancer, application, and dependencies. This approach avoids spending an hour reading application logs when requests never reached the application.
- Reproduce or inspect a representative failing request.
- Identify the layer that generated the response or timeout.
- Compare healthy and failing slices: region, version, route, tenant, and identity.
- Check recent changes, then validate the suspected cause with evidence.
- Mitigate safely before pursuing a complete root-cause explanation.
🧯 Mitigation Is Not the Same as Root Cause
Restoring service may mean rolling back a gateway rule, shifting traffic away from a region, increasing a constrained pool, or disabling a faulty feature flag. These actions can be appropriate even before the full explanation is known.
But a mitigation should not be mistaken for a root cause. Restarting an API might clear exhausted resources temporarily while leaving a slow database query, leaked connection, or retry storm untouched.
Record the timeline: observed symptoms, relevant changes, actions taken, and their effects. This separates evidence from assumptions and makes later review much more useful.
🧰 Design Health Endpoints With Care
Health endpoints are operational interfaces and deserve design attention. A minimal liveness endpoint should be cheap and unlikely to fail because of a temporary dependency issue. A readiness endpoint should reflect whether routing traffic is responsible.
Some teams also expose a richer diagnostic endpoint for protected internal use. It can report dependency states without becoming the public signal used to restart every instance.
Do not make health checks so deep that a brief third-party slowdown removes all healthy capacity at once. The right depth depends on whether an instance can still provide useful, safe responses during that dependency failure.
🛠️ Build APIs for Graceful Dependency Failure
Not every dependency failure should produce the same response. If a recommendation service is unavailable, an API may safely return a page without recommendations. If payment authorization is unavailable, it should fail clearly rather than pretend an order is complete.
Useful resilience patterns include circuit breakers, which temporarily stop calls to a repeatedly failing dependency, bulkheads that isolate resource pools, and carefully bounded queues. These techniques reduce cascading failures, but they add behavior that must be tested and observed.
Idempotency keys are particularly valuable for operations such as payments and order creation. They let a client safely repeat a request after a timeout without unintentionally performing the action twice.
📋 Test Failure Paths Before Production Does
Happy-path tests prove that components work under normal conditions. They do not show what happens when DNS is wrong, a token expires, a database becomes slow, a queue rejects publishes, or only one region loses access to a dependency.
Use staging environments, controlled fault injection, and game days to practice realistic failures. The purpose is not to create chaos; it is to verify assumptions about timeouts, fallbacks, alerts, runbooks, and recovery.
Include contract tests between API producers and consumers. They help catch incompatible changes before deployment, where a technically successful release could otherwise become a client-facing breakage.
👥 Clear Ownership Prevents Long, Circular Incidents
Request paths often cross teams: networking owns DNS, a platform team owns ingress, an identity team owns token verification, and product teams own business services. Without clear ownership, each group can reasonably say its local component is healthy.
Document service boundaries, dependency owners, escalation routes, and the expected behavior of critical APIs. Shared dashboards and traces reduce the handoff friction during an incident.
The goal is not to assign blame to a layer. It is to restore the end-to-end experience and learn which assumptions allowed a local green signal to conceal a user-facing failure.
🎯 The Core Principle: Availability Is End-to-End
An API is available only when its intended callers can reach it, authenticate where required, receive a timely response, and use that response to complete the expected action. Process survival is one prerequisite, not the definition of availability.
This principle also explains why no single health check is enough. Teams need layered checks: local process health, traffic readiness, dependency behavior, public reachability, and representative business journeys.
When an API breaks while the backend is still running, the right question is not “Why didn’t the server crash?” It is “Which part of the request contract stopped being true?” That question leads engineers toward evidence rather than false reassurance.
A running backend is only one link in the chain; reliable APIs are measured by successful end-to-end outcomes. Design, observe, and troubleshoot the whole path—from the caller’s first lookup to the final meaningful result. 🧩🔎🚀
