What makes for a good health check and why should every service have one?

I’ve seen it all during my years working on distributed systems: responses showing healthy but not really, ‘200’ status code returned forever even when broken, cached data masquerading as the real-time status, authenticated endpoints that monitoring tools can’t reach, and so much more. This guide teaches engineers the value and importance of health checks, what they must contain (and why), and what absolutely should not be included (with the reasons we’ve all learned through painful lessons).

The Two Truths of Health Checks:

  • Every service in a distributed system exists in one of two states: healthy or degraded.
  • Being able to assess these two states quickly is the foundation to operating a system at scale.


What I think a great health check should contain

Here is an example of what I think is a good starting point. (But you should read the whole article to understand why this is actually a bad example.) If every service had something like this I can confidently say life would be significantly better. 😊

Request: 
GET https://some.internal.service.com/services/my-service/health

Response: 
200 OK (or 5xx in this case)
{
    time: "2026-01-18T20:30:30Z",
    status: "FAILED",
    reason: "Storage not accessible on last check: no free space available",
    lastCacheRefresh: "2026-01-18T20:30:00Z",
    name: "my-service",
    deploymentId: "e4cf5efb3ed143fb8bc456e0d32d0617",
    region: "ap-southeast-2",
    dependencies: {
        database: "PASS",
        storage: "FAILED",
        messageBus: "PASS"
    }
}

Of course, each system might look different and each organization might decide certain things should be added or removed. However, this is a great starting point for you to refine. Let’s dive into each piece to understand why it is important and how to use it.

1. It’s easy to memorize

The health check URL should be easy to memorize. For obvious reasons - at 2 a.m. why would you want to make your life harder than it needs to be. Don’t do this:

GET https://some.internal.service.com/services/my-service/e4cf5efb3ed143fb8bc456e0d32d0617/health

2. Provide two ways to explicitly state the status

HTTP Status Code (‘200’ / ‘5xx’) and a Magic String

Return ‘200 OK’ when systems are functional; return appropriate error codes (like ‘503 Service Unavailable’, though not ‘404’) for failures. It is also really nice to return a magic string like ‘SUCCESS’ or ‘FAILED’.

Monitoring tools and load balancers watch these signals to trigger alerts, circuit breakers, or automatic failover without guessing what’s wrong from watching metrics data alone. And, unfortunately, not all network devices are created equally. I learned a long time ago that setting the status code and a magic string in the response made health checks much more useful and flexible.

  • network devices can sometimes rewrite the status code of a response or strip response headers (WAFs are notorious for this)
  • some devices can’t reliably parse status codes - they just give the script the entire raw HTTP response text to search on so a magic string makes this much more robust
  • some scripts can become defective over time and having 2 ways of checking health allows for a higher degree of certainty at scale

Note in the example above, my magic string is FAILED or SUCCESS. The dependencies say PASS because I only want the word SUCCESS to show up in the response if the system is healthy.

3. Prove that the response is timely and from the right source

Include the exact time of check (ISO8601 in UTC). If your endpoint doesn’t provide this within a certain window (say 60 seconds past expected) it is unhealthy or ambiguous, regardless of HTTP status code being ‘200’.

Latency spikes often accompany real problems. A fresh timestamp proves the service processed a request end-to-end recently. Without it, you’re stuck trusting blindly-issued success codes that may be stale data or cached responses.

4. Include some useful information about the environment

Respond with a current build hash, deployment ID, and region identifier—static identifiers to help operators and automation know exactly what is running without needing separate inventory systems.

When a bug appears at midnight, you want your alerts saying “service-v2.3.4 failed in us-east-1”, not just “something broke”. Many times the questions you need to answer are:

  • Did this fail because of a recent deployment?
  • Is only a certain region affected?
  • What exactly has failed in the network chain? Did the service fail or some device in front of it or some dependency behind it?

When you work on a system that can have multiple nodes in multiple regions, lots of things can go wrong. If you refresh the health check multiple times and see that the deployment id is not constant, you might have a situation where a deployment was only partially successful and there are now mutliple versions of your app running. The point is to add some minimal and very lightweight environment data to help narrow the cause of failure quickly.

5. Safe Failure Reasons and Dependency Status

Briefly state what is failing: dependency timeouts, disk pressure, connection refused, but never stack traces or sensitive internal data.

Admittedly, this part of the health check takes some discernment. Is this internally facing? If so, you probably can put more details in there. If not, maybe just return an error trace ID that the team knows how to look up in the logs.

{
    time: "2026-01-18T20:30:30Z",
    status: "FAILED",
    lastCacheRefresh: "2026-01-18T20:30:00Z",
    name: "my-service",
    deploymentId: "e4cf5efb3ed143fb8bc456e0d32d0617",
    region: "ap-southeast-2",
    errorTraceId: "baefffdf-3bdb-406f-8a2a-7e25c587188e"
}


What Health Checks MUST NOT Contain

1. Authentication Required

Health checks must be accessible to the operators without credentials. If your endpoint requires auth tokens, bearer claims or mutual TLS before saying whether it is alive, you have defeated its primary purpose entirely. If you decide that the health check endpoints cannot be public for some reason, block them at the edge but do not try to make them authenticated endpoints.

2. Fake Success Signals

Don’t let your health checks show ‘200’ while failing silently in the background when something critical like a database connection pool or message bus is broken.

Health checks are designed so that load balancers and monitoring tools can make failover decisions based on real-time observations. If they see success but your service fails with errors when handling user requests anyway you’ve added confusion instead of value, causing operators to wonder why alerts fire even after checking the health status endpoint repeatedly without understanding something deeper is broken (like missing dependencies).

3. Exposing Sensitive Information - Use a Trace ID Instead

Your response must only contain safe operational information that humans need to know when troubleshooting, never secrets like keys, passwords, internal paths or proprietary data structures customers shouldn’t see. A stack trace with an internal message can be disclose too much, so always control the response appropriately. You should treat health check responses as public even though they may not be.

4. Heavy Computation or Side Effects

Avoid expensive operations like database queries that involve joins and aggregations — or worse, actual workloads unrelated to readiness checks. Keep your endpoint lightweight enough for monitoring tools to call thousands of times per hour.

“SELECT 1” will make sure a database connection can be established and add almost no load to a database.

5. Be careful what you call a failure

In the first example above, I used a full disk volume as a signal to fail the health check. At first pass, that seems reasonable, but consider if it really is. If you run a volumn out of space, can the app still work in some capacity? Sure, you might want to alert someone that the volume is full, and maybe the side effect is that a customer can’t upload some new image or a process can’t fully run. But, in this case, a failed health check likely should and will kick all instances out of the load balancer and then customers will have no functionality at all.

Best Practices for Engineering Teams

Add it as your first endpoint

Create a health check endpoint before building anything else in your app. Make it part of every service’s codebase from day one. It is a great way to test the network connectivity of your app, start the patterns of your new app and not require any type of business logic you have to worry about. It’s a great first steel-thread piece of functionality.

Use them regularly

I guarantee that if your team knows about and uses the system health checks regularly you will elevate the readiness of your team to respond to issues. Health checks are the foundation that allows systems to become resilient and self-heal but they are just as useful for engineers to check if something is running, how well it is running, etc. And, they are just as useful in local and non-prod environments as they are in production.

Adapt across system types

Whether you’re building APIs, message passing apps with queues and topics, or systems that work on different protocols and technologies, consider how to apply these health check patterns. They will look different, the concepts are the same - what can I build that is lightweight, exercises the infrastructure but requires no business impact or functionality and gives me an accurate assessment if a component is healthy or degraded. It is not just for HTTP-based applications.