Your Health Check Is Probably Lying to You
We moved our Keycloak identity provider onto two machines so that one of them dying would stop mattering. Part of that is a health check: the load balancer asks each node whether it is well, and stops sending traffic to one that says no.
We pointed the check at the OpenID discovery endpoint, /.well-known/openid-configuration. It is the obvious choice. It is the document every client fetches first, it returns JSON, and if the server is broken it will not serve it.
That last sentence is wrong, and it took stopping a database to find out.
The test
On an isolated copy of production, running the same image and the same configuration, we stopped the database container outright. Not a slow database. Not a flaky network. Stopped.
discovery /.well-known/openid-configuration 200, 6489 bytes /realms/<realm> 200, 604 bytes JWKS /protocol/openid-connect/certs 200, 2941 bytes the login page 200, 8180 bytes, password field ACTUAL AUTHENTICATION 400 -- nobody can log in
Every read returned 200. The discovery document was byte-identical to the healthy one. The login page rendered a complete form, with a password field, exactly as a customer would see it.
And nobody could log in.
Why
Realm configuration lives in Keycloak's in-memory Infinispan cache. Once it is loaded, serving it does not touch the database at all. Only an actual authentication does — looking up the user, checking the credential, recording the attempt.
So the health check was not broken. It was answering a different question than the one we thought we were asking. It was reporting "this process is alive and holds a cached copy of its configuration", which was true, while the service was completely unable to do its job.
Worse: the endpoint is faster with the database gone, because a cache hit beats a query. A latency threshold would not have caught it either.
What we use now
Keycloak's management port exposes /health/ready, which carries an explicit “database connections” check. With the database stopped it flips to 503 in about thirteen milliseconds, and recovers in about twenty-one when the database returns.
The general form of the fix is not "use this endpoint". It is: a health check must perform a transaction of the same kind the service exists to perform. A cached read is not a transaction. If the check never touches the backing store, it is measuring the cache.
The same mistake, one layer down
Having moved the check onto the management port, we immediately made the opposite error.
The proxy checked port 9000 while sending traffic to port 8080. On one machine the health port was reachable and the traffic port was bound to loopback only. The check succeeded, the node was scored healthy, and every single request routed to it returned 502.
The check was fine. The service was fine. They were pointed at different places.
So the second rule, which we now write on anything that health-checks anything: if the check probes a different port, host or protocol than real traffic, it can report healthy for a path that cannot carry a request. The test that catches it is cheap — break the traffic path, leave the check path working, and confirm the check goes red. If it stays green, it is measuring the wrong object.
How to check yours in five minutes
- Stop the database your service depends on. Not the service. The database.
- Ask your health endpoint how it is.
- If it says 200, it is not a health check. It is a liveness check wearing the wrong label.
- Then break the traffic path only, and confirm the check notices.
We run a public IPv4 network and the operational tooling on top of it, and we have now been caught by both halves of this in a single day. It is not an exotic failure. It is the default one.