We Killed Our Own Database to See If Login Survived
We had two Keycloak servers. They shared one Postgres database, on one machine, in one container.
That is redundancy for the wrong failure. It survives a server process dying, which is the thing least likely to take you down for long. It does not survive the database going away, which takes both servers with it at once. Two of something in front of one of something else is a queue, not a pair.
What we did not do
We already run a Patroni-managed Postgres pair for a different part of the platform, with automatic promotion. Reusing it looked like the obvious answer and we measured it instead of assuming.
identity server -> existing pair, site A 41.9 ms identity server -> existing pair, site B 52.7 ms identity server -> local container under 1 ms
A single sign-in makes several database round trips. Moving the data there would have added those milliseconds to every authentication by every customer, permanently, in exchange for surviving a failure that has not happened. The existing pair also runs an older Postgres major version than our identity server wanted, so it was not a repoint; it was a downgrade.
We reused the part that was worth reusing — the existing etcd quorum Patroni coordinates through, which is what actually decides who is allowed to be primary — and gave the identity database its own pair, in the same building as the servers that query it, one network hop apart.
Then we stopped the primary
Not a graceful patronictl switchover. We stopped Patroni and killed the Postgres process.
database primary killed standby promoted timeline advanced, read-only flag cleared sign-in still working yes measured interruption about three seconds old primary returns rejoins as a streaming replica, zero lag
The part we are most pleased with is the assertion, not the result
Earlier the same day we learned that our login page renders perfectly with the database stopped, because the configuration behind it is cached. A screenshot of a working login form proves nothing at all.
So "did it survive" was not tested by loading a page. It was tested by sending real credentials and requiring the server to reject them for the right reason. A deliberately wrong password against a reachable database gives an invalid-credentials error, and that answer is only possible if the server actually looked the user up. A database that is gone gives a different error entirely.
Both errors look like failure. Only one of them means the system is healthy. If your failover test asserts "an error appeared" rather than "the expected error appeared", it will pass during an outage.
What this does not protect against, said plainly
A member that stays up and still looks healthy while being broken underneath is not detected and nothing fails over automatically, because from the outside it is answering. Automatic promotion handles a member that stops, not one that lies.
Both members also sit in the same building. Losing that building takes the pair, and no amount of promotion logic helps. We know where that line is and we would rather write it down than let the word "redundant" imply something broader.
Why bother
Every customer signing in to our network product goes through this. Before, a database restart was an outage and a restart of the server dropped every session. Now a member can die and sign-in keeps working.
None of that is a product. It is the floor under the ones we already sell, and the reason we test it by breaking it rather than by reading the documentation.