Here at Grootan, Keycloak deployments are our bread and butter.

We maintain various versions of Keycloak, from 1 to latest, Wildfly to Quarkus, been there, done that. We manage for a diverse clientele, supporting millions of users across various applications. We pride ourselves on being experts in this realm, ensuring smooth and secure access management for everyone. But even the most seasoned Keycloak warriors can be surprised. This is the story of how a seemingly routine patch turned into an unexpected debugging odyssey, reminding us that even the most familiar territory can hold unforeseen challenges.

The Patch

Our Mission was simple, backport a fix from a newer Keycloak version to an older one used by few clients. Routine work for a team like ours, right? Wrong. This "what if" scenario played out in the most unexpected way.We fired up our test systems, everything seemed to be normal and started to deploy in prod. Our deployments stuttered. Pods went crashing down with error messages about DB connection timeouts. Our carefully crafted house of cards, supporting millions of users, went down like a stack of cards caught in a tornado. Panic threatened to cloud our judgment, but years of experience steeled our nerves.

Into the rabbit hole

With millions of users are stake, there is no time to waste. We dive head first into the storm testing the limits of our knowledge. Our initial suspect were about connection pool configurations and database resources. But as the minutes turn into hours and the issue persists, we realize that we've entered uncharted territory. This is a challenge unlike any we've faced before

If and only If

The culprit remained stubbornly hidden. Refusing to accept defeat, we cast a wider net, venturing into the labyrinthine code of the backported patch itself. And there it was, lurking in the shadows – an innocuous-looking "if" statement. This seemingly harmless piece of code, recently patched in the newer Keycloak version, was the mastermind behind the chaos. It caused delays in loading offline tokens during startup, leading to those pesky connection timeouts. An "if" statement? We couldn't believe it! A sneaky little "if" statement in the code turned out to be our unexpected villain, causing everything to go haywire.

Fix, Celebrate and Contribute

Relief washed over us. A swift tweak to the “IF” statement streamlined the application loading and voila! Our applications roared back to life, restoring access to our realm. But our journey doesn't end with the resolution of the issue – it's just the beginning. Our responsibility extended beyond our immediate clients. Recognizing the value of our experience and insights, we make a commitment to contribute back to the Keycloak community. So, we submitted a pull request (PR) to the Keycloak project, proposing the fix and backporting it officially. The Keycloak team acknowledged the issue and promptly merged our PR.

Conclusion

This saga served as a valuable reminder, even for seasoned warriors like us:

• Backports: Backports can be time-saving, but thorough testing is crucial to avoid unforeseen consequences.

• Think Beyond the Obvious: Debugging often requires looking beyond the usual suspects and exploring unexpected areas.

• Open Source: Contributing to open-source projects not only helps the community but also strengthens the software for everyone.

So next time you face a bug, remember, after all, even the smallest "if" statement can have unforeseen consequences.

References:

https://github.com/keycloak/keycloak/issues/24295