Four hours. Two engineers. One bug that only showed up a couple of times a day. And the punchline: the code responsible was mine, written six months ago, with a comment that said “simple wrapper, do not overthink.”
I did not overthink it. That was the whole problem.
The Scene
A Tuesday afternoon, support forwarded us a ticket that nobody wanted to be true: a user was looking at another user’s dashboard. Their name, someone else’s numbers. Someone else’s charts, on their screen, logged in as themselves.
We couldn’t reproduce it, obviously. It happened maybe twice a day across thousands of sessions, and never for us when we tried.
The Spiral
Hour one was logs. Nothing. No errors, no stack traces, just requests happily returning data to the wrong people. Somehow that’s worse than an error.
Hour two we blamed the most recent release and rolled it back. Bug kept happening. Then we blamed WebSockets, because when you have no idea what’s going on, you blame WebSockets. My teammate tore apart the subscription logic for an hour. Nothing. (The WebSockets were innocent, for the record.)
Hour three he added distributed locking around the dashboard query, which felt very responsible, and the bug got more frequent. That’s the moment it clicked. This probably wasn’t a race condition at all. Locking made it worse, which meant whatever we had liked concurrency.
The Discovery
Hour four I finally opened our cache helper. Mine. The one with the confident comment. And there it was:
When the user ID was empty, the cache key fell back to the string
"shared". For every user. Every request.
A user with a glitchy session token would hit the endpoint with an empty ID, prime the cache under "shared", and everyone after them read that same entry until it expired. Fresh dashboards, served fast, to whoever came next.
The fix was one if statement. Four hours.
What I Learned
Here’s the thing I keep thinking about: intermittent bugs are usually data bugs that look like concurrency problems. If locking makes a bug worse, stop locking and start asking what data is being shared.
- A cache key is a contract. “Fall back to something” is not a key strategy, it’s a data leak with extra steps.
- Test with two users. Every integration test we had used one user, because one user is easy and two users is reality.
- Read your own old code with suspicion. Six-months-ago me was tired too, and he also wanted to go home.
What I’d Do Different
- Fail loud on a missing cache key. Never guess one.
- Add a test with two concurrent users to the dashboard suite.
- Log cache key misses so the next version of me finds this in minutes, not hours
So yeah. Four hours for one if statement. It happens. The cache layer is still fast, by the way. Just slightly less clever now.