Tuesday afternoon. I’m feeling good. Deploy went smooth, tests passed, coffee in hand. Then the alerts start rolling in.
The Problem
Users can’t login. Not all users, but enough to trigger panic. Error logs show something about JWT tokens. Fine. I’ve seen this before.
I open my terminal. Check the logs. Nothing obvious. I dig into the auth service. Code looks fine. I check the database. Connections are healthy. I check Redis. Sessions are storing correctly.
Three hours pass. I’m staring at the same 50 lines of code I’ve been staring at for the last hour. My brain is mush.
The Fix
My teammate asks, “Hey, did you check the environment variables?”
I hadn’t.
Turns out someone deployed to staging with production credentials. Token signing was using the wrong secret. Users hitting staging were getting tokens that production couldn’t validate.
Thirty seconds to fix. Three hours to find it.
What I Learned
- Assume nothing. The basics are usually where the problem lives.
- Take breaks. I was staring at the code so hard I couldn’t see what was missing.
- Ask for help earlier. Another pair of eyes saves time.
- Log everything. If we had logged which environment was being used, I’d have found it in 5 minutes.
Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. — Brian Kernighan
The embarrassing part? This isn’t my first time. It won’t be my last either. But that’s the job. You don’t get good at debugging. You just get faster at feeling stupid.