It was 2 AM when my phone buzzed. Not a notification — an alert. The kind you dread.
Our main API was down. Users couldn’t log in. Payments were failing. And I was the only developer awake.
The First 10 Minutes
My heart rate spiked. I SSH’d into the server, ran some commands, saw nothing obviously wrong. Tried restarting the app. Still nothing. My hands were shaking.
Looking back, I wasted those first 10 minutes trying to fix it alone. I didn’t call my co-founder. I didn’t check the logs properly. I just panicked.
What Actually Fixed It
Finally, I woke up my co-founder. Within 5 minutes of him looking at the logs, we found it — a missing environment variable that got deleted during a deployment. One line in our config, and everything collapsed.
We fixed it in 2 minutes. The whole outage lasted 45 minutes.
What I Learned
- Don’t panic alone. Wake someone up. Two pairs of eyes beat one panicked brain every time.
- Check logs first. I wasted time SSH’ing and running random commands. The answer was always in the logs.
- Have a rollback plan. We didn’t. Now we do.
- Document the fix. We wrote a post-mortem the next morning. Not to blame anyone, but so we never lose 45 minutes to the same problem.
That outage taught me more than any tutorial ever could. If you’re a junior developer: expect to face one. Don’t be hard on yourself when it happens.