Sunday, 2:47 PM. First support ticket: “can’t log in, is it just me?” It wasn’t just them. But it also wasn’t everybody, and that turned out to be the entire problem.
The Setup
Our stack was boring on purpose. Three app servers behind an nginx load balancer, Cloudflare in front, certbot renewing the Let’s Encrypt certs on the LB box. A nightly rsync copied those certs to the app servers too, because they terminate TLS as well (long story, legacy client, don’t ask). It had run untouched for a year and a half.
The Spiral
The tickets kept coming all afternoon and made no sense. One user refreshed, site loaded fine. Another got a certificate error on the same URL. My laptop worked every time I tested it, which I took as proof the users were confused. (Spoiler: the users were right. They usually are.)
nginx configs on all three nodes, identical. App logs, nothing. I renewed the cert manually, redeployed it to the LB, and the errors kept coming. The LB cert was fresh, so where were browsers getting an expired one?
Still with me? Good, because this is where I gave up on configs and started acting like a user. Instead of one curl, I ran twenty in a loop.
The Discovery
for i in $(seq 1 20); do curl -vsI https://ourapp.example.com 2>&1 | grep -i "expire"; done
Half the responses said valid until December. The other half said it expired Saturday night. Same domain, same minute, alternating. One of the three app servers had quietly stopped receiving the nightly rsync weeks ago (rotated SSH key, permission denied, cron complaints going to a mailbox nobody reads). Two nodes healthy, one dead, and every third handshake a user landed on the dead one.
The Fix
Fixed the rsync key, re-ran the sync to all nodes, then watched the loop until twenty responses in a row agreed on an expiry date. Tickets stopped. Four hours of a Sunday, gone.
The Lesson
- One dead node out of three is worse than three. Total failure is loud. Partial failure leaks users, and each one looks like a flaky client.
- Monitoring from inside the box lies to you. Our checks hit the health endpoint directly, no TLS over the public path, so the broken server passed every check it was ever given.
- Nothing watched the sync job itself. It had been dead for weeks, so the first thing that noticed was the expiry date. Browsers did our alerting for us.
- I still don’t know why the LB kept sending traffic at the stale node instead of failing it out. My guess is an HTTP-only health check. That’s a guess, not a fact, and it bothers me.
What I’d Do Different
- Monitor the real public URL from outside, with full certificate verification, the way users actually arrive.
- Alert at 30, 14, and 7 days before any cert expires. It costs nothing and ends this story early.
- Alert on failed sync jobs too. A cron failing silently for weeks is a server that’s already down and hasn’t said so.
So yeah. One expired cert, on one server, out of three. I set up three new alerts that evening and I’d defend two of them as strictly necessary. The third one stays because it earned it.