Half our users saw certificate errors all Sunday. Monitoring stayed green, my laptop worked every time, and the culprit was one rsync job that had been dead for weeks.
Category: Systems & Architecture
Our Alerts Woke Me Up 43 Times Last Month — Here’s How I Fixed the Noise
Forty-three pages in one month. Yes, I counted. Counting felt more productive than answering them. The Scene Here’s the setup. We’re five backend devs. When I joined I was the eager one who wired up Prometheus and Alertmanager over a weekend, copying alert thresholds from a blog post: CPU over 80%, disk over 70%, memory…