Our Alerts Woke Me Up 43 Times Last Month — Here's How I Fixed the Noise

Forty-three pages in one month. Yes, I counted. Counting felt more productive than answering them.

The Scene

Here’s the setup. We’re five backend devs. When I joined I was the eager one who wired up Prometheus and Alertmanager over a weekend, copying alert thresholds from a blog post: CPU over 80%, disk over 70%, memory over 85%. It worked great for two weeks.

The Spiral

Then the pages started, and they were never anything. One night it pinged me at 1:40 AM, and half asleep I opened the dashboard: CPU at 81%. The backup job runs every night at 1:30. (You see where this is going.) I silenced it and told myself I’d fix the rule on Monday.

I did not fix it on Monday. Or the Monday after that. Each page got a little easier to dismiss. Still with me? Because here’s the thing: when a real incident finally showed up, a disk filling up on the logs volume, I lumped the 11 PM push in with the usual noise and went to sleep. We lost two hours of logs. Nothing customer-facing, we got lucky, but I sat there the next morning staring at a graph I had trained myself to ignore.

The Discovery

So that day I finally opened Alertmanager and pulled the last thirty days of firing history.

41 of the 43 pages were the nightly backup job tripping a threshold I copied from a blog post. The other two were the disk incident and a test alert I never deleted.

Forty-one. I had been waking up for a cron job. I honestly didn’t know whether to laugh or be embarrassed.

The Fix

The fix took about two hours. First, the CPU alert only watches the app pods now, so the backup box can spike all it wants:

- alert: HighCPU
  expr: sum(rate(container_cpu_usage_seconds_total{pod!~".*backup.*"}[5m])) > 0.85
  for: 10m

Second, severity means something now: warnings go to Slack, only symptoms I’d act on at 3 AM page me:

routes:
  - matchers: [ severity = "warning" ]
    receiver: slack-only

I deleted the leftover test alert and two more I couldn’t explain. Shrugged, shipped it, waited.

The Lesson

An alert you’ve trained yourself to ignore is worse than no alert. After enough fake fires you stop trusting your own phone.
Thresholds go stale. That 80% number fit our traffic when I set it. Every firing is “obviously” the backup job, so nobody notices the drift.
Page for symptoms, not numbers. If I can’t say what I’d do at 3 AM when something fires, it goes to Slack instead.

What I’d Do Different

Look at firing history before writing a single new alert rule. The data was sitting there all along.
No runbook link, no page. That rule has already killed three alerts and I regret nothing.
Calendar reminder to audit alerts monthly. I’ve ignored it twice, which admittedly undercuts this whole post.

External Resources

Google’s Alerting on SLOs chapter. Should have read it before wiring any of this up.
Alertmanager routing docs, for the severity matcher syntax above.
PagerDuty’s incident response docs, whose alert quality section stung a little to read.

Final Thoughts

So yeah. The phone has gone off once in the two weeks since, a real disk warning I fixed in ten minutes at my desk. Turns out the quiet was the feature I never configured.