A month ago I gave an AI agent write access to our deploy pipeline. Real credentials. CI logs, error dashboards, the rollback button, even our Slack. I did this on purpose, which is the part my coworkers still bring up at lunch.
The pitch I made to myself was simple: deploys fail at 3 AM, humans are bad at 3 AM, maybe the agent isn’t. One month test. Worst case, we turn it off.
The Setup
The rules were boring on purpose. The agent could read build logs, watch the error rate panel in Grafana, and trigger a rollback. It could not touch infrastructure, run migrations, or approve its own changes. Every action went into an audit log that we (mostly) reviewed on Fridays.
The first week it did basically nothing. It watched deploys and stayed quiet. I started to think we had wasted a weekend setting it up.
Where It Actually Helped
In week two it caught something before any of us did. A migration was shipping with a row count that didn’t match the staging numbers, the agent flagged the log line as “inconsistent”, and held the deploy. It was right. We would have caught it too, but probably an hour later, after the first ticket came in.
Rollback time dropped as well. It used to be 10-15 minutes, mostly because the on-call person had to wake up, find the laptop, and remember the runbook. The agent does it in about 90 seconds. I have mixed feelings about how good that felt.
What Broke
Then a marketing campaign sent real traffic our way and things got dumber.
- It rolled back a healthy deploy. Error rates “looked weird” during the traffic spike. They weren’t errors. They were users. It reverted a perfectly good release at the worst possible moment and we spent the evening untangling it.
- It fought itself. One night it rolled back a deploy, the old version threw errors the new one had already fixed, so it rolled forward again. Then back. I woke up to 40 Slack messages and a rollback loop that had been running since 4 AM.
- It spammed us. Fourteen incidents in one night, all the same root cause. Nobody reads the fourteenth Slack ping with the same respect as the first.
What I Learned
Here’s the thing I keep coming back to: the agent was only as good as the blast radius we gave it. None of the failures were the model being clever and wrong. They were us being vague about when a rollback is even allowed.
- One automatic rollback, then a human. Always. No exceptions, even if the second failure looks identical to the first.
- Dedupe alerts. Fourteen pings for one bug is how you train humans to ignore the channel.
- Cooldowns. 15 minutes minimum between automatic rollbacks. It sounds harsh until you watch it flap.
What I’d Do Different
- Run it in suggest-only mode for the first two weeks instead of trusting my own hype
- Hard cooldown between rollbacks from day one
- Caps on actions per hour. Yes, even for the smart agent.
It’s still running, by the way. We never turned the agent off. It just lost its rollback-without-asking privilege. That’s a human job again, at least until it earns some trust back.