The Scene
Two support tickets landed the same morning, eleven minutes apart, saying the same thing. “I got charged twice for my invoice.”
Annual renewals run through our payment provider. A webhook marks the invoice paid, the receipt goes out, done. So I did what any confident engineer does at 9am: opened a case with the provider and quoted timestamps like I was presenting evidence in court. My theory: their API had a rough week and fired double callbacks. I even told their support “we send each payment exactly once.” I want that sentence back.
The Investigation
The provider replied a few hours later with a one-liner. Both charges were real, both came from our API keys, and one was flagged as a retry. A retry. We don’t retry. At least I was pretty sure we didn’t.
So I went digging. Grepped the gateway logs for the invoice id and there they were, two POSTs to the payments endpoint, 400 milliseconds apart, identical bodies. The first request had succeeded on their side. But our HTTP client had a 90 second timeout, their API was slow that afternoon (their status page admitted as much), so our client decided the request had failed and quietly sent it again.
Here’s the part that stung. The retry wasn’t something I wrote. It was a library default, max_retries=2, sitting in the docs the whole time. I had never opened that page.
The Discovery
Turns out “we don’t retry payments” was a feeling, not a config value. The library had been retrying since day one, and because we never sent an
Idempotency-Key, the provider had no way to know that second POST was the same invoice.
The Fix
The fix itself was boring, which is the best kind. We generate a key per invoice and pass it on every create call. The provider dedupes on their side.
key = f"invoice-{invoice.id}"
client.payments.create(payload, idempotency_key=key)Refunded both customers that afternoon, with apology emails I rewrote three times. Their support guy got an apology too. He was polite about it, which somehow made it worse.
The Lesson
- Every write call gets an idempotency key. Especially the ones that move money.
- Read your client library’s defaults. Retries, timeouts, backoff. All of it is configured without you.
- A timeout is not a failure. The request may have landed anyway.
- Blaming the vendor first cost me a day. The answer was in our own logs the entire time.
What I’d Do Different
- Check our own logs before opening a vendor ticket. Twenty minutes of grepping would have saved a support round trip.
- Set retry policy and timeout explicitly, in one place, in code I can find.
- Add an integration test that fires the same payment twice and asserts one charge.
Final Thoughts
Still with me? The annoying part is that nothing here was hard. The library did something reasonable, the provider had a bad afternoon, and my code had no seatbelt. That combination sat in production for two years.
So yeah. Two refunds, one config line, and a support ticket I had to withdraw. The provider guy signed off with “happy to help”, and honestly, he was.