The API Passed Every Test. Then It Charged the Customer Twice.
The support ticket arrived on Thursday morning.
A customer had been charged twice. The dev team wasn’t panicking yet — these things happen, right? Someone checked the logs. Someone else pulled up the monitoring dashboards. All clear. Then they went to the test results.
Every single API test was green.
So why did a real customer just get billed twice?
If you’ve been around software delivery long enough, this story lands differently. The obvious failures — a 500 error, a timeout that crashes the whole flow — those are almost a gift. They’re loud. They’re findable. The ones that keep us up at night are the quiet ones.
A timeout triggers a retry, and the retry creates a duplicate transaction. A developer changes a field type on a Friday afternoon, and a downstream integration silently dies over the weekend. A webhook arrives 40 seconds late and causes the system to act on stale data. The API did exactly what it was supposed to do. And a production incident happened anyway.
After 20+ years of QA work across financial services, healthcare, retail, and logistics, we’ve seen these enough times to recognize the pattern. Most API disasters are traced back to the same small set of failure points. The problem isn’t that teams skip testing. The problem is that they test the contract — what the API is supposed to do — instead of testing the risk — what happens when reality gets weird.
Here are the six failure modes that show up again and again.
1. The Retry That Became Two Orders
Picture this: a customer clicks “Purchase” on concert tickets. The screen hangs. Did it work? They click again. Meanwhile their browser already sent a retry. Their mobile app might have too. Now there are three nearly identical requests racing toward your API.
Does the system create one order or three?
This is the idempotency question, and it’s one most teams don’t test properly. The answer should always be one — the server should recognize the repeated request and not act on it twice. Most APIs handle this through an idempotency key in the header, but that only works if you’ve verified it holds up under real conditions.
What that looks like in practice: sending the same request twice with the same key, simulating a client retry after a timeout, running parallel retries underload. You are checking whether the response stays consistent and whether the resource gets created exactly once. It sounds simple but most test suites skip it entirely
2. The Field That Changed on a Friday
A developer changes a field from an integer to a string. Totally reasonable refactor. The API still returns 200 OK. Every test passes. They ship it and go home for the weekend.
Monday morning, a downstream integration starts failing. Nobody touched it. Nothing changed on their end. But suddenly nothing works.
This is a schema drift and it’s one of the sneakiest failure modes in API development. The issue isn’t that the API broke but that the API changed in a way that was invisible to everyone watching the normal success signals.
There is a fix for this, and its contract testing: validating every response against a versioned OpenAPI specification on every test run, so the test fails the moment the live response diverges from what it promised — even when the call itself returned 200. Without this, your API contract is a description and with it, it’s actually a constraint.
3. Pagination’s Edge Cases Are Not Boring
Pagination gets written off as the boring part of API design. It is not boring. It is a quiet source of customer pain.
The last page is empty, but the client expected one more record. The total count in the header doesn’t match the number of records that came back. A cursor expires while the user is mid-page. The underlying dataset shifts while someone is paging through results and now records are missing or doubled.
None of these are exotic. All of them happen. And they tend to surface in scenarios like “a customer can’t find the invoice they’re looking for” or “the report is showing the wrong totals.”
Testing pagination properly means going beyond the happy path: empty datasets, datasets that hit exactly on a page boundary, data that changes during iteration, cursors that should and shouldn’t be valid. Each combination exposes something different.
4. Two Users, Same Record, One Millisecond Apart
Two requests hit the same resource at nearly the same moment. Both read the current state. Both modify it. Both write back. One wins. The other overwrites silently.
Both requests returned 200. The API technically worked. One user’s changes are just… gone.
Race conditions are especially dangerous because they tend to pass standard validation easily — each request looks fine in isolation. They only show up under concurrent usage, which means they often slip through testing and land in production.
What you’re testing here is the server’s ability to enforce optimistic locking or conflict resolution, and whether it returns appropriate 409 responses when two requests genuinely clash. For systems built on eventual consistency, there’s a different set of questions — mainly whether the API actually surfaces the consistency boundary instead of hiding it.
5. The Rate Limit Problem Nobody Noticed Until Too Late
The API team documented their rate limits. The client team built in retry logic with exponential backoff. Everyone felt good about it.
Then production traffic came in. The client backed off aggressively at the first sign of a rate limit — and underutilized the API badly enough to cause performance problems. Or it didn’t back off enough and started hammering the server until it got banned.
The bug wasn’t in the rate limit implementation. It wasn’t in the retry logic either. It was in how the two interacted. That’s almost never what gets tested.
Real rate limit testing means running sustained traffic above the threshold and watching whether the server’s retry-after headers and the client’s backoff behavior actually reach a stable equilibrium together. Spoiler: they often don’t, and you find out about it at exactly the wrong time.
6. Webhooks Fail Differently
Request-response APIs fail in ways that are mostly predictable. Webhooks fail in ways that are weird.
Events arrive late. Events arrive twice. Events arrive out of order — the “completed” event shows up before the “processing” event. Sometimes they don’t arrive at all, and nobody’s watching the dead-letter queue.
Signature verification is often implemented loosely enough that it can be bypassed. Replay attacks succeed when there’s no timestamp or nonce validation. A retry mechanism designed to ensure delivery creates an ordering problem that breaks downstream business logic.
A functioning webhook system has to handle all of this gracefully, which means testing has to include: signature verification with valid, invalid, and replayed payloads; out-of-order delivery; idempotent consumer logic; and what happens when the consumer is down, returns a bad response, or times out.
Why the Test Suite Keeps Missing These
Across hundreds of engagements, the gaps tend to come from the same places.
Teams test documented behavior — the happy path described in the spec — rather than building coverage around a map of what the API can actually fail at. Nobody’s charting the risk surface; they’re charting the feature surface.
OpenAPI specs and Postman collections drift from production and nobody enforces the difference. The spec becomes descriptive rather than prescriptive.
Manual test authoring doesn’t scale. A team with hundreds of endpoints can’t hand-write edge case coverage for every failure mode. So they write tests reactively — after an incident, not before.
And real-world edge cases live in real-world data: nulls, Unicode characters, oversized payloads, partially populated objects. Synthetic test data covers the happy path. Getting to the rest requires data that actually looks like what customers generate.
What “Risk-First” API Testing Actually Looks Like
At CelticQA, we start by mapping failure modes before we write a single test. Our QA Maturity Model helps organizations build that map: which failure modes apply to which endpoints, and which tests cover which risks. Coverage becomes something you can reason about rather than just count.
Our Accelerate Automation Program builds API regression suites that actually cover these six failure points without requiring manual authoring at scale.
IV&V — Independent Verification and Validation — gives API integrations an outside perspective, which matters most when the team building the API is also the one evaluating it.
And through QAConnector, we help teams close the traceability gap: auto-generating test cases from the API contract, logging every result with full input-output-version context, and connecting tests to requirements so the coverage story holds up under scrutiny.
If you want to go deeper on how traceability changes the way executives trust QA reporting, we covered that in Trust, Transparency, and QA in AI Systems.
Frequently Asked Questions
What are the most common API failure points?
Six show up repeatedly in production: idempotency failures under retries, schema drift breaking downstream consumers, pagination edge cases, race conditions on shared resources, mismatches between client retry logic and server rate limits, and webhook delivery failures including signature verification and event ordering.
Why do QA teams miss API edge cases?
Most QA programs test the happy path against a contract document, not against a structured map of what the API can actually fail at. The edge cases that cause production incidents — retries, concurrency, schema drift — require risk-based test design, not just contract validation.
What is idempotency testing for APIs?
Idempotency testing verifies that retrying the same API call produces the same result without creating duplicates or side effects. POST endpoints especially need this — a network timeout that triggers a client retry should not create two records server-side.
How do you test for API schema drift?
Schema drift testing compares the live API response against a versioned OpenAPI contract on every test run. The test fails if the response adds, removes, or changes fields in ways the contract doesn’t allow — even when the API call itself returns 200.
What’s the difference between API contract testing and API integration testing?
Contract testing validates that the API matches its specification — schema, status codes, headers. Integration testing validates that the API behaves correctly under realistic use: retries, concurrency, real data, dependencies. Both are necessary. Most teams do the first and skip the second.
The Question Worth Asking
Most teams celebrate when all their tests pass. The teams that actually catch these failure modes ask a different question: What haven’t we tested yet?
That shift — from coverage as a count to coverage as a map of known risks — is where QA stops being a checkbox and starts being useful. Because customers don’t experience your test suite. They experience your software. And production has a way of finding whatever your testing strategy didn’t bother to look at.
When API risk is what’s standing between you and your next release, talk to CelticQA. We’ll map the risk and design the coverage — usually within a single engagement.
