The Problem Nobody Talks About Until It Breaks Production

The Problem Nobody Talks About Until It Breaks Production

You have shipped an AI agent that calls a dozen tools: search APIs, databases, email providers, payment processors, and deployment systems. In development, every call succeeds.
Then, at 2 AM, a tool call times out halfway through.
Should the agent retry? If it does, will it double-charge the customer? Did the email provider accept the message before the timeout? Which deployment steps completed before the process crashed?
These questions are not edge cases. A single user request such as “create an invoice, update inventory, and email the customer” may trigger three to seven external operations, each with different latency, reliability, and side effects. In production, transient failures commonly affect roughly 0.5% to 2% of calls, while timeout rates may range from 0.3% to 1.2%, depending on the provider and network.
Naive retry logic, such as “retry three times immediately,” can turn temporary failures into permanent business errors. Robust agent design treats retries as a state-management problem, not merely a network feature.

Why Agent Retries Are Different

Traditional microservices usually have deterministic retry logic. The service knows which endpoint failed, what request was sent, and often what state the operation left behind.
AI agents introduce additional uncertainty. They make decisions, call external tools, observe incomplete results, and then decide what to do next. A retry may therefore be influenced by a changed prompt context, a different model decision, or a partial result that the agent does not fully understand.
The central problem is simple:

A timeout tells you that the client did not receive a result. It does not tell you that the operation did not happen.
Consider three tool categories:

• A web search is usually safe to repeat. The main cost is extra latency or API usage.
• Sending an email may be unsafe to repeat. The provider could have accepted the message before the client timed out.
• A deployment workflow may leave an ambiguous state. The server may exist, DNS may be updated, and the final status record may be missing.
  The correct retry strategy must depend on the tool’s side effects and its ability to report state.

The Three Design Goals

I design agent retries around three goals:

1. Safety: Repeating a call must not create unintended duplicate effects.
2. Liveness: A temporary failure must not leave the workflow stuck forever.
3. Correctness: After partial failure, the system must converge on a known and valid state.
   Idempotency primarily protects safety. Timeouts and retry budgets protect liveness. Durable state tracking and recovery procedures protect correctness.

Idempotency: Make Retries Safe

An operation is idempotent when executing it multiple times produces the same final effect as executing it once. Reading a record is generally idempotent. Creating a payment is not, unless the payment provider supports idempotency keys.
Every side-effecting tool should accept a stable operation identifier. For example:

idempotency_key = agent_run_id + ":" + tool_call_id

The tool or gateway stores the result associated with that key. If the same request arrives again, it returns the original result instead of performing the side effect again.
A payment request might look conceptually like this:

{
  "order_id": "ORD-4812",
  "amount": 24900,
  "currency": "USD",
  "idempotency_key": "run-91f2:call-07"
}

If the client times out and retries, the payment service can return the existing transaction for run-91f2:call-07. It will not create a second charge.
Idempotency keys must be stable across retries but unique across genuinely different operations. Generating a new random key for every attempt defeats the entire mechanism. Conversely, reusing one key for unrelated orders can suppress valid work.
For email, idempotency is more complicated because many providers do not guarantee duplicate suppression. Store an outbound message record before dispatching:

message_id: MSG-2048
status: pending
recipient: [email protected]

The sender then transitions the record through states such as pending, sent, and unknown. A retry must inspect this state and use provider-level message identifiers or a deduplication layer where possible.

Timeouts: Protect Liveness Without Creating False Certainty

Timeouts are necessary, but they should not be interpreted as failures of the business operation. They are failures of observation.
Use separate timeout budgets for each stage:

• Connection timeout: 2 seconds
• Provider response timeout: 15 seconds
• Overall tool deadline: 30 seconds
• Agent workflow deadline: 2 minutes
  The exact values depend on the tool. A database lookup may need 3 seconds, while a document-generation service may reasonably need 60 seconds.
  Retries should be bounded and use exponential backoff with jitter. A practical schedule might be:

attempt 1: immediately
attempt 2: after 1.2 seconds
attempt 3: after 3.5 seconds
attempt 4: after 8 seconds

Random jitter prevents many agents from retrying simultaneously after a shared outage. Add a maximum retry count and a total deadline. Without those limits, an agent can spend several minutes repeating a call while the user waits indefinitely.
Most importantly, classify the timeout result explicitly. Instead of returning only failed, use states such as:

• succeeded
• failed_before_execution
• failed_after_execution
• unknown
• requires_reconciliation
  The unknown state is essential. It prevents the agent from confidently claiming that an operation failed when the provider may have completed it.

Partial Failure Recovery

Multi-step workflows are not automatically transactional. If an agent creates an invoice, updates inventory, and sends an email, the second operation may fail after the first succeeds.
A useful pattern is a durable workflow record:

workflow: invoice-4812
step 1: create invoice    succeeded
step 2: reserve inventory unknown
step 3: send email        pending

Write the state after every meaningful transition, not only at the end. If the agent crashes, a recovery worker can resume from this record rather than asking the model to reconstruct history.
Recovery should use reconciliation before compensation. For an ambiguous inventory reservation, query the inventory system for the reservation status. Do not blindly reserve again. For an ambiguous deployment, inspect the infrastructure provider and compare the actual state with the desired state.
When compensation is necessary, make it a deliberate operation. If an invoice was created but inventory cannot be reserved, the system might void the invoice, release related resources, or route the case to a human. Compensation itself needs idempotency keys and durable status tracking.
Avoid asking the language model to perform low-level rollback reasoning alone. The model can choose a recovery policy, but deterministic code should enforce allowed transitions, verify provider state, and prevent duplicate side effects.

Classify Tools Before Adding Retries

A simple tool policy table is often more valuable than a universal retry wrapper:

Tool type	Default retry policy
Read-only search	Retry automatically with backoff
Database read	Retry transient errors
Record creation	Require idempotency key
Payment or transfer	Provider idempotency plus reconciliation
Email or notification	Message deduplication and delivery status
Deployment	Query actual state before retrying
Destructive action	Require explicit confirmation or a compensating workflow
Also distinguish error classes. Retry network resets, rate limits, and selected server errors. Do not retry validation errors, permission failures, malformed requests, or explicit business-rule rejections.

Observability Is Part of Recovery

A retry system is incomplete if operators cannot explain what happened. Log a correlation ID, agent run ID, tool call ID, attempt number, idempotency key, timeout category, and final state.
Useful metrics include:

• Retry rate by tool
• Unknown-result rate
• Duplicate-prevention count
• Time spent in reconciliation
• Workflows requiring human intervention
• Success rate after each retry attempt
  For example, a tool with a 1% timeout rate may appear healthy if 99.9% of timed-out calls are safely reconciled. The same timeout rate is dangerous if each timeout creates a duplicate payment.

Implementation Checklist

Before enabling retries for an agent tool, ask:

1. Is the operation read-only, idempotent, compensatable, or irreversible?
2. Can the request carry a stable idempotency key?
3. What does a timeout mean, and can the result be unknown?
4. How will the system query the provider after an ambiguous outcome?
5. Where is workflow state persisted?
6. What is the maximum retry count and total deadline?
7. Which failures should never be retried?
8. Can an operator understand and repair the workflow?
   Reliable AI agents are not built by retrying everything. They are built by making uncertainty explicit, preventing duplicate effects, bounding execution, and recording enough state to recover from partial failure. Once those foundations exist, retries become a controlled safety mechanism rather than a production gamble.