The 2 A.M. Call That Wasn’t a Bug
At 2 a.m., an on-call engineer receives an alert: the final agent in a production workflow is rejecting input from the planner.
The pipeline worked in staging. Agent A extracted entities from a request, Agent B enriched them, and Agent C formatted the result for an API client. Nothing obvious changed in the business logic. Yet Agent C expected the date 2026-09-04 and received September 4th, 2026.
Both values mean the same thing. Neither is inherently wrong. But one satisfies the downstream contract and the other does not.
This kind of failure is rarely classified as a model hallucination. It is more subtle and often more dangerous: the model still produces fluent, plausible output, but the output’s structure gradually changes. A field is renamed, an array becomes an object, a required property disappears, or a value changes from a machine-readable format to a conversational one.
By the time the problem becomes visible, the original prompt change or model upgrade may be several deployments in the past.
What “Hallucination Drift” Really Means
Traditional hallucination describes invented or unsupported content: a fabricated citation, an imaginary customer, or a false numerical claim. Hallucination drift describes a different failure mode.
The model’s output remains plausible to a human while drifting away from the interface expected by software.
There are two common forms:
- Structural drift: The output no longer matches the expected schema. A date uses the wrong format, a key is missing, or a nested object moves to a different level.
- Semantic drift: The output remains structurally valid but changes meaning. For example,
priority: "high"becomespriority: "urgent"even though the schema permits both, or a currency field contains Japanese yen where the downstream service assumes US dollars.
The first form is straightforward to detect with automated validation. The second requires stronger domain rules, cross-checks, or human review. Contract testing cannot prove that every statement is true, but it can prevent a large class of silent failures before they spread through the pipeline.
Every Handoff Is an Implicit Contract
A multi-agent system is not one model answering one question. It is a chain of producers and consumers:
User request
-> Orchestrator
-> Planner
-> Tool arguments
-> Tool result
-> Memory retrieval and summarization
-> Final response
Each arrow is an interface. The producer creates data, and the consumer makes assumptions about that data.
Even when the interface is described only in a prompt, it is still a contract. The planner may promise to emit a list of executable steps. A tool wrapper may expect an object containing project_id and limit. A memory component may return records with timestamps and confidence scores. The final agent may require citations in a specific array.
The risk is that an LLM often participates on both sides of the boundary. It may generate the data, parse it, summarize it, or decide how to repair it. Model versions, sampling settings, prompt edits, retrieved examples, and tool descriptions can all influence the resulting shape.
If a pipeline contains 12 handoffs and each handoff has a 1% chance of producing an incompatible output during a release, the probability that at least one handoff fails is approximately 11.4%. Independent probabilities are not a perfect model of production behavior, but the lesson is useful: reliability decreases as implicit interfaces accumulate.
Contract Tests Verify the Shape
A contract test checks whether a component’s output satisfies the interface expected by its consumer.
For an AI agent, that means validating more than whether the response is non-empty. A useful contract might specify:
- Required keys and their types
- Permitted enum values
- Date, currency, and identifier formats
- Array length limits
- Maximum string lengths
- Whether additional properties are allowed
- Relationships between fields
Consider a planner contract:
{
"type": "object",
"required": ["steps", "priority"],
"properties": {
"priority": {
"type": "string",
"enum": ["low", "normal", "high"]
},
"steps": {
"type": "array",
"minItems": 1,
"maxItems": 8,
"items": {
"type": "object",
"required": ["tool", "arguments"],
"properties": {
"tool": { "type": "string" },
"arguments": { "type": "object" }
},
"additionalProperties": false
}
}
},
"additionalProperties": false
}
The test does not ask whether the plan is wise. It asks whether the plan can safely cross the boundary.
In TypeScript, a runtime validator such as Zod can enforce the same contract:
const Plan = z.object({
priority: z.enum(["low", "normal", "high"]),
steps: z.array(z.object({
tool: z.string(),
arguments: z.record(z.unknown())
})).min(1).max(8)
});
Python teams can use Pydantic. Language-independent systems can use JSON Schema with validators such as Ajv. The specific tool matters less than validating at runtime, at the exact point where data changes hands.
Putting Contract Tests in CI
A contract suite should combine static fixtures with representative live model outputs.
Static fixtures are deterministic and fast. Store examples for normal cases, empty results, multilingual input, long requests, tool errors, and ambiguous instructions. Run every fixture through the same parser and validator used in production.
A practical CI pipeline might contain four stages:
- Schema checks: Confirm that every contract is syntactically valid and that producers and consumers agree on required fields.
- Fixture tests: Run 50 to 100 deterministic payloads through each handoff.
- Model smoke tests: Run a small, pinned set of prompts against the candidate model configuration.
- Compatibility tests: Compare the current output against the previous release’s contract and record intentional changes.
Do not rely exclusively on snapshots of complete model responses. Snapshot tests are brittle because harmless wording changes create noisy diffs. Extract the structured payload first, then assert its contract. For text responses, test stable markers such as citation objects, action lists, or metadata rather than every sentence.
A useful failure report should identify:
handoff: planner -> tool_executor
path: steps[2].arguments.project_id
expected: string matching ^ANS-[0-9]{4}-[0-9]{3}$
received: null
model: agent-planner-v7
prompt_version: planner-2026-08-19
This turns an overnight mystery into a specific compatibility failure.
Model smoke tests should be pinned wherever possible. Record the model identifier, system prompt version, temperature, tool definitions, and schema version. If any of these change, the contract suite should run before deployment. A passing test does not guarantee correctness, but an unexplained schema change should never reach production unnoticed.
When the Contract Breaks
Validation is only half the design. A robust pipeline also defines what happens after a contract failure.
The first response should be local and bounded. Do not allow malformed data to travel downstream in the hope that another agent will understand it.
A reasonable recovery sequence is:
- Parse and validate the output.
- If validation fails, send a repair request containing the validation errors and the original output.
- Require the model to return only the corrected structured object.
- Validate again.
- After one or two failed attempts, use a deterministic fallback or escalate.
The repair prompt should be constrained:
The object failed validation:
- steps[0].arguments.project_id must be a string
- priority must be one of low, normal, high
Return only a corrected JSON object. Do not add explanations.
Structured output modes and provider-supported tool calling can reduce failures, but they are not substitutes for application-side validation. Providers may guarantee syntactic JSON while your application requires a particular date format, enum, or cross-field relationship.
Fallbacks should be explicit. A failed planner might fall back to a safe single-step workflow. A failed extraction stage might mark the request for human review. A failed final formatter should not silently return a partial answer that appears complete.
Every recovery event should be logged with a correlation ID, contract version, validation error, model version, and retry count. Track metrics such as contract-failure rate, repair-success rate, and fallback frequency. If repairs rise from 0.8% to 4.5% after a model rollout, the pipeline is already signaling drift even if users have not filed complaints.
The Pipeline Rots Quietly
AI systems change continuously. Prompts evolve, models are upgraded, tool descriptions are rewritten, and retrieved context varies from request to request. Treating agent handoffs as informal suggestions makes structural drift inevitable.
Contract testing creates a hard boundary around that uncertainty. It does not determine whether an agent’s reasoning is correct, and it cannot eliminate semantic mistakes. It does something more foundational: it ensures that every component receives data in a form it can safely process.
The next time a model changes a date from 2026-09-04 to September 4th, 2026, the failure should happen in CI, not at 2 a.m. in production.
Your pipeline will still evolve. Contract tests make sure it evolves without silently rotting.