Designing Deterministic Tests for AI Agent Outputs — Automatically Detecting Hallucinations and Regressions with eval-gate

by

“Looks Good” Is Not a Test

Traditional software gives us a reassuring property: for a known input, we can usually predict the output.

add(2, 3) -> 5

AI agents break that assumption. Ask an agent to investigate a repository twice and you may receive different wording, different tool calls, or a different ordering of findings. That variability is not automatically a defect. Exploration benefits from flexibility.
The problem begins when the variation affects facts.
If an agent claims:

The authentication middleware is implemented in src/auth/middleware.ts.

then that file should exist. If the agent reports that a project has a budget of $42,000, the underlying record should contain that project and the amount should be reproducible from authoritative data.
A response can sound professional, be well formatted, and still be wrong. Human review often catches obvious failures, but “I read the transcript and it looked reasonable” does not scale. It is also a poor defense against regressions introduced by a new model, tool version, prompt, or code change.
For consequential outputs, the right question is not whether the answer sounds good. It is whether the answer can pass a deterministic gate.

How One Hallucination Reaches Production

Consider a desktop assistant built with Rust and Tauri. It aggregates data from Nextcloud, an internal project system, budgeting tools, and inventory services. An autonomous loop can inspect a repository, update code, retrieve project data, and prepare reports.
One failure looked harmless at first. The agent generated a project summary containing a plausible-looking project_id. The identifier did not exist in the source system, but the surrounding customer name and cost values made the result appear credible.
That invented identifier then moved through several layers:

  1. A funding-plan lookup returned an empty or partial result.
  2. A fallback path filled in missing cost fields.
  3. A budget summary included the generated amount.
  4. A downstream report presented the result as if it came from the customer database.
    Nothing crashed. No exception appeared in the user interface. The error propagated silently because the system validated the shape of the response, but not its relationship to authoritative data.
    This distinction matters. A JSON object can be syntactically valid while being factually invalid. An agent can produce exactly the fields an API expects and still invent their values.
    The safest place to intervene is at the boundary between an agent claim and the evidence that should support it.

The Core Model: Claim, Evidence, Validator, Result

A deterministic evaluation can be designed as a four-part contract:

claim -> evidence -> validator -> PASS or FAIL

A claim is what the agent asserts. Evidence is the authoritative observation available to the system. The validator is a conventional program that compares the two. The result is an unambiguous decision.
Examples:

Claim Evidence Validator
project_id exists Database or API record Exact identifier lookup
Budget total is $42,000 Stored line items Deterministic sum
File exists at a path Filesystem Existence and type check
Code compiles Compiler output Exit status equals zero
Tool returned 17 items Saved tool observation Count and schema comparison
Funding payment is $1,240/month Loan inputs and formula Recalculate with fixed parameters
The principle is simple:

Do not ask an LLM to judge something a machine can prove.
An LLM may be useful for extracting claims from an unstructured response, but the final decision about file existence, identifier validity, arithmetic, schema compliance, or compiler success should belong to deterministic code.

What an eval-gate Actually Does

An eval-gate is not another conversational agent. It is a release boundary.
During execution, the system records both the agent’s actions and the observations returned by tools. The agent may produce a summary such as:

{
  "project_id": "ANS-2026-001",
  "main_cost": 26000000,
  "inventory_items": 14
}

The gate then loads authoritative evidence and evaluates each field:

  • Does ANS-2026-001 exist?
  • Does its current plan contain a main cost of 26,000,000 yen?
  • Does the inventory query return exactly 14 items?
  • Are all required fields present?
  • Did the agent introduce an unknown field or silently replace a missing value?
    A minimal gate might return a machine-readable result:
{
  "status": "FAIL",
  "checks": [
    {
      "name": "project_id_exists",
      "status": "PASS"
    },
    {
      "name": "main_cost_matches_source",
      "status": "FAIL",
      "expected": 26000000,
      "actual": 24500000
    }
  ]
}

The important property is that the same inputs produce the same result. A second run may generate different prose, but it should not change whether a verified amount matches the source.
A practical autonomous pipeline looks like this:

agent -> edit or propose -> build and test -> eval-gate -> accept or reject

If the gate fails, the change is rejected, marked for repair, or sent back for another attempt. The system must not convert a failed check into a successful run merely because the agent provides a persuasive explanation.

Protecting Self-Modifying Agents

This boundary becomes essential when an agent can modify its own code.
In a self-improving workflow, the agent may:

  1. Plan a change.
  2. Edit Rust or frontend files.
  3. Run the application or test suite.
  4. Produce a report.
  5. Publish or deploy the result.
    “It worked in my session” is not a definition of done. The session may have used a different working directory, cached dependency, tool response, model version, or undocumented environment variable.
    A deterministic gate can check the properties that matter independently of the agent’s narrative:
  • The expected files are present.
  • The source compiles.
  • Unit and integration tests pass.
  • Required commands return zero.
  • Generated artifacts have the expected schema.
  • No forbidden files or secrets were modified.
  • Recorded tool results match the final report.
  • External identifiers resolve to real records.
    For a Rust/Tauri project, this might include cargo test, cargo clippy, a production build, and repository-specific smoke checks. The agent can decide how to solve a problem, but it cannot decide whether a compiler error should count as success.
    This is why a strong gate uses zero LLM calls in its release decision. The model can explore, explain, and propose. It should not be the final authority on whether its own work is valid.

Separate Hallucinations from Regressions

Not every failure is the same, and the gate should record the distinction.
A hallucination is an unsupported or fabricated claim. For example, the agent reports a project identifier that cannot be found in the source system.
An unsupported claim may use real data but make an unjustified inference. For example, the source contains a cost estimate, while the agent reports it as a signed contract value.
A regression is a change that breaks behavior that previously passed. For example, a budget calculation was correct yesterday but now omits exterior construction costs after a refactor.
These categories require different metrics:

  • Hallucination rate: unsupported claims divided by claims checked.
  • Validation coverage: claims with an explicit validator divided by total claims.
  • Regression rate: previously passing checks that fail after a change.
  • False-positive rate: valid outputs blocked by an overly strict validator.
  • False-negative rate: invalid outputs that pass.
    Tracking these separately prevents teams from “improving” the system by simply weakening checks.

Determinism Requires Controlled Inputs

A deterministic validator cannot make an unstable environment deterministic by itself.
Pin the inputs that affect the result where practical:

  • Dependency versions.
  • Tool schemas and API fixtures.
  • Database snapshots or controlled test records.
  • Time zones and clock behavior.
  • Locale and numeric formatting.
  • Model and provider versions for reproducible scenarios.
  • Filesystem paths and working directories.
    External services can still be tested through recorded observations or fixtures. The production integration should have separate contract tests, while the core gate should be able to run offline and repeatedly.
    This division also clarifies the role of probabilistic evaluation. An LLM judge may be appropriate for assessing tone, helpfulness, writing quality, or whether an explanation addresses a vague question. Those are semantic qualities with no single exact answer.
    An eval-gate serves a different purpose: it decides whether facts, calculations, schemas, files, and code satisfy explicit contracts.

Conclusion

AI agents do not need identical prose to be reliable. They need verifiable claims and enforceable boundaries.
Use the agent for exploration, synthesis, and flexible problem solving. Use deterministic software to verify identifiers, arithmetic, files, schemas, compiler results, and recorded observations. Store the evidence, define the validator, and make the outcome an explicit PASS or FAIL.
Machines prove. LLMs explore.
Once an agent can touch production code, customer records, budgets, funding plans, or inventory, a polished response is not enough. Put an eval-gate between the agent and the consequences.