Regression testing for AI agents
Regression testing for AI agents is the practice of re-verifying an agent's established behavior after something changes — a model version, a prompt edit, a tool schema, an upstream API — so a job that worked last week is proven to still work now instead of assumed to. The durable form of the test is a recorded baseline: capture the tool calls from a run that worked, re-execute them with an assertion on every step, and compare each new run against the baseline so any divergence fails a named step instead of being absorbed by fresh model output. The preposition carries the meaning: this term covers testing your agent. The mirror-image practice — using AI agents to run regression tests on conventional software — is sold as agentic regression testing, and it is a different product category.
Two meanings share one keyword
Search this phrase and you find two industries talking past each other. Tricentis and the test-automation vendors mean agentic regression testing: autonomous AI agents that select, run, and maintain regression suites for ordinary software. In that meaning the agent is the tester. This page is about the opposite direction — the agent is the thing under test. Comet's Opik team uses the same preposition this page does, regression testing for AI agents, for that direction.
If you came here to have an AI run your existing test suite, the sources below include the other meaning's explainer. If you came here because your agent did the job right on Tuesday and you want proof it still does on Friday, keep reading.
What regresses in an agent
Conventional software regresses when someone edits the code. An agent regresses when anything in its stack moves, and most of the stack is not your code: the provider swaps the model behind the same API name, a prompt tweak shifts how the plan comes out, a tool renames a field, an upstream API changes shape. None of these throw a compile error. The agent reasons around the change and hands you output that looks fine, which is the failure mode a regression test exists to catch: silent adaptation instead of a red build.
So the test target splits in two. The judgment layer — did the summary land the tone, did the triage pick the right ticket — needs evals: datasets, plain-English assertions, a model as judge. Opik's test suites are that lane, and Reelier does not compete in it. The deterministic core — the tool calls the agent makes to do the job — is the lane replay covers, and it is where regressions are checkable with no model in the loop.
The recorded baseline, step by step
The baseline comes from a run that happened. Where the platform keeps a scannable history, reelier scan reads it — Claude Code's ~/.claude/projects transcripts are the native format. Where there is none (Cursor keeps its history in a SQLite database, so it has no history scan), the reelier mcp --wrap recording proxy captures every call while the job runs once. Either way the recording compiles into a SKILL.md with one assertion per checkable step, in a small grammar: status, type checks, regex matches, contains, numeric comparisons. Small on purpose — those are the checks that survive expected variation, so a fresh timestamp passes and a changed schema fails.
The regression test is then two commands. reelier run my-job.skill.md re-executes the recorded calls — at the default --max-level 0 an LLM client is never constructed — and writes a receipt to .reelier/runs/. reelier diff my-job compares the last two runs in that file and prints a per-step verdict: SAME when every outcome and escalation level matches the baseline, DRIFTED when a step changed outcome, appeared, disappeared, or needed a different escalation level to reach the same outcome. Exit code 0 or 1, so a cron or CI job can gate on it.
Note what is not drift: fresh data through the same steps. The diff compares structure, outcomes, and healing level, never bound values. This week's download count differing from last week's is the system working; a changed outcome or a vanished step is the regression.
Where the method stops
Replay is read-only by default: recorded write steps do not re-fire unless you pass --allow-writes. That rule earns its keep in a regression suite, which by definition runs when nobody is watching. It also names the method's hardest limit — in the measured corpus behind Reelier's launch writing, only about 1.1% of scanned sessions were read-only end-to-end. A regression suite for a job that writes needs an explicit decision per write step: idempotent writes replayed on purpose, destructive ones never.
The other boundary is judgment. A green diff proves the deterministic core still behaves; it says nothing about whether the model's reasoning got worse. Model help during replay is opt-in and BYOK, in levels — L0, the default, never calls a model; L1 and L2 add bounded healing for drifted steps. A regression practice that covers both layers pairs replay for the tool-call core with eval suites for the judgment on top.
A real receipt
The smallest possible regression test, run for this page: the read-only registry-latest portfolio skill from the Reelier repo, replayed twice with the published CLI, then diffed against the baseline. The second replay and the diff verdict below are verbatim.
npx -y reelier@latest run registry-latest.skill.md ✓ Step 1 — Latest dist-tag from the registry [passed] 164ms PASSED: 1/1 steps ok, 0 failed, 164ms total npx -y reelier@latest diff registry-latest = Step 1 "Latest dist-tag from the registry": passed, unchanged. ✓ SAME — 1 step(s), every outcome and escalation level identical to the baseline.
Run by the author on 2026-07-21: examples/portfolio/registry-latest.skill.md copied from seldonframe/reelier into a scratch directory, replayed twice with npx -y reelier@latest (resolved to reelier@0.13.0), then diffed — output verbatim, exit code 0. Read-only: each replay is one HTTP GET to the npm registry.
FAQ
How is this different from running evals?
Evals score quality against a dataset or a judge model and cost tokens per run. A regression test pins behavior that already worked and re-checks it mechanically: at level 0 a Reelier replay spends 0 LLM tokens and returns pass/fail per step. They answer different questions — use evals for the judgment layer, replay for the deterministic core.
Does the regression suite need an LLM key?
Not at the default. reelier run at --max-level 0 never constructs an LLM client, so CI needs no model key to replay and diff. Credentials for the systems the recorded calls hit still apply, and model escalation (L1/L2) is opt-in BYOK, for when you want a drifted step healed rather than only reported.