# Snapshot testing for AI agents

> Snapshot testing for AI agents is the practice of recording a known-good agent run — the ordered tool calls it made, with their arguments and results — as a committed baseline, then comparing every later run against that baseline and failing loudly when they diverge. It borrows the discipline of UI snapshot testing, where a rendered component is checked against a stored reference and any mismatch fails the test; for agents, the artifact under test is the tool-call trace rather than markup. A failing snapshot names the step that changed and what changed about it, which turns “the agent behaves differently now” from a vibe into an exit code — the property that lets an agent workflow gate CI the way ordinary code does.

Markdown twin of https://www.reelier.com/learn/agent-snapshot-testing — one entry in Reelier's glossary of agent-replay terms. The blockquote above is the canonical definition; quote it with the link.

## Where the term comes from

The pattern predates agents. In Jest's canonical formulation, a snapshot test renders a UI component, compares it to a reference snapshot stored alongside the test, and fails on any mismatch: either the change is unexpected, or the reference needs updating. What matters is the discipline: unreviewed change fails the build.

Agents gave the pattern a new artifact. An agent run's observable behavior is its tool-call sequence — typed JSON requests and typed JSON responses, in order — and that sequence is data you can store and compare. The phrasing is now standard across the category: agentsnap, an independent zero-dependency library, describes itself as “Snapshot tests for AI agents” and frames the loop the same way — record tool-call traces, diff against baselines, fail CI on regressions. Reelier did not coin the pattern. The tools differ on what happens at check time.

## Two ways to take the snapshot: observe, or replay

Trace-comparison tools run the agent live on every test and diff the fresh trace against the stored baseline. agentsnap, per its own README, monitors tool invocations during a live run and compares the trace structurally; nothing re-executes. This tests the model's decision-making (same task, same calls?) and bills a model session per check, because a model is in the loop every time.

Replay-based snapshot testing makes the baseline itself executable. Reelier compiles the recorded run into a `SKILL.md` with an assertion on every step that recorded a clean result; `reelier run` re-executes the recorded calls against the live systems with no model in the loop (at the default `--max-level 0` an LLM client is never even constructed), and each run writes a receipt to `.reelier/runs/`. `reelier diff <skill>` then compares the last two runs in that record: SAME or DRIFTED, per step, exit 0 or 1.

The two answer different questions. If you are testing the model — a prompt change, a model upgrade — you need a live run to snapshot. If you are testing the workflow's deterministic core on a schedule, replay gives the same regression signal at 0 tokens per check.

## What counts as a changed snapshot

UI snapshot testing earned a reputation for brittleness: snapshot the whole output and every timestamp breaks the build. The agent version has the same trap: a workflow that fetches live data returns different bytes on every honest run, so a bytewise snapshot fails forever.

Reelier's diff dodges this by comparing structure and outcomes, never bound data. Drift is: a step's outcome changed (passed to failed), a step was added or removed, or a step that used to run clean needed a different escalation level to pass. Timing is never drift, and same steps with fresh data is not drift — this week's npm download count replacing last week's changes the receipt, not the verdict.

Assertions carry the same policy into each step. The `registry-latest` portfolio skill asserts `json.version matches /^\d+\./` — a shape, not a pinned number — so publishing a new release updates the bound value on the receipt without failing the snapshot.

## The limits, stated before you hit them

Judgment is not snapshot-testable. Triage, synthesis, prose: those steps need a model every run, and a snapshot of last week's judgment is not a test of this week's. In Reelier, model help during replay is opt-in and BYOK, in levels (deterministic L0 default, then L1/L2), and the default never asks for a key.

Writes are the sharp edge of running snapshots on a schedule. Replay is read-only by default: recorded write steps do not re-fire unless you pass `--allow-writes`. In the measured corpus behind Reelier's launch writing, only 1.1% of scanned sessions were read-only end-to-end, so for most real workflows the write policy is most of the safety story. Decide it before the replay goes into cron.

Capture is bounded by what the platform exposes. `reelier scan` reads Claude Code's transcript history natively; Cursor keeps its history in a SQLite database, so there is no history scan for it — capture there goes through the recording proxy (`reelier mcp --wrap`) while the job runs once as usual.

## A real receipt

The output below is a real snapshot check, not a mockup: the `registry-latest` portfolio skill replayed twice against the live npm registry, then `reelier diff` comparing the two runs. No model was constructed at any point, and the exit code — 0 here, 1 on drift — is the value a CI job gates on.

```
$ npx -y reelier@latest run registry-latest.skill.md
✓ Step 1 — Latest dist-tag from the registry [passed] 276ms

PASSED: 1/1 steps ok, 0 failed, 276ms total

$ npx -y reelier@latest run registry-latest.skill.md
✓ Step 1 — Latest dist-tag from the registry [passed] 153ms

PASSED: 1/1 steps ok, 0 failed, 153ms total

$ npx -y reelier@latest diff registry-latest
  = Step 1 "Latest dist-tag from the registry": passed, unchanged.

✓ SAME — 1 step(s), every outcome and escalation level identical to the baseline.
```

Run by the author on 2026-07-21 in a scratch directory: `examples/portfolio/registry-latest.skill.md` from seldonframe/reelier, replayed twice with the published CLI (`npx -y reelier@latest`, resolved to `reelier@0.13.0`), then diffed. `reelier diff` exited 0.

## FAQ

**Is snapshot testing for agents the same as running evals?**

No. A snapshot test checks exactness against one recorded known-good run: binary verdict, no model, no rubric. Evals score output quality across a dataset, and the judge is often a model itself. They compose: snapshot the deterministic core of the workflow, and spend eval effort on the judgment steps a snapshot cannot cover.

**The diff failed because the upstream changed on purpose. Now what?**

Update the baseline, the same move as updating a UI snapshot. When the contract change is small, edit the failing assertion in the `SKILL.md` — it is markdown in your repo. When the workflow itself changed, run the job once with a real agent and re-record: Reelier's compiler builds steps only from a run that actually happened, never from instruction text.

## Related

- [Agent regression testing — the wider practice a snapshot gate belongs to](https://www.reelier.com/learn/agent-regression-testing)
- [Agent drift — the failure mode the diff is there to catch](https://www.reelier.com/learn/tool-call-drift)
- [Deterministic replay — the mechanism that makes the baseline executable](https://www.reelier.com/learn/deterministic-replay)
- [From the blog: the cron + replay + exit-1 recipe, with the committed baseline explained](https://www.reelier.com/blog/scheduled-agents-zero-token)
- [Set it up on Claude Code — scan your history, freeze a run, diff it forever](https://www.reelier.com/for/claude-code)

## Sources

- [Jest: Snapshot Testing — the origin of the render-compare-fail discipline](https://jestjs.io/docs/snapshot-testing)
- [agentsnap — independent trace-comparison snapshot tests for agents (the observe variant)](https://github.com/MukundaKatta/agentsnap)
- [seldonframe/reelier — the open-source (AGPL-3.0) replay implementation this page describes](https://github.com/seldonframe/reelier)

Other terms: [Deterministic replay](https://www.reelier.com/learn/deterministic-replay) · [Agent drift](https://www.reelier.com/learn/tool-call-drift) · [Run receipt](https://www.reelier.com/learn/run-receipt) · [Level-0 (zero-token) replay](https://www.reelier.com/learn/level-0-replay) · [Session-to-skill (skill compilation)](https://www.reelier.com/learn/skill-compilation) · [Recording MCP tool calls](https://www.reelier.com/learn/mcp-recording) · [Regression testing for AI agents](https://www.reelier.com/learn/agent-regression-testing) · [Idempotent write gating](https://www.reelier.com/learn/idempotent-write-gating) · [Assertion grammar for agent outputs](https://www.reelier.com/learn/assertion-grammar) · [Agent memory vs deterministic replay](https://www.reelier.com/learn/agent-memory-vs-replay) · [Headless agent](https://www.reelier.com/learn/headless-agent)