Snapshot testing for AI agents
Snapshot testing for AI agents is the practice of recording a known-good agent run — the ordered tool calls it made, with their arguments and results — as a committed baseline, then comparing every later run against that baseline and failing loudly when they diverge. It borrows the discipline of UI snapshot testing, where a rendered component is checked against a stored reference and any mismatch fails the test; for agents, the artifact under test is the tool-call trace rather than markup. A failing snapshot names the step that changed and what changed about it, which turns “the agent behaves differently now” from a vibe into an exit code — the property that lets an agent workflow gate CI the way ordinary code does.
Where the term comes from
The pattern predates agents. In Jest's canonical formulation, a snapshot test renders a UI component, compares it to a reference snapshot stored alongside the test, and fails on any mismatch: either the change is unexpected, or the reference needs updating. What matters is the discipline: unreviewed change fails the build.
Agents gave the pattern a new artifact. An agent run's observable behavior is its tool-call sequence — typed JSON requests and typed JSON responses, in order — and that sequence is data you can store and compare. The phrasing is now standard across the category: agentsnap, an independent zero-dependency library, describes itself as “Snapshot tests for AI agents” and frames the loop the same way — record tool-call traces, diff against baselines, fail CI on regressions. Reelier did not coin the pattern. The tools differ on what happens at check time.
Two ways to take the snapshot: observe, or replay
Trace-comparison tools run the agent live on every test and diff the fresh trace against the stored baseline. agentsnap, per its own README, monitors tool invocations during a live run and compares the trace structurally; nothing re-executes. This tests the model's decision-making (same task, same calls?) and bills a model session per check, because a model is in the loop every time.
Replay-based snapshot testing makes the baseline itself executable. Reelier compiles the recorded run into a SKILL.md with an assertion on every step that recorded a clean result; reelier run re-executes the recorded calls against the live systems with no model in the loop (at the default --max-level 0 an LLM client is never even constructed), and each run writes a receipt to .reelier/runs/. reelier diff <skill> then compares the last two runs in that record: SAME or DRIFTED, per step, exit 0 or 1.
The two answer different questions. If you are testing the model — a prompt change, a model upgrade — you need a live run to snapshot. If you are testing the workflow's deterministic core on a schedule, replay gives the same regression signal at 0 tokens per check.
What counts as a changed snapshot
UI snapshot testing earned a reputation for brittleness: snapshot the whole output and every timestamp breaks the build. The agent version has the same trap: a workflow that fetches live data returns different bytes on every honest run, so a bytewise snapshot fails forever.
Reelier's diff dodges this by comparing structure and outcomes, never bound data. Drift is: a step's outcome changed (passed to failed), a step was added or removed, or a step that used to run clean needed a different escalation level to pass. Timing is never drift, and same steps with fresh data is not drift — this week's npm download count replacing last week's changes the receipt, not the verdict.
Assertions carry the same policy into each step. The registry-latest portfolio skill asserts json.version matches /^\d+\./ — a shape, not a pinned number — so publishing a new release updates the bound value on the receipt without failing the snapshot.
The limits, stated before you hit them
Judgment is not snapshot-testable. Triage, synthesis, prose: those steps need a model every run, and a snapshot of last week's judgment is not a test of this week's. In Reelier, model help during replay is opt-in and BYOK, in levels (deterministic L0 default, then L1/L2), and the default never asks for a key.
Writes are the sharp edge of running snapshots on a schedule. Replay is read-only by default: recorded write steps do not re-fire unless you pass --allow-writes. In the measured corpus behind Reelier's launch writing, only 1.1% of scanned sessions were read-only end-to-end, so for most real workflows the write policy is most of the safety story. Decide it before the replay goes into cron.
Capture is bounded by what the platform exposes. reelier scan reads Claude Code's transcript history natively; Cursor keeps its history in a SQLite database, so there is no history scan for it — capture there goes through the recording proxy (reelier mcp --wrap) while the job runs once as usual.
A real receipt
The output below is a real snapshot check, not a mockup: the registry-latest portfolio skill replayed twice against the live npm registry, then reelier diff comparing the two runs. No model was constructed at any point, and the exit code — 0 here, 1 on drift — is the value a CI job gates on.
$ npx -y reelier@latest run registry-latest.skill.md ✓ Step 1 — Latest dist-tag from the registry [passed] 276ms PASSED: 1/1 steps ok, 0 failed, 276ms total $ npx -y reelier@latest run registry-latest.skill.md ✓ Step 1 — Latest dist-tag from the registry [passed] 153ms PASSED: 1/1 steps ok, 0 failed, 153ms total $ npx -y reelier@latest diff registry-latest = Step 1 "Latest dist-tag from the registry": passed, unchanged. ✓ SAME — 1 step(s), every outcome and escalation level identical to the baseline.
Run by the author on 2026-07-21 in a scratch directory: examples/portfolio/registry-latest.skill.md from seldonframe/reelier, replayed twice with the published CLI (npx -y reelier@latest, resolved to reelier@0.13.0), then diffed. reelier diff exited 0.
FAQ
Is snapshot testing for agents the same as running evals?
No. A snapshot test checks exactness against one recorded known-good run: binary verdict, no model, no rubric. Evals score output quality across a dataset, and the judge is often a model itself. They compose: snapshot the deterministic core of the workflow, and spend eval effort on the judgment steps a snapshot cannot cover.
The diff failed because the upstream changed on purpose. Now what?
Update the baseline, the same move as updating a UI snapshot. When the contract change is small, edit the failing assertion in the SKILL.md — it is markdown in your repo. When the workflow itself changed, run the job once with a real agent and re-record: Reelier's compiler builds steps only from a run that actually happened, never from instruction text.