Assertion grammar for agent outputs
An assertion grammar is the small, fixed language a replay tool uses to check an AI agent's outputs mechanically: pass/fail predicates evaluated against the typed responses the agent's tool calls return, never against its prose. Reelier's grammar has six assertion forms — status checks, substring checks, JSON type and presence checks, regex matches, length bounds, and scalar comparisons — plus two bind forms that extract fresh values onto the run receipt. Because agent outputs vary between runs, the grammar pins properties (the API answered 200, the version looks like semver, the hits field is an array) rather than exact values, and an expression outside the grammar errors loudly instead of passing as a silent no-op.
Why exact-match assertions break
Every guide to testing AI agents runs into the same wall: run the same job twice and the transcript differs, so string equality fails on the second run. That is the model's variance. There is a second source that survives even after you remove the model: the world's. A download count changes weekly. A version string goes stale the day of a release. An assertion that pins yesterday's value fails tomorrow for reasons that are not bugs.
Reelier deals with the two sources differently. The model's variance is removed by construction: a Level-0 replay re-executes recorded tool calls with no model in the loop, so assertions run against typed observations — status, headers, body — and there is no fresh prose to vary. The world's variance is what the grammar is shaped around: assert what must hold, bind what moves. json.downloads is number outlives any particular count; json.version matches /^\d+\./ holds for every release that will ever ship; version = json.version puts the fresh value on the receipt without pinning it.
This is property-based testing's oldest move — check properties, never a single remembered example — applied to agent outputs. Reelier did not invent it; the grammar makes it the path of least resistance.
The grammar, and what the compiler writes for you
Six assertion forms: status == 200 (and !=); body contains "text" (and not contains); json.<path> is array (also set, number, string, boolean, null); json.<path> matches /regex/; json.<path> length >= 1 (and <=, >, <); and scalar comparison json.<path> == <value> (with != and ordering forms, which compare numbers only). Two bind forms: name = json.<path> and name = body match /regex/, which extracts the first capture group. Anything else raises a parse error that lands on the receipt — an assertion can never rot into a silent no-op.
You mostly do not write these by hand. reelier from-session and reelier compile follow a minimal-assertion policy: status == 200 on each recorded step that succeeded, plus json.<path> is set on each result value a later step consumes. Steps with no derivable assertion stay assertion-less and get listed as open questions instead of being papered over — and at run time a step with zero assertions reports unchecked, never passed.
Failure is specific and terminal. A failing assert prints the observed value (failed: got "0.13.0"), every failing assert on the step lands on the receipt, the run stops there with later steps marked skipped, and the process exits 1. Write steps get gated before assertions even run: replay is read-only by default, so a recorded write does not re-fire without --allow-writes, and the refusal shows up as a named failure rather than a quiet skip.
Where deterministic assertions stop — and where LLM-as-judge fits
The grammar checks structure and properties of typed outputs. It cannot grade meaning: whether a summary is faithful, whether an answer helped, whether the tone fits. That is evaluation territory, and the standard tool there is LLM-as-judge — a strong model grading another model's output. The canonical study (MT-Bench and Chatbot Arena) measured GPT-4 judges agreeing with human preferences over 80% of the time, matching human-to-human agreement, while also documenting position, verbosity, and self-enhancement biases. A judge is a model call: it costs tokens per check and its verdicts carry model variance. Use a judge to grade judgment; use a deterministic grammar to gate plumbing.
Reelier draws that line as an escalation ladder. At the default --max-level 0, only the grammar runs and an LLM client is never constructed. When an assertion drifts, model help is opt-in and BYOK: Level 1 lets a model propose a patched assert/bind set for the same already-captured observation — nothing re-executes, and the patch only counts if the deterministic evaluator passes it. Level 2 may additionally re-execute one read or idempotent-write step with patched args; destructive steps never auto-rerun. A successful heal is written back to the skill file, so the next run is deterministic again. The model proposes; the grammar disposes.
A real receipt
Two steps, one live endpoint — the npm registry's latest dist-tag for the reelier package. Step 1 asserts the shape (json.version matches /^\d+\./) and binds the fresh value; step 2 pins the exact string the endpoint served when it was verified earlier that day (json.version == "0.12.1"). A release shipped in between. The pin fails with the observed value on the receipt; the shape assertion passes untouched.
npx -y reelier@latest run shape-vs-pin.skill.md ✓ Step 1 — Shape assertion: latest looks like semver [passed] 175ms ✗ Step 2 — Exact pin: latest equals the version verified at recording time [failed] 50ms - json.version == "0.12.1" failed: got "0.13.0" FAILED: 1/2 steps ok, 1 failed, 225ms total
Produced by running the published CLI (npx -y reelier@latest, resolved to reelier@0.13.0) on 2026-07-21 against the live registry: a two-step scratch skill built from the portfolio's registry-latest skill (seldonframe/reelier examples/portfolio), plus one exact-pin step. The pinned 0.12.1 was the value that skill's seed run verified live earlier the same day; 0.13.0 was published at 17:15 UTC. Exit code 1.
FAQ
Are these soft assertions?
In one of the term's two senses. Within a step, yes: every assert line is evaluated and every failure prints on the receipt, so one broken check never hides another. Across steps, no: a diverged step stops the run, and later steps are marked skipped rather than executed against state that no longer holds. The sense that matters for agent outputs is the property sense: assert is number, matches, length >= so legitimate freshness passes and structural drift fails.
Can the grammar test what the agent wrote — summaries, answers, tone?
No, and it does not try. The grammar evaluates typed observations: status codes, bodies, JSON paths. Prose quality is an evaluation problem — LLM-as-judge or human review — and costs a model call per check. In a Level-0 replay there is no model output to grade anyway: the model was removed, the recorded tool calls re-execute, and the assertions check what those calls returned.