How to gate CI on agent drift
A drift gate turns "the agent quietly did something different" into a red build. You record one run that worked, freeze it as a SKILL.md baseline, and in CI you replay it and diff it against the last known-good run. Both reelier run and reelier diff exit 1 the moment the tool calls diverge — so a drifted agent blocks the merge like any failing test, instead of adapting around a change nobody approved.
This guide wires that gate with the reelier CLI: replay the baseline and read its exit code, diff two runs to catch structural drift, then drop the one-liner into your pipeline. Every command is real and every output below is verbatim from an actual run.
Before you start
- Reelier installed —
npm i -g reelier(or usenpx -y reelier@latestin place ofreelierin every command below). - A compiled baseline
SKILL.mdfor the workflow you want to guard. Record one and compile it withreelier from-session— walked end to end in the snapshot-test guide. Reelier replays tool calls (mcp__server__tool,http.get/http.post), notBash/Read/Edit/Write, so the baseline's value has to live in its API calls. - Prefer a read-only baseline. Replay is read-only by default (
--allow-writesunlocks idempotent writes only; a destructive step still needs--yes) — a gate that re-fires writes on every run is a footgun. - No API key needed. At the default
--max-level 0no model is ever constructed, so the gate is the deterministic core of the workflow at 0 tokens and runs on a bare CI runner with no secrets.
Replay the baseline — the exit code is the gate
reelier run <skill> re-executes the recorded tool calls against the live systems and checks each assertion. It exits 0 when every step holds and 1 when an assertion breaks — that exit code is the only thing CI needs to read.
The baseline here is the registry-latest example skill — a hand-authored example in the repo — whose one http.get keeps its assertions minimal: status == 200 (what the compiler emits for a clean recorded result) plus a hand-written shape check, json.version matches /^\d+\./, never a pinned value. That is deliberate: a fresh version on the receipt updates the record without tripping the gate. A compiled step that recorded no result would stay assertion-less and replay unchecked, never a false passed.
reelier run registry-latest.skill.md✓ Step 1 — Latest dist-tag from the registry [passed] 212ms PASSED: 1/1 steps ok, 0 failed, 212ms total
Diff two runs — a green gate exits 0
One receipt proves a single run; reelier diff <name> compares the last two runs in .reelier/runs/<name>.jsonl step by step and reduces the whole comparison to one exit code. It reports SAME and exits 0 when every step's outcome and escalation level match the baseline.
Run the skill a second time, then diff. Below, two live calls to the npm registry seconds apart returned the same shape, so the verdict stayed SAME even though the response body changed between them. Data that legitimately varies run to run is not drift — the diff compares structure, outcomes, and escalation levels, never bound values.
reelier diff registry-latest= Step 1 "Latest dist-tag from the registry": passed, unchanged. ✓ SAME — 1 step(s), every outcome and escalation level identical to the baseline.
Catch a drift — a changed tool call exits 1
Drift is when a step's outcome flips, a step is added or removed, or a step needs a different escalation level to pass. To make the gate fire, the baseline's version assertion was pinned to a stale /^0\.11\./ — the exact brittle-pin anti-pattern the shipped skill warns against — and the live registry answered 0.14.0. The next run fails, and reelier diff turns that divergence into a per-step verdict and exit code 1.
This is the whole product: instead of the agent silently absorbing a changed upstream, CI goes red and names the step that moved. Why tool calls drift, and exactly what counts, is the glossary term — this guide stays on the gate.
reelier diff registry-latest✗ Step 1 "Latest dist-tag from the registry": passed → failed — json.version matches /^0\.11\./ failed: got "0.14.0". ⚠ DRIFTED — 1 outcome/structure change(s): #1 outcome-changed.
Put the gate in your pipeline
Chain the two commands as your CI step: replay the baseline, then diff the last two runs. && short-circuits, so a non-zero exit from either one fails the job — run guards the assertions you wrote, diff guards the shape of the whole workflow. On GitHub Actions a step that exits non-zero fails the check and blocks the merge; the ready-to-paste workflow file is on the platform page, so this guide does not restate it.
Because the gate runs at --max-level 0, it needs no API key and burns no tokens — a plain runner with Node is enough. Point it at a read-only baseline and it is safe to run on every push and on a schedule.
reelier run registry-latest.skill.md && reelier diff registry-latestThe receipt
$ reelier run registry-latest.skill.md ✗ Step 1 — Latest dist-tag from the registry [failed] 139ms - json.version matches /^0\.11\./ failed: got "0.14.0" FAILED: 0/1 steps ok, 1 failed, 139ms total $ echo $? 1 $ reelier diff registry-latest ✗ Step 1 "Latest dist-tag from the registry": passed → failed — json.version matches /^0\.11\./ failed: got "0.14.0". ⚠ DRIFTED — 1 outcome/structure change(s): #1 outcome-changed. $ echo $? 1
The gate firing, verbatim: a drifted replay and the diff that blocks the merge, both exiting 1. Produced on 2026-07-22 with reelier v0.14.0 in a scratch directory, replaying the registry-latest example skill from the seldonframe/reelier repo against the live npm registry. The baseline replayed green twice (SAME, exit 0); pinning the version assertion to a stale /^0\.11\./ was the only edit before this run — the registry answered 0.14.0, so the run failed and reelier diff reported DRIFTED. Nothing mocked.
FAQ
reelier run already exits 1 on a failed assertion — why also run diff?
They guard different things. run checks each assertion against live systems; diff compares two run records and catches drift an assertion can't name — a step added or removed, or a step that still passes but now needs escalation to do it. In CI, chain them: reelier run <skill> && reelier diff <skill>.
Won't the gate go red every time the upstream returns fresh data?
No. The assertions here are shape-level — a pattern like /^\d+\./, not a pinned 0.14.0 — and diff compares outcomes and escalation levels, never bound values. A new response body updates the receipt without failing the gate — it only goes red when a step's outcome actually flips.
Does the CI gate need an API key or burn tokens?
No. At the default --max-level 0 the model client is never constructed, so the replay is the deterministic core of the workflow at 0 tokens — no secrets on the runner. Escalation to a model is BYOK and opt-in with --max-level 1|2; a destructive step never escalates and still needs --yes.