Test your MCP server for regressions by replaying a golden run
The mental model is one line: put Reelier in front of your MCP server, record one healthy pass through the tools you ship, and freeze it as a golden run. Every replay after re-issues those exact tool calls against the live server and fails the moment a result stops matching — a regression test for your server that you get by recording, not by hand-writing request/response fixtures.
This guide runs the whole loop with the reelier CLI against a stdio MCP server: wrap it with the recorder, compile the golden run to a SKILL.md baseline, replay it, and diff two runs so a regression becomes an exit code you can gate CI on. Every command below is real and every output is verbatim from an actual run.
Before you start
- Reelier installed —
npm i -g reelier(or usenpx -y reelier@latestbelow). - The MCP server you want to guard, startable from the command line over stdio — the thing under test.
- A client or agent that can drive the golden run's tool calls once while the recorder is up — any MCP client works.
- Nothing else. No account, no API key: deterministic replay never calls a model.
Put the recorder in front of your MCP server
reelier mcp --wrap is the recorder: it launches your MCP server as a downstream and re-exposes every one of its tools through a proxy, plus three control tools — reelier_start_recording, reelier_note, reelier_stop_recording. Point your agent or client at the proxy instead of the server directly, and it sees the same tools it always did.
This command runs as a stdio MCP server and prints nothing on its own — it sits between the client and your server, capturing calls. What recording captures on each call is covered by the glossary; this guide stays on the how-to. Swap node weather-mcp.mjs for whatever launches your server (npx -y @your/mcp-server, python -m your_server, and so on).
reelier mcp --wrap "node weather-mcp.mjs"Record one golden run and compile the baseline
With the recorder up, have your client call reelier_start_recording, drive the exact tool calls that represent a healthy server — the sequence you want to guard — then call reelier_stop_recording. Each call lands in a trace .jsonl under .reelier/traces/. In the run below, one read-only get_forecast call was recorded to .reelier/traces/mcp-forecast-golden-1.jsonl.
reelier compile turns that trace into a SKILL.md — a plain markdown file, checked into your repo, that IS the golden baseline. The compiler writes minimal assertions: a step whose call recorded a clean result gets exactly one assert — status == 200 — and no brittle value-pins, so a fresh response body on each honest run never trips a false failure. It also classifies the effect from the tool's readOnlyHint, so a read-only tool is marked read and is safe to replay repeatedly.
reelier compile .reelier/traces/mcp-forecast-golden-1.jsonl -o mcp-forecast-golden.skill.mdWrote mcp-forecast-golden.skill.md steps: 1 asserts: 1 binds: 0 effects: read=1 idempotent-write=0 destructive=0 Open questions: (none)
Replay the golden run against the server
reelier run re-executes the recorded tool calls against a freshly launched copy of your server and checks each assertion — pass the same --wrap so they hit a live server, not a stub. No model is in the loop: at the default --max-level 0 an LLM client is never even constructed, so the replay costs 0 tokens and tests the deterministic core of your server, nothing re-reasoned.
Replay is read-only by default: a recorded write step does not re-fire unless you pass --allow-writes (idempotent writes only; a destructive step still needs --yes). A read-only tool like this one is safe to run on every push. Every run appends a receipt to .reelier/runs/<skill>.jsonl — the append-only record the next step diffs against.
reelier run mcp-forecast-golden.skill.md --wrap "node weather-mcp.mjs"✓ Step 1 — get the forecast for Denver [passed] 2ms PASSED: 1/1 steps ok, 0 failed, 2ms total
Diff two runs — a regression becomes an exit code
Replay the skill a second time, then reelier diff <name> compares the last two runs in that record, step by step. On a healthy server it reports SAME and exits 0 — every step's outcome and escalation level match the baseline. Wire that exit code into CI and the job stays green as long as your server keeps answering the golden run the way it did the day you recorded it.
The moment a tool regresses — it starts erroring, or a step that passed now fails its status == 200 assertion — the same command flips to DRIFTED and exits 1, blocking the merge. The receipt below is that catch, verbatim: the golden run passes against the healthy server, then fails against a regressed build, and the diff exits 1. Run it in a GitHub Actions job and that is the whole regression gate.
reelier diff mcp-forecast-golden= Step 1 "get the forecast for Denver": passed, unchanged. ✓ SAME — 1 step(s), every outcome and escalation level identical to the baseline.
The receipt
$ reelier run mcp-forecast-golden.skill.md --wrap "node weather-mcp.mjs" ✓ Step 1 — get the forecast for Denver [passed] 2ms PASSED: 1/1 steps ok, 0 failed, 2ms total $ reelier run mcp-forecast-golden.skill.md --wrap "node weather-mcp-broken.mjs" ✗ Step 1 — get the forecast for Denver [failed] 2ms - status == 200 failed: got 500 FAILED: 0/1 steps ok, 1 failed, 2ms total $ reelier diff mcp-forecast-golden ✗ Step 1 "get the forecast for Denver": passed → failed — status == 200 failed: got 500. ⚠ DRIFTED — 1 outcome/structure change(s): #1 outcome-changed.
The regression caught, verbatim: the mcp-forecast-golden skill replayed against the healthy server (passed), then against a deliberately-broken build whose get_forecast returns an error — standing in for the day your real server breaks. Reelier maps the tool error to status 500, the status == 200 assertion fails, and reelier diff exits 1 (DRIFTED). Produced on 2026-07-22 with reelier v0.14.0; the golden run was recorded by reelier mcp --wrap fronting a minimal example stdio MCP server (one read-only get_forecast tool). No model was constructed at any point.
FAQ
How is this different from writing request/response fixtures by hand?
You never write the expected values. Reelier records a real healthy call and freezes its actual result as the baseline, then replays that call against the live server on each run. There is no hand-maintained mock to drift out of sync — the golden run IS the fixture, and the server itself produced it.
My server's tool returns data that changes every call. Will that show as a regression?
No. The compiler writes minimal assertions — one status == 200 per clean step, no value-pins — so a fresh response body on each honest run updates the receipt without failing the diff. A step only drifts when its outcome flips (passed to failed), a step is added or removed, or it needed a different escalation level to pass.
Can I replay a server whose tools write data?
Replay is read-only by default, so a recorded write step does not re-fire unless you pass --allow-writes (idempotent writes only; a destructive step still needs --yes). For a scheduled regression test, record a read-only path, or gate the writes deliberately — the compile output states each step's effect so you know what is safe to replay.