# Reelier vs promptfoo: assertions judge outputs, Reelier replays the trajectory

> promptfoo scores an agent's outputs against assertions and LLM-as-judge rubrics — did the answer meet your criteria; Reelier deterministically replays the recorded tool-call sequence and fails any step that drifts from a known-good run — did the agent still do the same thing.

Markdown twin of https://www.reelier.com/vs/reelier-vs-promptfoo — an honest comparison of promptfoo and Reelier. It leads with where promptfoo genuinely wins, then draws the line where Reelier fits. The blockquote above is the one-sentence distinction; quote it with the link.

## Where promptfoo wins

Reach for promptfoo when the question is about output quality. It is, in its own words, a CLI and library for evaluating and red-teaming LLM apps: you write test cases and assertions, then run your prompts, agents, and RAGs against them and compare models side by side. The assertion catalog is deep — deterministic checks (`equals`, `contains`, `icontains`, `regex`), model-graded `llm-rubric` where a judge model scores the answer against criteria you write, and custom `javascript`/`python` asserts for anything bespoke. If your question is "did this answer meet the bar" or "is model B better than model A on my suite," promptfoo is built for exactly that and Reelier is not.

Do not mistake it for a final-string matcher. promptfoo evaluates agents and tool use, and its red-teaming is a whole category Reelier does not touch: it auto-generates adversarial inputs and probes prompt injections, jailbreaks, PII leaks from RAG context, and tool-based vulnerabilities like unauthorized data access, across chatbots, RAGs, and agents. That is security testing you should be doing, and promptfoo does it.

It is also MIT-licensed and, as of March 2026, part of OpenAI while remaining open source — a large provider catalog, an active project, and CI/CD integration for continuous eval and red-team runs. For grading output quality across model upgrades and for adversarial security testing, promptfoo is the right tool and nothing below is an argument to rip it out.

## Where Reelier fits

Reelier works on a different axis: the tool-call trajectory, not the output text. It records the agent's own tool calls — MCP tool calls and its `http.get`/`http.post` builtins — and compiles them into a SKILL.md with an assertion on every step it can check. A step whose recorded result came back clean gets a status/shape assert; a step that failed or returned no usable result stays assertion-less and replays `unchecked`, never silently "passed." On replay it re-executes those exact calls against the live systems and diffs each step against the known-good baseline.

The overlap is regression testing an agent, and the line there is clean. promptfoo answers "does the output still meet my rubric" — you author the dataset and the criteria, and its `llm-rubric` checks call a judge model on every run. Reelier answers "does the agent still make the same calls with the same outcomes" — the assertions come from the recording, and at the default `--max-level 0` an LLM client is never constructed, so a replay-plus-diff spends zero model tokens and returns SAME or DRIFTED per step, exit 0 or 1 for a cron or CI gate.

The honesty about what that diff means: it compares structure, outcomes, and escalation level, never bound values — this week's version number differing from last week's is the system working, not drift. And a green diff proves the tool path is intact; it says nothing about whether the model's reasoning got better or worse. That judgment layer is promptfoo's lane. The two compose: replay for the deterministic tool-call core, promptfoo for the output-quality and security layer on top.

The remaining limits are the same ones every honest replay tool carries. Replay is read-only by default — a recorded write step does not re-fire unless you opt in, and the gates are separate: `--allow-writes` unlocks idempotent writes only, while a destructive step is refused unless you also pass `--yes`. Model help during replay is opt-in and BYOK in levels (L0 the default never calls a model; L1/L2 add bounded healing; destructive steps are never escalated). In the measured corpus behind Reelier's launch only about 1.1% of scanned sessions were read-only end-to-end, so most real jobs need a per-write decision before they enter a replay suite.

## Side by side

| | promptfoo | Reelier |
| --- | --- | --- |
| What it checks | The agent's OUTPUT against assertions/rubrics (`contains`, `regex`, `llm-rubric`, custom code) | The recorded TOOL-CALL sequence, re-executed and diffed against a known-good run |
| How the check is authored | You write the test cases and assertions (and rubric prompts for LLM-as-judge) | Derived from the recording — an assertion on every step it can check, none faked on result-less steps |
| Model tokens per run | Deterministic asserts cost none; `llm-rubric`/model-graded checks call a judge model | Zero at the default `--max-level 0` — no LLM client is constructed |
| What a regression looks like | An output that no longer meets the assertion or rubric | A step that changed outcome, appeared, disappeared, or needed a different heal level — `reelier diff` exits 1 |
| Red-teaming / security | Yes — auto-generated adversarial inputs, jailbreaks, PII and tool-vuln probes across RAGs and agents | No — deterministic tool-path only; nothing on output safety |
| Primary job | Output-quality evals and red-teaming across prompts, models, and agents | Drift gate on a recurring agent's tool calls against live systems |

## A real receipt

The output below is a real replay-plus-diff, not a mockup: the read-only `registry-latest` portfolio skill — one recorded HTTP call to the npm registry, two recorded assertions (`status == 200` and `json.version matches /^\d+\./`) — replayed a second time and diffed against its baseline with the published CLI. There is no rubric and no judge model in the loop; the verdict is a structural SAME/DRIFTED on the tool path, the check promptfoo's output grading does not make.

```
npx -y reelier@latest run registry-latest.skill.md

✓ Step 1 — Latest dist-tag from the registry [passed] 149ms

PASSED: 1/1 steps ok, 0 failed, 149ms total

npx -y reelier@latest diff registry-latest

  = Step 1 "Latest dist-tag from the registry": passed, unchanged.

✓ SAME — 1 step(s), every outcome and escalation level identical to the baseline.
```

Produced by the author on 2026-07-21: `examples/portfolio/registry-latest.skill.md` from seldonframe/reelier copied into a scratch directory, replayed twice with `npx -y reelier@latest` (resolved to `reelier@0.13.0`) then diffed — output verbatim, exit code 0. Read-only: each replay is one HTTP GET to the npm registry, no account, no key.

## Related

- [Regression testing for AI agents — the overlap, split into the judgment lane and the tool-path lane](https://www.reelier.com/learn/agent-regression-testing)
- [Assertion grammar — the small deterministic checks a compiled step carries](https://www.reelier.com/learn/assertion-grammar)
- [Reelier vs LangSmith — evals and tracing vs deterministic replay](https://www.reelier.com/vs/reelier-vs-langsmith)
- [Reelier vs VCR / Polly.js — cassettes mock, Reelier re-executes](https://www.reelier.com/vs/reelier-vs-vcr)
- [Reelier vs behavior caches (Muscle-Mem) — a hit skips the model; Reelier re-executes and asserts](https://www.reelier.com/vs/reelier-vs-behavior-caches)

## Sources

- [promptfoo — "a CLI and library for evaluating and red-teaming LLM apps"; MIT-licensed; now part of OpenAI](https://github.com/promptfoo/promptfoo)
- [promptfoo assertions — deterministic (contains/icontains/regex/equals), llm-rubric (LLM-as-judge), and custom javascript/python asserts](https://www.promptfoo.dev/docs/configuration/expected-outputs/)
- [promptfoo red-teaming — auto-generated adversarial inputs across prompts, RAGs, and agents](https://www.promptfoo.dev/docs/red-team/)
- [promptfoo joining OpenAI (March 9, 2026) — "agreed to be acquired by OpenAI" and "will remain open source"](https://www.promptfoo.dev/blog/promptfoo-joining-openai/)
- [seldonframe/reelier — the open-source (AGPL-3.0) replay engine this page describes](https://github.com/seldonframe/reelier)

Other comparisons: [VCR / Polly.js](https://www.reelier.com/vs/reelier-vs-vcr) · [LangSmith](https://www.reelier.com/vs/reelier-vs-langsmith) · [Temporal](https://www.reelier.com/vs/reelier-vs-temporal) · [Codex Record & Replay](https://www.reelier.com/vs/reelier-vs-codex-record-replay) · [Muscle-Mem](https://www.reelier.com/vs/reelier-vs-behavior-caches) · [a hand-written script](https://www.reelier.com/vs/reelier-vs-hand-written-scripts) · [LLM-executed agent skills](https://www.reelier.com/vs/reelier-vs-agent-skills)