Reelier vs promptfoo: assertions judge outputs, Reelier replays the trajectory

01Where promptfoo wins

Reach for promptfoo when the question is about output quality. It is, in its own words, a CLI and library for evaluating and red-teaming LLM apps: you write test cases and assertions, then run your prompts, agents, and RAGs against them and compare models side by side. The assertion catalog is deep — deterministic checks (equals, contains, icontains, regex), model-graded llm-rubric where a judge model scores the answer against criteria you write, and custom javascript/python asserts for anything bespoke. If your question is "did this answer meet the bar" or "is model B better than model A on my suite," promptfoo is built for exactly that and Reelier is not.

Do not mistake it for a final-string matcher. promptfoo evaluates agents and tool use, and its red-teaming is a whole category Reelier does not touch: it auto-generates adversarial inputs and probes prompt injections, jailbreaks, PII leaks from RAG context, and tool-based vulnerabilities like unauthorized data access, across chatbots, RAGs, and agents. That is security testing you should be doing, and promptfoo does it.

It is also MIT-licensed and, as of March 2026, part of OpenAI while remaining open source — a large provider catalog, an active project, and CI/CD integration for continuous eval and red-team runs. For grading output quality across model upgrades and for adversarial security testing, promptfoo is the right tool and nothing below is an argument to rip it out.

02Where Reelier fits

Reelier works on a different axis: the tool-call trajectory, not the output text. It records the agent's own tool calls — MCP tool calls and its http.get/http.post builtins — and compiles them into a SKILL.md with an assertion on every step it can check. A step whose recorded result came back clean gets a status/shape assert; a step that failed or returned no usable result stays assertion-less and replays unchecked, never silently "passed." On replay it re-executes those exact calls against the live systems and diffs each step against the known-good baseline.

The overlap is regression testing an agent, and the line there is clean. promptfoo answers "does the output still meet my rubric" — you author the dataset and the criteria, and its llm-rubric checks call a judge model on every run. Reelier answers "does the agent still make the same calls with the same outcomes" — the assertions come from the recording, and at the default --max-level 0 an LLM client is never constructed, so a replay-plus-diff spends zero model tokens and returns SAME or DRIFTED per step, exit 0 or 1 for a cron or CI gate.

The honesty about what that diff means: it compares structure, outcomes, and escalation level, never bound values — this week's version number differing from last week's is the system working, not drift. And a green diff proves the tool path is intact; it says nothing about whether the model's reasoning got better or worse. That judgment layer is promptfoo's lane. The two compose: replay for the deterministic tool-call core, promptfoo for the output-quality and security layer on top.

The remaining limits are the same ones every honest replay tool carries. Replay is read-only by default — a recorded write step does not re-fire unless you opt in, and the gates are separate: --allow-writes unlocks idempotent writes only, while a destructive step is refused unless you also pass --yes. Model help during replay is opt-in and BYOK in levels (L0 the default never calls a model; L1/L2 add bounded healing; destructive steps are never escalated). In the measured corpus behind Reelier's launch only about 1.1% of scanned sessions were read-only end-to-end, so most real jobs need a per-write decision before they enter a replay suite.

03Side by side