# Reelier vs LangSmith: observability watches the trace, Reelier replays it

> LangSmith records your agent's runs so you can watch, trace, and score them on its platform; Reelier runs the recorded tool calls again on your own machine, asserting each step and failing the diff when the live systems drift.

Markdown twin of https://www.reelier.com/vs/reelier-vs-langsmith — an honest comparison of LangSmith and Reelier. It leads with where LangSmith genuinely wins, then draws the line where Reelier fits. The blockquote above is the one-sentence distinction; quote it with the link.

## Where LangSmith wins

Reach for LangSmith when the question is about the model, not the plumbing. LangSmith is LangChain's framework-agnostic observability and evals platform, and it traces at a depth Reelier never attempts: a full run tree where every LLM call, prompt-formatting step, retrieval call and tool invocation nests under one operation, each with its inputs, outputs, latency and token cost. For debugging why an agent said what it said — chasing a bad answer down to the exact retrieval that poisoned it — that trace tree is the tool, and Reelier has nothing like it.

Its evaluation half is a second thing Reelier does not do. You collect examples into a `dataset`, run your app over them as an experiment, and score the outputs — including with `LLM-as-judge` evaluators that grade quality with or without a reference answer — then compare experiments side by side to catch a regression before it ships. That is quality measurement, and Reelier deliberately stays out of it: Reelier asserts that a deterministic step still executed, not that a model's prose got better or worse.

And LangSmith does replay, so don't read this page as “observability can only watch”. Open a recorded LLM run in the Playground, swap the model or edit the prompt, and re-run that call to compare the output; run your whole assistant over a dataset to re-execute it under test. On top of that sit dashboards, alerts, automatic issue detection, and annotation queues for a team debugging live production traffic. If that is your job, LangSmith is the right tool and Reelier is not the replacement.

## Where Reelier fits

Reelier replays a different layer, and without a model. LangSmith's re-runs keep the reasoning in the loop: the Playground swaps one model for another and samples fresh, an experiment re-executes your app so the LLM runs again, and the LLM-as-judge scorer is itself a model call. Reelier records the agent's typed tool calls — MCP tool calls and its own `http.get`/`http.post` builtins — and on replay re-executes exactly that sequence against the live systems, checking an assertion on every step it can check. No prompt, no sampling: the receipt reads 0 LLM tokens because none were spent.

The output is a gate, not a score. An eval hands you a number to read and judge; Reelier hands you a pass/fail. When a recorded assertion fails against this week's live response, the step fails loudly — a named failing step on the receipt, `reelier diff` exiting 1 with the failing assertion as the why. That exit code is the point: it drops into cron or CI as a drift gate, where a response that changed shape or a newly-required field trips the build, instead of scoring 0.9 on a dashboard nobody watched.

And the receipt is a local file. Reelier writes runs to `.reelier/runs/` on your own machine — no account, no key, the engine AGPL-3.0 and forkable. LangSmith is a hosted suite; cloud, hybrid and self-hosted options exist, but the model is that traces flow to the platform, because that is what powers the dashboards and the fleet view. The trade is symmetric: give up LangSmith and you lose the production observability and the collaborative debugging; keep your job off the platform and your run data stays on disk.

Two boundaries Reelier states up front. Replay is read-only by default — `--allow-writes` unlocks idempotent writes only, and a destructive step needs `--yes` — so a recorded job that writes does not silently re-fire on every gate run. And judgment still needs a model: triaging a drift, deciding what to do about it, writing prose is opt-in and BYOK, in levels (at `--max-level 0` an LLM client is never even constructed; destructive steps are never escalated). The two tools compose — trace and evaluate a new feature in LangSmith, then gate the recurring deterministic job with Reelier so it runs at 0 tokens and speaks up only when something real moves.

## Side by side

| | LangSmith | Reelier |
| --- | --- | --- |
| What re-runs | An LLM call re-run in the Playground against a new model/prompt, or your app re-run over a dataset — reasoning included | The recorded tool-call sequence, re-executed against the live systems with no model in the loop |
| Model in the loop on re-run | Yes — that's the point: swap models, and LLM-as-judge scores the output | No — 0 LLM tokens by default; judgment is opt-in, BYOK, and never touches destructive steps |
| What a re-run produces | A trace and an eval score you read and judge | A pass/fail receipt; `reelier diff` exits 1 on drift — a cron/CI gate, not a number |
| Where the run data lives | The LangSmith platform (cloud, hybrid, or self-hosted) | Local files in `.reelier/runs/` — no account, no key, engine AGPL-3.0 |
| What it records | Full trace trees: LLM calls, prompt steps, retrieval calls, tool invocations | The agent's typed tool calls only (MCP + `http` builtins) — the deterministic core |
| Primary job | Observe, evaluate and debug LLM-app quality, especially on live traffic | Drift gate on a recurring deterministic agent job against real systems |

## A real receipt

The output below is a real replay receipt, not a mockup: the `registry-latest` portfolio skill — one recorded HTTP call to the npm registry — replayed on 2026-07-21 with the CLI pinned to the version that recorded the seed (`npx -y reelier@0.12.1`), so it reproduces verbatim. No trace was uploaded and no model ran; the pass and the token count are read off the local run record, not a hosted dashboard.

```
npx -y reelier@0.12.1 run examples/portfolio/registry-latest.skill.md

=== registry-latest ===
✓ Step 1 — Latest dist-tag from the registry [passed] 152ms

PASSED: 1/1 steps ok, 0 failed, 152ms total
```

Verbatim seed receipt recorded in seldonframe/reelier examples/portfolio/README.md (reelier@0.12.1, 2026-07-21). Replay it yourself from a checkout of the repo — read-only, no account, no key.

## Related

- [Deterministic replay — re-running recorded tool calls without a model](https://www.reelier.com/learn/deterministic-replay)
- [Agent regression testing — replay as a pass/fail gate, not an eval score](https://www.reelier.com/learn/agent-regression-testing)
- [Run receipts — the local record a replay writes, no account required](https://www.reelier.com/learn/run-receipt)
- [Reelier vs Promptfoo — deterministic replay next to prompt/output evals](https://www.reelier.com/vs/reelier-vs-promptfoo)
- [Reelier vs Temporal — an eval/observability platform's cousin: durable execution vs replay](https://www.reelier.com/vs/reelier-vs-temporal)

## Sources

- [LangSmith docs — observability, evaluation, monitoring, and deployment options](https://docs.langchain.com/langsmith)
- [LangSmith observability — a trace is a collection of runs (LLM call, prompt step, retrieval call)](https://docs.langchain.com/langsmith/observability-concepts)
- [LangSmith evaluation — datasets, LLM-as-judge evaluators, and comparing experiments](https://docs.langchain.com/langsmith/evaluation-concepts)
- [LangSmith Studio — open an LLM run in the Playground to swap model/prompt and re-run](https://docs.langchain.com/langsmith/observability-studio)
- [seldonframe/reelier — the open-source (AGPL-3.0) replay engine this page describes](https://github.com/seldonframe/reelier)

Other comparisons: [VCR / Polly.js](https://www.reelier.com/vs/reelier-vs-vcr) · [promptfoo](https://www.reelier.com/vs/reelier-vs-promptfoo) · [Temporal](https://www.reelier.com/vs/reelier-vs-temporal) · [Codex Record & Replay](https://www.reelier.com/vs/reelier-vs-codex-record-replay) · [Muscle-Mem](https://www.reelier.com/vs/reelier-vs-behavior-caches) · [a hand-written script](https://www.reelier.com/vs/reelier-vs-hand-written-scripts) · [LLM-executed agent skills](https://www.reelier.com/vs/reelier-vs-agent-skills)