Reelier vs LangSmith: observability watches the trace, Reelier replays it

01Where LangSmith wins

Reach for LangSmith when the question is about the model, not the plumbing. LangSmith is LangChain's framework-agnostic observability and evals platform, and it traces at a depth Reelier never attempts: a full run tree where every LLM call, prompt-formatting step, retrieval call and tool invocation nests under one operation, each with its inputs, outputs, latency and token cost. For debugging why an agent said what it said — chasing a bad answer down to the exact retrieval that poisoned it — that trace tree is the tool, and Reelier has nothing like it.

Its evaluation half is a second thing Reelier does not do. You collect examples into a dataset, run your app over them as an experiment, and score the outputs — including with LLM-as-judge evaluators that grade quality with or without a reference answer — then compare experiments side by side to catch a regression before it ships. That is quality measurement, and Reelier deliberately stays out of it: Reelier asserts that a deterministic step still executed, not that a model's prose got better or worse.

And LangSmith does replay, so don't read this page as “observability can only watch”. Open a recorded LLM run in the Playground, swap the model or edit the prompt, and re-run that call to compare the output; run your whole assistant over a dataset to re-execute it under test. On top of that sit dashboards, alerts, automatic issue detection, and annotation queues for a team debugging live production traffic. If that is your job, LangSmith is the right tool and Reelier is not the replacement.

02Where Reelier fits

Reelier replays a different layer, and without a model. LangSmith's re-runs keep the reasoning in the loop: the Playground swaps one model for another and samples fresh, an experiment re-executes your app so the LLM runs again, and the LLM-as-judge scorer is itself a model call. Reelier records the agent's typed tool calls — MCP tool calls and its own http.get/http.post builtins — and on replay re-executes exactly that sequence against the live systems, checking an assertion on every step it can check. No prompt, no sampling: the receipt reads 0 LLM tokens because none were spent.

The output is a gate, not a score. An eval hands you a number to read and judge; Reelier hands you a pass/fail. When a recorded assertion fails against this week's live response, the step fails loudly — a named failing step on the receipt, reelier diff exiting 1 with the failing assertion as the why. That exit code is the point: it drops into cron or CI as a drift gate, where a response that changed shape or a newly-required field trips the build, instead of scoring 0.9 on a dashboard nobody watched.

And the receipt is a local file. Reelier writes runs to .reelier/runs/ on your own machine — no account, no key, the engine AGPL-3.0 and forkable. LangSmith is a hosted suite; cloud, hybrid and self-hosted options exist, but the model is that traces flow to the platform, because that is what powers the dashboards and the fleet view. The trade is symmetric: give up LangSmith and you lose the production observability and the collaborative debugging; keep your job off the platform and your run data stays on disk.

Two boundaries Reelier states up front. Replay is read-only by default — --allow-writes unlocks idempotent writes only, and a destructive step needs --yes — so a recorded job that writes does not silently re-fire on every gate run. And judgment still needs a model: triaging a drift, deciding what to do about it, writing prose is opt-in and BYOK, in levels (at --max-level 0 an LLM client is never even constructed; destructive steps are never escalated). The two tools compose — trace and evaluate a new feature in LangSmith, then gate the recurring deterministic job with Reelier so it runs at 0 tokens and speaks up only when something real moves.

03Side by side