The Model Update Broke My Agent: Catching Silent Regressions Before They Bill You
A provider bumps the model under your running agent and the bill and behavior move silently. Freeze the run, replay, and diff drift into a red CI check.
- 01When a provider retires or bumps the model under your agent, two costs move at once: a pricier tier (Haiku 4.5 $1.00/MTok input to Opus 4.8 $5.00) and ~30% more tokens on the newer tokenizer.
- 02Hard breaks throw a 400 and you see them; silent ones keep returning 200s and you find out from the invoice weeks later.
- 03Freeze the run that worked, replay it, and diff — a silent regression becomes a red CI check (exit 1) before it bills you.
You didn't schedule this upgrade. A model id you pinned gets retired, or a "latest" alias rolls to a newer version, or your platform swaps its default. The agent keeps running on its timer. Nothing in your code changed. The bill did.
There are two ways an upgrade breaks a running agent. A hard break throws on the next call, so you see it. A silent one keeps returning 200s while the cost per run climbs or the output quietly shifts shape — and you learn about it from the monthly invoice, or a downstream system that chokes on the new output weeks later. This post turns the silent kind into the loud kind, before it bills you.
When the update errors (the break you can see)
Some upgrades fail closed, which is the good case. A pinned model id works until its retirement date, then requests to it fail outright — Anthropic's deprecation page states "Requests to models past the retirement date will fail," with at least 60 days' email notice (retrieved 2026-07-21); claude-opus-4-20250514 was retired June 15, 2026, replaced by claude-opus-4-8. Parameters fail closed too: set temperature, top_p, or top_k to a non-default value and the call "returns a 400 error … on Claude Opus 4.7 and later, including Claude Opus 4.8, and on Claude Sonnet 5" (same source). Either way, you catch it on the first run, because it throws.
The dangerous upgrades are the ones that don't throw.
When the update just costs more (the break you can't see)
Two documented ways a silent upgrade raises the bill, neither of which errors:
- A pricier tier. Input prices from the Anthropic pricing page (retrieved 2026-07-21): Haiku 4.5 $1.00/MTok, Sonnet 5 $2.00 (introductory through Aug 31 2026; $3.00 from Sep 1), Opus 4.8 $5.00. A swap from Haiku to Opus is 5× the price, output included ($5 → $25/MTok). Anthropic's own example prices 10,000 support tickets at ~$37.00 on Haiku 4.5 (same source); Opus 4.8 is exactly 5× the Haiku tiers (input $1→$5 and output $5→$25 are both 5×), so the same job runs ~$37 × 5 ≈ $185.
- More tokens for the same text. Opus 4.7-and-later and Sonnet 5 "use a newer tokenizer [that] produces approximately 30% more tokens for the same text" (same source). Your prompts didn't grow; your token count did.
Put the tier change into the formula every recurring bill obeys — calls/month × tokens/call × price/MTok. Reelier's published benchmark clocked a small two-step lookup at 18,390 input tokens per run; at a 15-minute cadence (2,880 runs/month):
| Model tier (input) | Cost/run | Monthly @ 2,880 runs |
|---|---|---|
| Haiku 4.5 ($1.00/MTok) | $0.018 | $52.96 |
| Sonnet 5 ($2.00/MTok, intro) | $0.037 | $105.93 |
| Opus 4.8 ($5.00/MTok) | $0.092 | $264.82 |
| ✓ Reelier replay · Level 0 | $0.000 | $0 |
Output tokens are on top — ~136/run here, negligible next to input. Sonnet's row becomes $158.89/month after the introductory price ends Sep 1 2026. A silent tier swap moves you down that table with no code change, and the ~30% tokenizer inflation lands on top.
Behavior drifts too, and it drifts quietly
Cost is the half you see on a statement. The other half is tool-call drift: under a new model the agent reasons to a slightly different sequence of calls, or a model-backed endpoint returns a differently-shaped response. A scheduled agent papers over it and keeps going — which is why you don't notice until what it wrote downstream is wrong.
Deterministic replay inverts that. You record one run that worked and compile it into a SKILL.md baseline with minimal assertions: only steps whose recorded result came back clean get one; a step that failed or returned nothing stays assertion-less and replays as unchecked, never silently marked passed. Then you replay and diff against that baseline on the schedule.
reelier diff compares two run records and exits 1 on drift, so it gates a scheduled job like any failing test. Here it is on a baseline whose model-backed classify step drifted from clean to failing — real output, exit code included:
$ reelier diff triage-open-tickets
= Step 1 "Fetch open support tickets": passed, unchanged.
✗ Step 2 "Classify each ticket via the model-backed summariser endpoint": passed → failed — assert json.category is set failed: field absent in response.
⚠ DRIFTED — 1 outcome/structure change(s): #2 outcome-changed.
$ echo $?
1A two-run baseline recorded to show the gate: json.category is set passes in the baseline and fails in the candidate after the upgrade drops that field — the notes, verdict, and exit 1 are the CLI's, not hand-written. Wire that nonzero exit to your alert, and the run where the upgrade changed something gets attention instead of an invoice.
Three boundaries, stated plainly:
- Level-0 replay never calls a model — 0 tokens for the deterministic core, by construction (Level-0 replay). It can't "notice the model got smarter"; it catches the observable consequence at the tool boundary — a changed shape, a failed assertion, an added or removed step. If an upgrade is invisible there,
diffsays SAME, honestly. - To catch a changed call sequence, re-record under the new model and diff the two records — a deliberate pre-upgrade check that spends one model run before you flip the version, not tokens on every heartbeat. That is agent regression testing against a frozen baseline.
- Replay is read-only by default. A write step re-fires only with
--allow-writes(idempotent writes only); a destructive step needs an explicit--yes. A drift check can't re-send an email while it hunts for regressions.
Run your own numbers
Plug your schedule, context size, and the two tiers into the agent cost calculator — it prints the swing between the row you're on and the row an upgrade moves you to, next to the replay column pinned at $0. The token-cost pillar maps the recurring-job math; OpenClaw heartbeats and heartbeat economics work the schedule side, and Reelier vs. LangSmith shows where a trace-and-eval tool sits next to a replay gate.
The steady state you're protecting — the run that hasn't drifted — prints its own receipt, from the project README:
Your receipt: skill: reelier-init-demo steps: 2 total, 2 passed, 0 unchecked, 0 failed replay time: 44ms [measured] LLM tokens: 0 [measured] An agent doing a comparable task re-reasons every run (~2.8s, ~18k tokens on our benchmark). Your replay: 44ms, 0 tokens.
Record the run that worked. Replay it on the schedule. Let the diff, not the invoice, be the thing that tells you the model changed.
npx -y reelier run <job>.skill.md && npx -y reelier diff <job>