---
title: "The Model Update Broke My Agent: Catching Silent Regressions Before They Bill You"
description: A provider bumps the model under your running agent and the bill and behavior move silently. Freeze the run, replay, and diff drift into a red CI check.
date: 2026-07-21
slug: model-upgrade-regression
author: Maxime Houle
tags: cost, model-upgrade, drift, reelier
kicker: Cost engineering
---

```short-version
- When a provider retires or bumps the model under your agent, two costs move at once: a pricier tier (Haiku 4.5 $1.00/MTok input to Opus 4.8 $5.00) and ~30% more tokens on the newer tokenizer.
- Hard breaks throw a 400 and you see them; silent ones keep returning 200s and you find out from the invoice weeks later.
- Freeze the run that worked, replay it, and diff — a silent regression becomes a red CI check (exit 1) before it bills you.
```

You didn't schedule this upgrade. A model id you pinned gets retired, or a "latest" alias rolls to a newer version, or your platform swaps its default. The agent keeps running on its timer. Nothing in your code changed. The bill did.

There are two ways an upgrade breaks a running agent. A hard break throws on the next call, so you see it. A silent one keeps returning 200s while the cost per run climbs or the output quietly shifts shape — and you learn about it from the monthly invoice, or a downstream system that chokes on the new output weeks later. This post turns the silent kind into the loud kind, before it bills you.

## When the update errors (the break you can see)

Some upgrades fail closed, which is the good case. A pinned model id works until its retirement date, then requests to it fail outright — [Anthropic's deprecation page](https://platform.claude.com/docs/en/about-claude/model-deprecations) states "Requests to models past the retirement date will fail," with at least 60 days' email notice (retrieved 2026-07-21); `claude-opus-4-20250514` was retired June 15, 2026, replaced by `claude-opus-4-8`. Parameters fail closed too: set `temperature`, `top_p`, or `top_k` to a non-default value and the call "returns a 400 error … on Claude Opus 4.7 and later, including Claude Opus 4.8, and on Claude Sonnet 5" (same source). Either way, you catch it on the first run, because it throws.

```source
url: https://platform.claude.com/docs/en/about-claude/model-deprecations
meta: Anthropic model deprecations · retrieved 2026-07-21
link: See the deprecation policy
> "Requests to models past the retirement date will fail."
```

The dangerous upgrades are the ones that don't throw.

## When the update just costs more (the break you can't see)

Two documented ways a silent upgrade raises the bill, neither of which errors:

1. **A pricier tier.** Input prices from the [Anthropic pricing page](https://platform.claude.com/docs/en/about-claude/pricing) (retrieved 2026-07-21): Haiku 4.5 $1.00/MTok, Sonnet 5 $2.00 (introductory through Aug 31 2026; $3.00 from Sep 1), Opus 4.8 $5.00. A swap from Haiku to Opus is 5× the price, output included ($5 → $25/MTok). Anthropic's own example prices 10,000 support tickets at ~$37.00 on Haiku 4.5 (same source); Opus 4.8 is exactly 5× the Haiku tiers (input $1→$5 and output $5→$25 are both 5×), so the same job runs ~$37 × 5 ≈ $185.
2. **More tokens for the same text.** Opus 4.7-and-later and Sonnet 5 "use a newer tokenizer [that] produces approximately 30% more tokens for the same text" (same source). Your prompts didn't grow; your token count did.

Put the tier change into the formula every recurring bill obeys — **calls/month × tokens/call × price/MTok.** Reelier's [published benchmark](https://github.com/seldonframe/reelier/blob/main/docs/strategy/reelier-launch/benchmark-results.md) clocked a small two-step lookup at **18,390 input tokens per run**; at a 15-minute cadence (2,880 runs/month):

| Model tier (input) | Cost/run | Monthly @ 2,880 runs |
|---|---:|---:|
| Haiku 4.5 ($1.00/MTok) | $0.018 | $52.96 |
| Sonnet 5 ($2.00/MTok, intro) | $0.037 | $105.93 |
| Opus 4.8 ($5.00/MTok) | $0.092 | $264.82 |
| ✓ Reelier replay · Level 0 | $0.000 | $0 |

Output tokens are on top — ~136/run here, negligible next to input. Sonnet's row becomes $158.89/month after the introductory price ends Sep 1 2026. A silent tier swap moves you down that table with no code change, and the ~30% tokenizer inflation lands on top.

## Behavior drifts too, and it drifts quietly

Cost is the half you see on a statement. The other half is [tool-call drift](/learn/tool-call-drift): under a new model the agent reasons to a slightly different sequence of calls, or a model-backed endpoint returns a differently-shaped response. A scheduled agent papers over it and keeps going — which is why you don't notice until what it wrote downstream is wrong.

[Deterministic replay](/learn/deterministic-replay) inverts that. You record one run that worked and compile it into a `SKILL.md` baseline with minimal assertions: only steps whose recorded result came back clean get one; a step that failed or returned nothing stays assertion-less and replays as `unchecked`, never silently marked passed. Then you replay and diff against that baseline on the schedule.

`reelier diff` compares two run records and exits 1 on drift, so it gates a scheduled job like any failing test. Here it is on a baseline whose model-backed classify step drifted from clean to failing — real output, exit code included:

```
$ reelier diff triage-open-tickets
  = Step 1 "Fetch open support tickets": passed, unchanged.
  ✗ Step 2 "Classify each ticket via the model-backed summariser endpoint": passed → failed — assert json.category is set failed: field absent in response.

⚠ DRIFTED — 1 outcome/structure change(s): #2 outcome-changed.
$ echo $?
1
```

A two-run baseline recorded to show the gate: `json.category is set` passes in the baseline and fails in the candidate after the upgrade drops that field — the notes, verdict, and exit 1 are the CLI's, not hand-written. Wire that nonzero exit to your alert, and the run where the upgrade changed something gets attention instead of an invoice.

Three boundaries, stated plainly:

- **Level-0 replay never calls a model** — 0 tokens for the deterministic core, by construction ([Level-0 replay](/learn/level-0-replay)). It can't "notice the model got smarter"; it catches the observable consequence at the tool boundary — a changed shape, a failed assertion, an added or removed step. If an upgrade is invisible there, `diff` says SAME, honestly.
- **To catch a changed call sequence, re-record under the new model and diff the two records** — a deliberate pre-upgrade check that spends one model run before you flip the version, not tokens on every heartbeat. That is [agent regression testing](/learn/agent-regression-testing) against a frozen baseline.
- **Replay is read-only by default.** A write step re-fires only with `--allow-writes` (idempotent writes only); a destructive step needs an explicit `--yes`. A drift check can't re-send an email while it hunts for regressions.

## Run your own numbers

Plug your schedule, context size, and the two tiers into the [agent cost calculator](/tools/agent-cost-calculator) — it prints the swing between the row you're on and the row an upgrade moves you to, next to the replay column pinned at $0. The [token-cost pillar](/blog/agent-token-costs-pillar) maps the recurring-job math; [OpenClaw heartbeats](/blog/openclaw-cron-costs) and [heartbeat economics](/blog/heartbeat-economics) work the schedule side, and [Reelier vs. LangSmith](/vs/reelier-vs-langsmith) shows where a trace-and-eval tool sits next to a replay gate.

The steady state you're protecting — the run that hasn't drifted — prints its own receipt, from the [project README](https://github.com/seldonframe/reelier):

```receipt
Your receipt:
  skill:        reelier-init-demo
  steps:        2 total, 2 passed, 0 unchecked, 0 failed
  replay time:  44ms  [measured]
  LLM tokens:   0     [measured]

  An agent doing a comparable task re-reasons every run (~2.8s, ~18k tokens on
  our benchmark). Your replay: 44ms, 0 tokens.
```

Record the run that worked. Replay it on the schedule. Let the diff, not the invoice, be the thing that tells you the model changed.

```sh
npx -y reelier run <job>.skill.md && npx -y reelier diff <job>
```