How to prove a Dependabot bump didn't break your AI agent

The bump-check recipe

Dependabot opened a PR. Will it break your agent?

You don’t know, and neither does Dependabot. It bumped a package version because a semver range allowed it — it has no idea what your agent actually does with that package at runtime, so a green Dependabot PR tells you nothing about whether your agent still works. The same is true for Renovate, and for a bumped MCP server package.

Your regular test suite might not help either: a lot of what an agent does — call an MCP tool, hit an API, parse a response shape — is never exercised by unit tests. The actual risk lives in the tool-call sequence, and that’s exactly what Reelier records.

The fix: record once, then re-run live on every bump

Reelier’s answer isn’t a mock or a simulation — it’s a recorded, real agent run that gets re-executed live, for real, against the bumped dependency:

Step	What happens
Record (once)	Freeze a real agent session into a `.skill.md`file — the exact tool calls, args, and assertions that worked. Commit it to the repo.
Dependabot / Renovate opens a PR	The bump lands as a normal pull request, touching your lockfile.
CI checks out the PR and installs the bumped version	The bump-check workflow (below) runs on that PR branch, so the new dependency version is what’s actually installed when Reelier replays.
Reelier re-runs the recorded skill, live	`reelier run <skill> --max-level 0`replays the same tool calls against the now-bumped dependency and checks the same recorded assertions against the fresh result — a real call, not a mock. No LLM is constructed or called at level 0, so this costs 0 tokens per run.
Pass, or a precise diff	Byte-identical when nothing drifted: every step passes, the check goes green. Something changed: the step’s assertion fails, `reelier run` exits non-zero, and `reelier diff`shows exactly which step and what changed — not just “something broke.”

Copy-paste: the bump-check workflow

Add .github/workflows/reelier-bump-check.yml. It triggers on any PR that touches your dependency manifest — exactly the PRs Dependabot and Renovate open — installs the bumped dependency first, then replays your recorded skill against it:

name: Reelier bump-check
on:
  pull_request:
    paths:
      - "package.json"
      - "package-lock.json"
      - "requirements.txt"
jobs:
  bump-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4        # checks out the PR branch — the bumped lockfile
      - uses: actions/setup-node@v4
        with:
          node-version: 20
      - run: npm ci                      # installs the version Dependabot/Renovate just bumped
      - uses: seldonframe/reelier@v1
        with:
          skill: my-job.skill.md
          max-level: "0"                 # 0 LLM tokens — pure deterministic re-run, live
          cloud-key: ${{ secrets.REELIER_CLOUD_KEY }}  # optional receipt push

Swap the npm cistep for your stack’s install command (pip install -r requirements.txt, bundle install, etc.) and the paths:list for the manifest files that actually change. If it’s an MCP server package that’s bumping, add its manifest path too — the same recipe catches an MCP-tool bump, not only an npm/pip dependency.

Don’t have a recorded skill yet? See Recording to freeze one from a real session, or Quickstart for the 60-second guided path.

What this proves (and what it doesn’t)

Honest boundary: this checks dependency and MCP-tool bumps— whether the tool-call sequence your agent relies on still behaves the same way after a package version changes. Level 0 replay never constructs or calls an LLM, so it does nottell you whether it’s safe to upgrade the model itself — that’s a different question this check isn’t answering. It also doesn’t replay open-ended reasoning or generation, only concrete tool calls with a checkable outcome (typed JSON in, typed JSON out).

A step with zero assertions is reported as unchecked, never folded into a pass — Reelier never claims to have verified something it didn’t actually check.

FAQ

Does this replace my existing test suite?

No — it’s one more gate, aimed at the part unit tests usually miss: the real tool-call sequence an agent depends on. Keep your existing CI; this adds a check for agent-specific regressions on dependency bumps.

Does it work with Renovate, not just Dependabot?

Yes — the workflow above triggers on any pull request that touches the manifest, regardless of which bot opened it. Renovate and Dependabot both work unmodified.

Does this prove a model upgrade is safe?

No. Level 0 replay (the default, and what this workflow runs) never calls a model — that’s what makes it 0 tokens. It answers “did this dependency bump change the tool calls,” not “is the new model version safe.”

What if I have more than one skill to gate?

The skill input takes one path at a time — repeat the seldonframe/reelier@v1 step (or use a matrix) once per skill. Each replay fails independently, so one drifted skill is enough to block the merge.

Full CI reference (inputs, outputs, triggers): Reelier for GitHub Actions. The general CI-gate walkthrough: Replay an agent skill on every PR.