Agentic Visual Testing - Judging and Harnesses

A failing check isn’t always a regression. This page covers the two things that turn a raw diff into a decision: judging (is this change real, intentional, or noise?) and harnesses (driving the page - login, interactions - so the right thing gets screenshotted in the first place).

The judging model

The heuristic verdict pipeline emits one of four labels per failing entry:

For ambiguous, --judge host writes a JudgmentRequest to .blazediff/judgments/<id>/request.json with:

• regions[] - bounding boxes, pixel counts, and change types per region
• paths.locator (locator.png) - a ~400px overview with regions outlined in red
• paths.tiles (regions.png) - a vertical stack of [baseline | actual] pairs
• paths.{baseline,actual,diff} - full-page PNGs as a fallback

The host agent writes its verdict to .blazediff/judgments/<id>/verdict.json:

{
  "id": "agent",
  "verdict": {
    "label": "intentional-likely",
    "headline": "Em-dash replaced with hyphen in copy",
    "rationale": ["region tile shows only typographic substitution"],
    "action": "rewrite-if-intended"
  },
  "rationale": "Full paragraph explanation...",
  "confidence": 0.95
}

Then merge verdicts into the report - no re-screenshot:

blazediff-agent check --apply-judgments --json

Accept an intentional change by re-baselining the entry (mask/viewport/waitFor are preserved; only the PNG regenerates):

blazediff-agent rewrite agent --json       # by id
blazediff-agent rewrite --failed --json    # all failures from the last check

Why agents fail at visual review

Handing a coding agent two screenshots and asking “what changed” fails reliably. The failures have specific causes, and the judgment request is shaped to avoid each one.

The rule underneath all of it: describe each side separately, then diff the descriptions. A model asked to compare two images in one step will fill gaps with what it expects to see. A model asked what a single crop contains is answering a much easier question, and the comparison afterwards is arithmetic. The --judge local backend follows the same shape - Moondream describes, then a deterministic word diff runs, then Qwen classifies the result.

If your agent is still returning bad verdicts, check in this order:

1. Is it reading regions.png and locator.png, or reaching straight for the full-page PNGs? The full pages are a fallback, not the input.
2. Is the entry actually ambiguous? A regression-likely entry does not need a verdict, it needs a look.
3. Is the diff real, or is the region non-deterministic? Masking a flake is correct; asking an agent to rule on a spinner is not.

Harnesses

A harness is a pluggable ESM script in .blazediff/harnesses/<name>.js, attached to an entry via its harnesses: [{ name, params? }] list. Login is just one kind - anything that drives the page before or around a screenshot is a harness. Two phases:

• setup - runs before navigation (establish a session, e.g. login).
• interact (default) - runs after the base screenshot; drives the page and may emit extra named screenshots via screenshot(name), each its own baseline entry <entry>__<name>.

Interaction harness

// .blazediff/harnesses/weather-menu.js
/** @type {import("@blazediff/agent").Harness} */
export default {
  async run({ page, screenshot }) {
    await page.getByRole("button", { name: "More options" }).click();
    await screenshot("menu"); // -> baseline "weather__menu"
  },
};

{ "id": "weather", "url": "/weather", "harnesses": ["weather-menu"] }

Login harness

Routes behind a login flow capture through a setup harness. Credentials live in environment variables - never in the harness file, the manifest, or LLM context.

/** @type {import("@blazediff/agent").Harness<{ persona?: string }>} */
export default {
  phase: "setup",
  async run({ page, params }) {
    const upper = (params.persona ?? "default").toUpperCase().replace(/[^A-Z0-9]/g, "_");
    const email = process.env[`BLAZEDIFF_AUTH_${upper}_EMAIL`];
    const password = process.env[`BLAZEDIFF_AUTH_${upper}_PASSWORD`];
    if (!email || !password) throw new Error(`missing BLAZEDIFF_AUTH_${upper}_*`);
    await page.goto("http://127.0.0.1:3000/login");
    await page.locator('input[name="email"]').fill(email);
    await page.locator('input[name="password"]').fill(password);
    await Promise.all([
      page.waitForURL((u) => !u.pathname.startsWith("/login")),
      page.getByRole("button", { name: /sign in|log in/i }).click(),
    ]);
  },
};

Attach it per entry, and drop credentials in .blazediff/.env (auto-gitignored):

{ "id": "dashboard", "url": "/dashboard",
  "harnesses": [{ "name": "auth", "params": { "persona": "default" } }] }

For OAuth/SSO, magic links, MFA, or captcha - record interactively instead: blazediff-agent auth init --persona default --login-url http://127.0.0.1:3000/login.

Masking flaky regions

When a diff is noise-likely - or a real-looking diff is actually caused by something non-deterministic - mask it, don’t rebaseline. A rebaseline just resets the clock on a flake; a mask removes it. Mask auto-cycling animations, third-party iframes, timestamps, per-session randomness, and personalization noise. Don’t mask real content that happens to be changing - that’s the change you want caught.

The agent always masks any element matching [data-blazediff-agent-mask] - no manifest change needed. Add it to a shared component and it applies on every route:

<div data-blazediff-agent-mask="report-carousel">...</div>

When you can’t edit the source (third-party embed), fall back to a per-entry CSS selector. The mask list replaces the existing one, so include every selector you want kept:

cat <<'EOF' | blazediff-agent capture --stdin --mode baseline --json
[
  {"id": "examples-vanilla", "url": "/docs/ui-components/vanilla", "mask": ["iframe"]}
]
EOF

Full reference: @blazediff/agent docs →.