How we workA method, not a product

Evidence from agents. Decisions from people.

The agent test loop is how we improve a process, an automation or a system together with AI agents. Agents test it the way a real user would and report what they find; people decide what changes. It's worked well for us, so we're sharing it.

Narrated, with captions in the video. Sample data throughout.

Read the transcript

Most teams fix whatever fails their test cases. Over time the system learns the tests, not the users.

This loop keeps the tests as evidence, and leaves the decisions with people.

Start with a handful of real cases that an expert has already done and corrected. That is the answer key.

Seed the data the system would really hold, and vary where the work begins. Real work is mostly picking up where something stopped.

A fresh agent gets exactly what a real user’s agent would: the product’s own tools and a short brief. No hints, no source, no answers.

It uses the live system freely, the way an outsider would, including the waits and the guesses.

When it asks a question, the answer comes only from the key. Coach it, and the gaps it would have found disappear.

An independent reviewer scores each result against the key, field by field.

Every difference gets a cause: a product miss, the test setup, the key itself, or simply how the expert happens to work.

Report the honest range, with the method, and read the transcript for the experience: the waits, retries and guesses.

Then ask the simulated user why it did what it did: what slowed it, where it guessed, what a daily user would want.

Before anything counts as a bug, read the same fact from two places. Some “bugs” are just a slow job or a timing race.

Every finding gets exactly one scope, decided with a person, by one question.

Only system findings change the product. A finding from a single case needs a second case before it becomes a fix.

A case can reveal a problem, but it never defines the behavior. That is what stops the product overfitting to the tests.

Each agreed system finding gets its own fix: reproduce, find the root cause, fix it in general terms, add a regression check, review it independently.

Data and test findings are fixed where they live. Nothing merges until the wave is over, and a person approves every merge.

The next wave runs on the new release. It tests the fixes as well, and the first wave stays as the baseline.

Each wave adds harder starting states, so the system has to pick up work, not just start it.

We’re still learning from every wave. If you’re testing systems that agents use, we’d like to hear what’s working for you.

Why we needed it

Fix whatever fails the test, and the process learns the test

The usual way to improve a process is to collect some test cases, run them, fix what fails and run them again. It feels rigorous, and for a while it works. Then the process starts bending toward the cases you happened to pick, and the next real case, with different data, runs into the same kind of problem anyway.

Fixing whatever fails the test
A test case fails, someone adds a special case, and it passes. A few rounds later the process handles the test cases well and the next real case badly.
Testing once, by hand, before launch
It catches what's obviously broken. It says little about how the process holds up when someone, or something, uses it in an order nobody planned.
Agents left guessing
An agent doesn't read the onboarding page. When a step is unclear it guesses, waits or retries, and nobody sees it happen.
Mistakes nobody can explain
When the result is wrong, there's no record of what was assumed or what was missing, so the same mistake comes back.

Fixes this month

all tests pass

Case 3 failedIf the field is blank, use the default

Case 1 failedSkip this check for that one customer

Case 4 failedMerge the two records when the names match

Case 3 failed againBut not when the default is missing

Case 2 failedAccept the old format too

Every test passes.The next real case doesn't

The loop

Six steps, run in waves

Agents do the testing, the reviewing and the legwork. A person decides at triage, and again before any fix goes in.

01Prepare

Real cases, known answers

A handful of real cases that an expert has already done and corrected, with the work starting in different states.

02Run

An agent works the cases

A fresh agent gets only what a real user would get: the process's own tools and a short brief.

03Review

Check it, then ask why

A reviewer compares every result with the known answer; then the agent explains where it guessed or waited.

04Triage

One question per finding

With a person, each finding gets one scope: the process, the data or setup, or the test itself.

05Fix

Each fix on its own

Reproduce it, find the cause, fix it in general terms, add a check, and have someone else review it.

06Repeat

The next wave

Run again on the improved process. The new wave retests the fixes, and the first stays as the baseline.

Anyone would hit it

Change the process

The finding gets its own fix, reviewed by someone who didn't write it, and a person approves it before it goes in.

Only this test hit it

Fix the data or the test, never the process

Incomplete setup data gets fixed where it lives; a quirk of the test gets fixed in the test. The process stays as it is.

The one question

Would a different user, with a different setup, have seen this?

Every finding gets exactly one answer, decided with a person. A test case can reveal a problem, but it never gets to define how the process should behave. That's what keeps the process from bending toward the tests.

The process

Anyone, with any setup, would run into it.

Change the process, and only the process.

The data or setup

It happened because this case's data or settings were incomplete.

Fix the data. The process stays as it is.

The test itself

It happened because of how the test was set up or run, not because of the process.

Fix the test. The process stays as it is.

A problem that rests on a single case needs a second case, or a clear reason it would happen to anyone, before it becomes a fix. And nothing gets added only to make testing easier: a progress display or an audit trail goes in only if a real user would want it too.

What we've learned

Small habits that made the difference

Don't coach the agent doing the testing

The most useful findings came from an agent nobody had warned off anything. Tell it what to avoid and the gaps it would have found stay hidden. Put the safeguards in the process, not the prompt.

Start the work in different places

Real work is rarely a fresh start. Seed cases that are half done, on hold or already picked up by someone else, so the process has to pick work up, not just begin it.

Check a surprise in two places

Some "bugs" were a slow job or a timing race. Read the same fact from two sources before calling anything broken.

Keep the answer key strict

When the key is silent, that's a finding about the key. It isn't permission to invent an answer for the agent.

Give reviewers the whole source

Working from a summary, a reviewer will find problems that were in the original all along.

Two at a time is plenty

Run more groups at once and the person in the middle becomes the bottleneck, and slow runs muddy each other's timing.

Where it fits

When it's worth it, and when it isn't

It gives evidence about correctness, consistency and how the process feels to use. Whether the process is worth using at all still takes real people.

It fits when

The process is used through tools, connectors, tickets or forms, by people or by agents
Its results can be checked against work an expert has already done
Many different users or setups will rely on it, so one case shouldn't shape it

Skip it when

A small process, where a few checks by hand and a conversation will do
No known-good answers yet: build those first, or start with ordinary user testing
A run costs more than the problems it could find

We're still learning from every wave

If you're improving a process with agents, we'd like to hear what's working for you.

Compare notes See more case studies