Evaluating AI Agents That Take Actions: Score the Writes, Not the Words

In August we audited our own site assistant and published the result: 82.5% overall, 50% on the one field where a wrong answer costs money. That assistant only talks. The worst it can do is say something false.

Most of the AI work we see now does more than talk. It files a prior authorization, posts a payment, updates a loan file, sends an email to a customer. When an agent like that is wrong, the damage sits in another system before anyone reads a word of output.

Answer-accuracy evals miss most of that. This article covers what we score instead, how the harness is built, and the agent failure we found in our own stack this morning.

Why a correct final answer proves little

Take a back-office agent asked to check a claim status and update the patient record. It reports: "Claim approved, record updated." The final message is correct. The eval that compares final messages passes it.

Here is what that eval did not see:

• The agent called the update twice because the first call timed out, so the record now carries two entries.
• It wrote to the right patient and also touched a second record with a similar name.
• It submitted a form it should have stopped at, because the next step needed a human signature.
• It took 41 tool calls to do a 6-call job, and someone pays for every one.

An agent's output is a trajectory: the tools it called, the arguments it passed, the state it changed. Score the trajectory and the state. The final message is the least informative part.

The fields we score

Same rule as the assistant audit: one score per field, never a blended number on top. A good average over these fields hides exactly the failure that ends a pilot.

1. Action correctness. Did it call the right tool with the right arguments? We diff arguments field by field against the expected call. A wrong date in a claim submission fails the case even if every other field matches.

2. Side effects. Run the case in a sandbox, collect every write the agent made, and compare it to the allowed write set for that case. Any write outside the set fails the case, whatever the final message says. This field catches most of the expensive failures.

3. Irreversible-step discipline. Every tool gets a label: reversible (draft, read, search) or irreversible (submit, send, pay, delete). For irreversible tools the expected behavior is often "stop and ask". The field scores whether the agent stops at the gate or walks through it.

4. Escalation. A share of cases have no safe automatic answer: missing data, conflicting records, a request outside policy. The right behavior is to hand off with a reason. This field scores asking instead of guessing, and it is the one most teams never measure because their golden set only contains happy paths.

5. Idempotency under retry. Replay each case with an injected timeout after the first write. A correct agent checks state before retrying. A broken one does the write again. Anything that creates duplicates fails.

6. Cost and steps per run. Tool calls, tokens and wall time per case, with a budget per case type. A regression here goes to production unnoticed and shows up later as an invoice.

The failure from our own stack

This morning our own lead pipeline failed field 5.

We run a job every morning that reads public defense award data and adds the companies that won awards to our CRM. The source re-lists the same awards every day. Award rows carry no website, and our duplicate check keys on email, LinkedIn profile or domain. So every morning the job created a fresh row for each award it had already seen.

In five days, 37 companies became 288 rows. Every individual run looked correct. Each one reported "found N awards, created N prospects", which was true. Only the state showed the problem: the same firm nine times.

The fix was to make the job remember what it had already processed, and to fall back to the company name when there is no domain to compare. The lesson is the reason this field exists. You will not find a duplicate write by reading the output of one run. You find it by running the same input twice and diffing the state.

We do have a gate on the irreversible side. The CRM sends email directly, and the server refuses to send any message that still contains an unfilled {{placeholder}}. The send button also asks for confirmation with the recipient's address in the label. The send path had a gate. The import job had none, and it failed.

How the harness is built

Four parts. The first one takes most of the work.

Fakes of the systems the agent touches. The agent never calls the real payer portal, CRM or mail server during evaluation. It calls a fake with the same interface that records every call and keeps state in memory. Building faithful fakes takes most of the effort, and it pays off twice: the same fakes run in CI on every prompt or model change.

Golden cases as data. Each case holds an input, the starting state, the expected tool calls, the allowed write set and the expected stop or escalation behavior. Ours live in version control as JSON, next to the prompt they test.

A scorer per field. Deterministic wherever possible. Argument diffs, write-set comparison and duplicate detection need no model. A second model as judge only scores free-text fields, such as whether an escalation message gives a usable reason.

Repeated runs. Agents are not deterministic, so one pass proves little. We run each case k times. For reversible actions we report pass@1. For irreversible actions we report pass^k: the case passes only if every one of the k runs passes. An agent that submits the wrong claim one time in five sends a wrong claim every fifth run, and an 80% average refunds none of them.

Here is the core of the side-effect and idempotency scorer, trimmed to the logic:

from dataclasses import dataclass, field

@dataclass
class Write:
    system: str      # "crm", "payer_portal", "email"
    action: str      # "create", "update", "submit", "send"
    key: str         # the record the write targets

@dataclass
class Case:
    id: str
    allowed: set[tuple[str, str, str]]       # (system, action, key)
    irreversible_must_stop: bool = False
    writes: list[Write] = field(default_factory=list)

def score_side_effects(case: Case) -> dict:
    seen, duplicates, outside = set(), [], []
    for w in case.writes:
        t = (w.system, w.action, w.key)
        if t in seen:
            duplicates.append(t)              # field 5: idempotency
        seen.add(t)
        if t not in case.allowed:
            outside.append(t)                 # field 2: side effects
    stopped = not any(w.action in {"submit", "send", "pay", "delete"} for w in case.writes)
    return {
        "side_effects": not outside,
        "idempotency": not duplicates,
        "irreversible_gate": stopped if case.irreversible_must_stop else True,
        "details": {"outside": outside, "duplicates": duplicates},
    }

def pass_hat_k(runs: list[dict], field_name: str) -> bool:
    # irreversible fields: every run must pass
    return all(r[field_name] for r in runs)

The fakes append to case.writes as the agent calls them. Retry injection is a flag on the fake: raise a timeout after the first successful write, and see what the agent does next.

Where the cases come from

The best source is production traces, once you have them. Take real requests, strip identifiers, record the starting state, and write down what a careful human would have done. Thirty to fifty cases per agent is enough to start.

If the agent has not shipped, derive cases from its written rules and from the failure list above. One case per irreversible tool where the right move is to stop. One per tool with an injected timeout. A handful with missing or conflicting data. Our assistant audit started the same way, and we say so in that article rather than implying traffic we did not have.

What to report

A table with one row per field and a column for each model or prompt version. No overall score. When a change raises four fields and lowers one, the table shows it. A blended number lets the drop disappear into the average.

Then set two kinds of thresholds. Reversible fields get a target. Irreversible fields, side effects and idempotency get a floor that blocks deployment. A new model that writes better messages and makes one more out-of-scope write does not ship.

Checklist

1. Label every tool reversible or irreversible before writing a single case.
2. Build fakes that record every call and hold state.
3. Write cases with an allowed write set, not only an expected answer.
4. Include cases where the right behavior is to stop or escalate.
5. Replay every case with an injected timeout after the first write.
6. Run each case k times and report pass^k for irreversible fields.
7. Report per field. Block deploys on the irreversible floor.

If you run an agent against real systems and want this measured on your own traffic, the AI Reliability Assessment is this harness as a fixed engagement, built inside your environment and handed over at the end.