Evaluations are table stakes. Outcomes are not.

If you are nine months into an AI pilot, you have probably sat through this meeting. The vendor's monthly review walks through test dashboards. The scores look healthy. The AI is answering accurately, staying on task, and getting caught by its safety filters when it should be. The deck closes with a recommendation to expand into the next workflow. Then you open the operations report on the same workflow and the cost per resolution has crept back to where it was before the pilot started. Handle time is roughly the same. The number you actually bought the AI to move has not moved.

The model is behaving. The business is not winning.

Those dashboards have a name in the industry. They are called evaluations, and they are automated tests that check whether the AI is doing its job properly: is the answer relevant, is it grounded in real source material rather than invented, did it use the right tool, did it stay inside the rules. Six months ago, "we run rigorous AI evaluations" was a defensible thing for a firm like ours to say. It is not anymore. Microsoft, AWS, and Google now ship these tests built into the platforms you are already buying, by default, out of the box. Two of the three shipped or consolidated theirs inside a single spring: AWS in March 2026, Google in April 2026. Across roughly six months, "we do evaluations well" stopped telling you anything about one AI firm versus another.

That shift is not the interesting part. The interesting part is what it makes visible. Those tests, however good they get, measure whether the model is working. They do not measure whether your workflow is still moving the number it was bought to move. Those are two different questions, and the one you are paying for is the second.

Where custom tests genuinely earn their keep

The easy reading of all this is "the built-in tests are enough, custom ones are noise." It is wrong, and the platform vendors say so themselves.

Microsoft's own documentation names the boundary: the built-in tests do not cover accuracy inside your specific domain, your brand's tone, or whether the output arrives in the exact format the next system expects. AWS goes further and names healthcare and finance as cases where conformance to business rules, correct regulatory citations, and a complete audit trail all need logic somebody has to write. AWS also concedes two sharper limits:

1. A skipped tool passes in silence. If the AI skips a tool entirely, the tests that check tool use have nothing to look at, so they return a pass.

2. Every step can pass while the task is still wrong. When work runs across several steps, an error in step two quietly corrupts steps three and four, and each individual step still scores fine.

Custom tests close that gap.

1. Checking the shape of the output, not just the words. A generated spreadsheet can be perfectly valid and still write "ARR Trailing 12" into a cell the next system reads as "ARR YTD."

2. Checking your own rules. Whether the response applied the right tiered discount, or cited the right regulatory clause.

3. Watching the whole run, not each step. Following the AI across the full task, so a chain of individually passing steps that adds up to a wrong answer gets caught.

Hamel Husain, one of the most-read practitioners on testing AI systems, has the cleanest summary of the discipline: decide what counts as a real failure, build a test that detects that specific failure, and check the test against human judgment. Generic scores will not catch a real product failure, and good scores on generic tests do not mean the system works.

So the argument is not that custom tests are unnecessary. They are necessary for any serious AI workflow, and we build them as a matter of course. The argument is that the whole testing layer, built-in and custom together, sits below the layer where the money is. Husain is right that you cannot tie a test to a business number before you have built the test that detects the failure. That is an ordering argument, not a stopping point. Most vendors never get to the second step.

What none of those tests measure

None of them measures whether your number moved.

The practitioner literature gets close to naming this. Anthropic's engineers separate testing from live A/B testing: testing gives you confidence to ship and surfaces model problems, while A/B testing measures what actually happened to users.

The vendor framing stops there. Nobody in the platform documentation names a third tier: the line running from the test, to the operational number, to the money on your P&L. That third tier is what most AI programs skip, and it is precisely the layer the dashboards do not cover.

This is where AI returns quietly die. The model keeps behaving. The scores stay green. The workflow does not move the number, and nobody is looking at the workflow.

We learned this on ourselves

A few months ago, an internal review presented our testing practice as a forward-looking differentiator for the firm. The work behind that practice was real and rigorous and still is. In the same review, the team was also designing a redesign discipline with explicit business targets attached. The two collided. Within roughly two quarters, the platforms shipped the same test catalogue we were positioning around, and the differentiator depreciated. Nobody got it wrong. The category moved fast and we tightened the playbook in response.

The tightening looks like this. The testing discipline stays, because rigorous testing is a prerequisite for everything else. Custom tests stay, because the gaps in the built-in versions are real and show up on every engagement. What we add on top is a layer that does not exist in the platform documentation: a discipline that runs the line from the tests, to your operational number, to the money, and keeps running after launch rather than stopping at it. We call it Outcome Assurance.

The phrase is not industry standard, and the principle of tying testing to business outcomes has been argued under other names. What is ours is the operator-facing packaging, not the principle.

What the discipline looks like in delivery

The shape is easier to see in a real engagement. We recently built a finance workflow for a Canadian healthcare services company: several AI steps working in sequence inside the client's own Azure environment, pulling figures out of documents, sorting them into the right financial categories, and producing an output an analyst could sign. Three controls carry the discipline.

1. Low-confidence calls go to a person. When the AI is unsure which category a figure belongs in, it does not quietly push its guess into the output. A reviewer sees it first.

2. Two kinds of wrong get tested separately. One test asks whether the AI understood the input correctly. A second asks whether the output is valid for the system that receives it. Those fail differently and get fixed differently, and collapsing them into one score loses both signals.

3. A Golden Dataset that gets re-run every time anything changes. Real client cases paired with verified correct answers, run again on every change, with a check that flags any drift before it reaches somebody using the system for real.

None of those three is exotic on its own. The discipline is in keeping all three running on a cadence, and tying each one to a number that maps to the workflow's business result rather than to the model's behaviour.

A test you can run this week

The question, weighing a vendor, is not whether they run tests. Every serious vendor does, and the platforms ship the basics by default. Strong testing is the floor. It is necessary, and it is not the same as knowing whether the workflow paid.

Open the most recent AI proposal on your desk. Look for a single page, slide, or paragraph that draws a line from one of the tests the vendor is selling you, to the operational number that is supposed to move, to the money on your P&L that number rolls into.

If you find it, the proposal is unusual. Most cannot. If a vendor can name the workflow, the person whose job changes, the number that moves, and how often the line gets checked, that is a proposal somebody thought past the dashboard. If they can only describe accuracy scores and error rates, they are selling plumbing.

This is the same posture the one-claim test argued from the homepage side: read the firm's claim, run it against a test you can apply inside ninety days, and watch how many fail. The testing version of that is narrower. Show me the line from a test to an operational number to money. A vendor can answer or they cannot.

On our side, the discipline that makes that answer concrete lives on the Outcome Assurance page. It names what we measure, how we tie it to your number, and what "continuous" means in practice. If you want to test us on the same question we are asking you to take to everybody else, start there.