Evals
Evals are how you find out whether a change to an agent helped, before a customer does. You keep a set of tests - situations the agent should handle - run them against your latest changes, and get a result rather than an impression. Every workspace has them, and the Agent Builder proposes a first set while it drafts an agent.
Domain members can read tests and their results. Adding, editing or removing tests and their checks, starting or cancelling runs, and changing the publish requirement need Editor or Owner access. The conversations a run produces are read-only. See Roles and permissions.
Open Evals from the Agent page's sidebar, under Observe. It is live, not part of the Setup draft: a test you add applies straight away, to every version. It has two tabs - Tests, where you write and run them, and Runs, where you read what each run found.
Tests
The Tests tab lists every test with its type and how it did in its latest run. Above the list, the latest run's result links to its full report.
There are three types of test:
- Simulated - Add test, then Simulated. You script the conversation: what the person says, message by message, and what each tool answers. Nothing runs for real: every tool call is answered from your script, and a call your script does not cover is refused without running. It is how you test what has not happened yet - a customer who leaves out their order number, a tool that fails, an account whose sign-in has expired, the same request sent twice - and it works before the Agent's accounts are even connected. Tick Answer as a failure to make a tool fail with the error you write.
- From a conversation - a conversation you were happy with, saved with Save as a test from its details. Running it replays that conversation: tools answer from what they returned at the time, so nothing runs for real and an email is never sent twice.
- Live - Add test, then Live. It carries one message and no script, so it runs for real, live tools and writes included. It can pause for an approval like any other conversation.
A test saved from a conversation keeps its own copy of it. It still replays correctly months later, after the original conversation's tool calls have aged out of the 90-day call history. And deleting the conversation does not delete the test - the test keeps working, and says that the conversation it came from is gone. If you need the conversation's content removed everywhere, remove the test as well.
Select a test to open it: what the person says, what the tools answer, its checks, and its recent results, each linking to what the agent did that time.
Run the tests
Run tests runs every test against your latest changes - what the next Publish would ship. Run this test, on a test's menu or in its drawer, runs just that one.
Each test runs chooses how many times each test runs, up to five, to find out whether it passes every time or only sometimes. A test passes only when every one of its runs does, and each run costs what a conversation costs.
Checks turn a run into a result
A run on its own only shows you what the agent did. Add checks to a test and each run is judged instead: passed, failed, or - with no checks - unscored. Unscored is never a pass; it means nothing was asked of the test.
| Check | Passes when |
|---|---|
| Reply contains / omits | Some reply contains the text, or none does (capital letters ignored). |
| Tool called / not called | The tool was used at least once, or never. |
| Tool order | One tool was used before another. |
| Tool sent a value | Some use of the tool sent exactly this value in this field - the right recipient, the right amount. |
| Tool called exactly | The tool was used exactly this many times - 0 for never, 1 to catch a duplicate. |
| No divergence | Every tool call was answered by the recorded conversation or the script. |
| Token and round limits | The run stayed within a size budget. |
Tool sent a value names the field with a path such as /to or /order/id. A field whose name marks it as a password or key is stored hidden, so a check on it fails and says why rather than passing on a value it cannot see.
A check is held to its own test when you save it. On a test saved from a conversation, a check the recorded conversation would fail is refused as a typo. On a simulated or live test there is nothing recorded to try it against, so a check that cannot be read as written - a field path without its leading /, a count that is not a whole number - is refused instead.
Each check shows how it did in the test's latest run, across every one of its runs - Passed 2 of 3 is a failure - and, when it failed, why. Select a check to change what it looks for or how it is described; its type stays the same. An edit is held to the same rules as a new check, and the check reads Changed since the last run until the test runs again.
Hold a test out
Hold out from Agent Builder keeps a test's contents out of what the Agent Builder can read: it sees only whether the test passed. When Builder proposes a fix, held-out tests are the ones it could not have written the fix to satisfy, so a fix that also passes them is more likely to be a real improvement.
Runs
The Runs tab lists every run, newest first, with its result. Open one to see:
- what it tested, how many times each test ran, which model answered, and what it cost;
- each test's result, with every one of its runs - open a run to see what the agent did;
- What changed - which checks started passing or failing since an earlier run. It compares with the run before by default; choose another under Compared with.
A run's totals are counted from the tests inside it, so the two can never disagree.
Require passing tests to publish
Turn on Require passing tests to publish, on the Runs tab, and the Agent refuses to publish unless a run tested exactly the changes you are publishing and at least one test passed with none failed.
The match is exact: a run of changes you have edited since does not count, because it did not test what you are about to ship. Requiring one test to pass means a run of entirely unscored tests cannot let a publish through.
It is off by default, by design. Requiring a pass while your tests are still new tends to teach people to delete checks.
What a run costs
Runs spend real money: they call the model, and a live test uses your real tools. Simulated tests and tests saved from a conversation use no tools. Running each test several times multiplies the cost by that number. That spend is billed like any other, and it is kept out of your usage statistics and cost reports, which answer what your automations did rather than what you were testing. Each run reports its own cost, so the two never have to be untangled.