Evaluate an agent before you change it
Build checks from an agent's past runs, fix the expected answers, and compare a change to its settings or prompt against how it works today. Regressions are listed first, and nothing is saved until you save it.
The Evaluate tab on an agent’s page tries a change on work the agent already did, before you save the change. You see which answers get worse, which get better and what the change would cost.
When you’d use this
- You want to switch an agent to a different model and need to know whether its answers hold up.
- You rewrote a custom agent’s prompt and want to see its effect before you publish it.
- You changed a setting, such as a threshold, and want to know which past cases now come out differently.
- Someone corrected the agent’s answers a few times and you want to keep those corrections as checks.
Finding it
Open the agent’s page and click the Evaluate tab:
- Sidebar → Agents, open an agent you turned on, then Evaluate.
- Sidebar → Agents → My agents → Custom, open the agent, then Evaluate. If the agent has more than one running copy, pick one under Running copy to check. The checks come from that copy’s runs.
A custom agent’s page also has Compare before publishing in its header. It opens Evaluate with A new prompt already chosen.
Anyone who can see the agent can see its checks and past comparisons. Building checks, editing them and running a comparison need permission to manage the agent. Everyone else sees “Ask an owner to run comparisons.”
Build checks from past runs
- On Evaluate, click Build from past runs.
- Choose How many runs (1 to 50) and Runs from: the last 7, 30 or 90 days.
- Leave Reviewed runs first on to use runs someone approved or corrected before the others.
- Optionally give the set a Name. Using a name you used before adds the new runs to those checks.
- Click Build checks.
Each run becomes one check: what the agent was given, and the answer it should give.
| Expected answer from | What it means |
|---|---|
| Approved, Edited, Corrected, Rejected | Someone reviewed the run. The expected answer is the one they approved or corrected |
| Not reviewed | Nobody reviewed the run. The expected answer is what the agent produced at the time |
| Edited here | You changed the expected answer on this tab |
The result says how many checks were added, how many were reviewed and not reviewed, and lists any run that was left out with the reason.
A Not reviewed check only tells you whether the answer changed, not whether it was right. Review or fix it if you want it to count as the correct answer.
Fix expected answers
Open a set of checks to see each one. For each check you can:
- Edit answer to say what a correct answer looks like.
- Untick Included to leave it out of comparisons.
- Remove it from the set. Comparisons already run keep their results, and the run it came from is not changed.
- Open the run to see what the agent did.
A check is Scored by one of two methods:
| Scored by | How a pass is decided |
|---|---|
| Match fields | The answer’s fields are compared with the expected fields. A pass needs: Every field the same, Listed fields the same, Listed fields present or Numbers close enough |
| Judged by AI | An AI reads the agent’s answer against What a correct answer says and decides whether it passes |
Compare a change
Open a set of checks, then go to Compare a change and choose What to try:
| Option | What it tries |
|---|---|
| Unsaved settings | The agent’s settings as it runs today, with your changes. Only what you change is tried. The model is a setting, so this is how you try a different model |
| A new prompt | Custom agents only. Starts from the prompt the agent uses today |
| A saved prompt version | Custom agents only. An older published version |
Before anything runs, the panel says “This will run N cases twice and use about X credits.” Click Compare. Each included check runs twice: once as the agent works today, and once with your change. Nothing is saved or sent, and no record is changed. The page updates on its own while the comparison runs.
Read the results
Results are grouped, with the cases that need your attention first:
| Group | What it means |
|---|---|
| Regressions | Passed today, failed with your change |
| Improvements | Failed today, passed with your change |
| Unchanged | Same result both ways |
| Not graded | No result on one side, so it cannot be compared |
Press a group’s count to show only that group.
Each side shows how many checks passed, for example 12/15 passed. A check is Not graded when the AI judge gave no verdict, or the agent could not run the case. Not graded checks are left out of the pass count and never counted as a regression.
The results also show what the comparison used, Used about X credits, and the time per case.
Suggested changes
When checks fail with your change, the results include Suggested changes, written from the checks that failed. Nothing is applied for you. Click Copy to take the text, then Open settings to apply it and save it yourself. Compare again before you save if you want to be sure.
When your corrections point to one change
When you give an agent’s answer a thumbs down and write What should it have done?, the correction is saved as a check on the agent’s Evaluate tab, in a set called Corrections. A second correction on the same run updates that check instead of adding another.
When three corrections on the agent’s current instructions say the same thing, the agent’s Overview shows Your corrections point to one change. Two similar corrections are not enough: the card waits for the third. Only agents you built get this card.
- See the change writes the one change to the instructions those corrections point to and shows it as a before and after. Nothing is saved yet.
- Check it on past cases reruns the corrected runs with the change, the same comparison as Compare a change. The card says how many it fixed and how many it broke. See every case opens the results here.
- Apply saves the change as a new version of the instructions. To undo it, open the agent’s Versions tab and click Roll back on the earlier version.
Suggest improvements
On the Overview of an agent you built, Suggest improvements looks for better instructions using the agent’s checks that have a reviewed answer: approved, edited, rejected, corrected or written by you. It needs at least 20 of them. It runs in the background, and the card shows “Looking for better instructions…” until it finishes.
The result is the change as a before and after, with how many of the checks passed as it is and with the change. Nothing changes until you click Apply, which saves a new version you can roll back from Versions. The search tries the instructions without the agent’s tools, so check the change on past cases before you apply it. It uses AI credits, listed as Quality checks.
Limits
- Up to 50 cases per comparison. With more included, Compare stays disabled until you untick some.
- One comparison of a set of checks runs at a time. Starting another shows “A comparison of these checks is already running.” with a link to it.
- A comparison runs only when you start it. Saving or publishing an agent does not start one.
- Comparisons use AI credits. They are listed as Quality checks on the Usage Breakdown tab. See AI Credits.
When it goes wrong
| What you see | Why | What to do |
|---|---|---|
| “Nothing to check yet” | The agent has not run yet | Run it once, then build checks from its runs |
| “Add cases first. This set has no included checks.” | Every check is unticked, or the set is empty | Tick Included on some checks, or build more |
| “Change at least one setting to compare.” | The settings are the same as today | Change a value, or add a setting |
| “Change the prompt to compare it.” | The prompt is the same as the one in use | Edit the prompt |
| “Up to 50 cases per comparison.” | Too many checks are included | Untick some checks |
| A card to add credits | This month’s AI credits are used up | Add credits, then compare again |
| “This comparison did not finish.” | The comparison stopped before every case ran. The reason follows the message | Fix what it names, then click Compare again |
| “These results could not be loaded” | The page could not reach the results | Click Try again |
| “Ask an owner to run comparisons.” | Your role can see the agent but not change it | Ask an owner or admin |
| “Suggest improvements needs at least 20 checks with a reviewed answer; this agent has N” | Too few checks have an answer a person gave | Correct more answers, or build checks from reviewed runs |
| “The search could not be trusted: N AI calls failed during it.” | AI calls failed during the search, so its result was thrown away | Try again later |
Next step
- Building a Custom Agent: the prompt, versions and Roll back.
- Turning On an Agent and Reading Its Runs: the agent’s settings and runs.
- AI Credits: what comparisons cost.