Test the work and the way it gets done
Engineering field guide · Evaluation · Both days
How do you know an agent is doing useful work?
Day 1 builds benchmarks from real tasks. Day 2 adds behavior specs and regression tests for decisions inside long agent runs.
All topic guides · Practical takeaways
Where the approaches meet
Section titled “Where the approaches meet”Bridgewater, Vals AI, and Basis all make quality explicit. Real tasks and examples of accepted work give you a standard that a generic model ranking cannot supply.
How the approaches differ
Section titled “How the approaches differ”| Session | Approach |
|---|---|
| Bridgewater · Day 1 | Measures retrieval with a small labeled question set. Precision at ten checks the usefulness of a search result list. |
| Vals AI · Day 1 | Replays completed tasks to compare success, cost, and speed. Partial credit and fully correct answers support different deployment decisions. |
| Basis · Day 2 | Adds behavior specs for the decisions inside a long run. These help inspect trajectories when you cannot label every possible final outcome. |
Bridgewater · Day 1 · Vals AI · Day 1 · Basis · Day 2
Where the evidence stops
Section titled “Where the evidence stops”A behavior check can explain a failure without proving the final answer is correct. Bridgewater’s reported search improvement and Vals’ spending reduction describe their own evaluations. Neither predicts your success rate.
Try this on your workload
Section titled “Try this on your workload”- Choose twenty completed tasks and keep their accepted outputs as references.
- Run two agent configurations on the same tasks. Grade final results against a written rubric.
- Inspect failed runs against one behavior spec, such as checking a source before making a claim.
Measure the result
Section titled “Measure the result”Record fully correct tasks, behavior violations, reviewer time, and cost per accepted result. Keep some tasks out of prompt tuning.
Read the source sessions
Section titled “Read the source sessions”- Bridgewater · Day 1. Our session notes · Official reading edition · Source video
- Vals AI · Day 1. Our session notes · Official reading edition · Source video
- Basis · Day 2. Our session notes · Source video
The comparison and experiment are our editorial synthesis. They are not a reported benchmark or the speakers’ prescribed procedure.