Capability belongs to a configuration
Agent performance depends on the model and its operating conditions: instructions, tool schemas, orchestration, permissions, context, budgets, and environment. If you change the model and the harness at the same time, you cannot tell which change caused the result. Record the configuration alongside each run.
Start with tasks you can check
Make a small set of representative tasks. Each case should include a starting state, a task prompt, allowed actions, a success condition, and a validator where possible. A validator can check an artifact directly—for example, whether a test passes or a JSON object matches a schema—instead of asking another model whether the answer “looks good.” Use human review for qualities that cannot be measured reliably by code.
| Input | Success check | Trace to inspect |
|---|---|---|
| Fix a failing test in a sandboxed repository | Target test passes and unrelated files are unchanged | File reads, patch, test command, tool errors |
Measure outcomes and process
Track at least task success and failure categories. Add process measures that explain the result: steps and tool calls, invalid calls, retries, timeout rate, latency, and token or monetary cost. A faster harness that silently skips validation may be worse; report quality and cost together rather than compressing every tradeoff into one score.
Make comparisons fair
- Choose one variable to change, such as a tool schema or stopping rule.
- Keep model, task cases, environment, permissions, and budgets fixed.
- Run both configurations on the same cases; repeat runs if the model is stochastic.
- Inspect failures and trajectories, not only the average score.
- Hold some cases back while tuning so you do not optimize only for the examples you already know.
Harness-Bench evaluates 106 sandboxed tasks and reports results from 5,194 execution trajectories. It finds substantial variation across model–harness pairings. That is evidence for measuring the pair, not proof that one harness is best for every application. SWE-agent likewise makes the agent’s interface to its coding environment part of the system under study.
Ask your teacher to help turn one of your real tasks into a test case.