Use a test pyramid
- Unit tests: verify tool input validation, permissions, adapters, and state transitions with fakes.
- Integration tests: check that the model provider, MCP client, tools, and sandbox exchange the right messages.
- Task evaluations: replay realistic end-to-end tasks and validate their final artifacts and trajectories.
Each layer catches a different problem. Unit tests are fast and deterministic but cannot prove the model chooses the right tool. Integration tests catch wiring problems but may be costly. Task evaluations reveal system behavior, but need careful cases, validators, and trace review.
Write evaluation cases before tuning
For each case, save the starting state, task, allowed tools, environment version, expected outcome, and grading method. Include ordinary cases, edge cases, and adversarial or ambiguous inputs that matter to the product. Avoid leaking every evaluation case into the prompt or code: the agent may memorize the test rather than learn a robust behavior.
{
"id": "catalog-search-001",
"task": "Find three Python harnesses that expose MCP tools",
"fixture": "catalog-snapshot-2026-10",
"allowed_tools": ["search_harnesses"],
"checks": [
"returns_at_most_three_results",
"every_result_has_a_source_url",
"does_not_invent_catalog_entries"
]
}The exact grader depends on the task. Use deterministic assertions for format, tool permissions, file changes, or database state. Use a human rubric—or a carefully validated model grader—for qualities such as usefulness, and do not treat a model grader as ground truth without calibration.
Capture enough trace to diagnose the failure
Log a stable run ID; model and harness versions; prompt and tool-schema versions; task and environment IDs; tool calls, arguments, outcomes, and timing; final artifact; validator output; token use; and cost. Redact credentials and sensitive content. A final answer alone cannot tell you whether a tool failed, was skipped, or returned an irrelevant result.
Compare one change at a time
Run the same cases with the same model, budget, tools, permissions, and environment, changing only the component under study. Repeat stochastic runs and report variation. Track at least:
| Metric | What it reveals |
|---|---|
| Validated task success | Did the requested artifact satisfy the check? |
| Tool error / invalid-call rate | Could the agent select and use the interface reliably? |
| Turns and tool calls | How much interaction did completion require? |
| Latency and cost | What resources did a successful run consume? |
| Recovery rate | Could the system recover from a realistic tool failure? |
Never collapse these into one score without documenting the weighting. If two harnesses trade success for cost, the right choice depends on the task’s constraints. Your MCP verifier can return the component scores, failed cases, and run configuration so a calling agent can make a grounded recommendation.
Improve from failure categories
Group failures by cause: missing capability, ambiguous tool schema, wrong tool selection, invalid arguments, poor observation, unsafe permissions, context loss, timeout, or weak validation. Change one thing tied to the largest relevant failure category. Re-run the evaluation and retain the old result for comparison. This is a loop for engineering the harness, just as the model’s tool cycle is a loop for completing one task.
Ask your teacher to review a real evaluation case or help interpret its results.