Dependency triage agent — evaluation report

Does tool routing actually help? Measured against two deliberately unintelligent baselines.

Final measurement. Run once, on a sealed task set.Split: held_out. These tasks were held back from the whole of development and executed exactly once, after the last change to anything the agent does. Nothing below was iterated against, and no re-run is possible: the reservation that permitted this execution is spent.
Two held-out probes (minsafe-deep-03, minsafe-deep-04) were run during development before the split was being honoured, and are permanently excluded. The sealed set is therefore 38 tasks, not 40, and that reduction is disclosed rather than absorbed.

Headline

System	Answer accuracy	Ungradable	Mean tool calls
Agent (tool routing)	92% 35/38 95% CI 79–97%	0	2.79 n=38
no_agent (calls every tool)	94% 34/36 95% CI 82–98%	2	4.95 n=38
always_lookup (one tool)	92% 35/38 95% CI 79–97%	0	1.58 n=38

The baselines are not weak opponents.no_agent calls every tool on every package and lets the model answer once, so it always holds strictly more information than the routing agent. Routing is therefore dominated whenever calling everything is affordable and fits in the context window. That is arithmetic, not a property of this task set, and the task set was deliberately not redesigned until the agent won.

Paired comparison

Same tasks, different systems, so the information is in the discordant pairs — tasks where exactly one system was right. Treating the two as independent proportions discards most of it.

Agent vs `always_lookup`

Wins	Ties	Losses	Discordant	Exact McNemar p	Excluded
2	34	2	4	1.00	0

Cannot resolve at this size. With 4 discordant pair(s) the smallest attainable two-sided p is 0.12; at least 6 are needed to reach p < 0.05 under any split of wins and losses. Reported as inconclusive, not as equivalence.

Agent vs `no_agent`

Wins	Ties	Losses	Discordant	Exact McNemar p	Excluded
2	31	3	5	1.00	2

Cannot resolve at this size. With 5 discordant pair(s) the smallest attainable two-sided p is 0.06; at least 6 are needed to reach p < 0.05 under any split of wins and losses. Reported as inconclusive, not as equivalence.

Under injected tool failure

Each tool result is corrupted one way per run: an HTTP 503, a valid-but-empty payload, or a record with the field the answer depends on removed. This is the one condition the baselines structurally cannot compete in — both are single-shot scripts and cannot retry, switch tool, or abstain.

System	error (503)	empty	malformed
Agent (tool routing)	50% 10/20 95% CI 30–70%	58% 11/19 95% CI 36–77%	90% 18/20 95% CI 70–97%
no_agent (calls every tool)	0 gradable 20 crashes	68% 13/19 95% CI 46–85%	100% 19/19 95% CI 83–100%
always_lookup (one tool)	0 gradable 20 crashes	50% 10/20 95% CI 30–70%	84% 16/19 95% CI 62–94%

The error arm is an operational difference, not an accuracy win.With zero gradable baseline records every pair is excluded, and zero usable pairs cannot support an accuracy comparison. What is directly observed is that the baselines crash and the agent does not.

Diagnostics

These explain the headline; they are not competitors to it. High tool recall alongside low answer accuracy would mean the agent gathered the right evidence and still concluded wrongly.

System	Tool recall	Tool precision	Negative conflation	Termination	Over-call ratio	Repeat calls
Agent (tool routing)	0.99 n=38	1.00 n=38	0% 0/14 95% CI 0–22%	95% 36/38 95% CI 83–99%	1.32 n=38	0% 0/38 95% CI 0–9%
no_agent (calls every tool)	1.00 n=38	0.51 n=38	0% 0/14 95% CI 0–22%	100% 38/38 95% CI 91–100%	2.95 n=38	0% 0/38 95% CI 0–9%
always_lookup (one tool)	0.75 n=38	1.00 n=38	0% 0/14 95% CI 0–22%	100% 38/38 95% CI 91–100%	0.95 n=38	0% 0/38 95% CI 0–9%

Tool recall measures agreement with the reference path, not logical necessity.Measured: 53 of 54 minimum-safe-version cases are answerable from the advisory tool alone, because advisories carry fix boundaries. A shorter correct path than the reference resolver’s exists, so this is not a minimality proof.

Fabrication

Every version string and advisory identifier in an answer is checked against the entire frozen snapshot. A token that appears nowhere in it was invented; one that exists but belongs to another package is misattributed.

System	Per assertion	Per task	Misattributed
Agent (tool routing)	≥ 0% 0/6 95% CI 0–39%	≥ 0% 0/38 95% CI 0–9%	0
no_agent (calls every tool)	≥ 10% 2/21 95% CI 3–29%	≥ 3% 1/38 95% CI 0–13%	2
always_lookup (one tool)	≥ 0% 0/13 95% CI 0–23%	≥ 0% 0/38 95% CI 0–9%	0

Both rates are lower bounds.An agent that implies a version without naming it is undetectable by string extraction. The detector itself is not validated against hand labels, so its own precision is unmeasured. Assertions within one task are dependent, so the per-assertion interval is descriptive only.

What this cannot tell you

Sample size. Twenty development tasks carry roughly ±15 percentage points at 95% confidence. More tasks is not the fix: eighty reaches only about ±11.
One model, one domain. Nothing here generalises beyond this model on Python dependency triage.
Reproducibility is partial. Greedy decoding with a fixed seed did not make every metric deterministic; one baseline is not repeatable run to run, and the observed variance is published rather than smoothed.
Correct under a fault is not verified recovery. A system can reach the right answer without having noticed the failure at all.

Rendered from committed evaluation artifacts for the held_out split. Model qwen3:8b, temperature 0.0, seed 42, step budget 8. No value on this page was produced at page-load time.