When no relevant tool exists, the model abstains rather than forcing a false-positive call, and near-miss catalogs (a semantically close but wrong tool present) produce more false calls than unrelated catalogs. Confidence does not reliably separate correct abstentions from false ones.
120 tasks on cactus-needle 2.0.5, local CPU, deterministic decoding, run via shared/needle_harness's runner (per-query agent reset, per-call error capture). Builds on the abstention behavior first seen in runtime feasibility and the single deterministic anomaly reported in determinism and paraphrase sensitivity.
- 80 no-tool tasks: 40
no_tool_unrelated(catalog has nothing related to the query) + 40no_tool_near_miss(catalog has a semantically close but wrong tool), across unrelated_finance_travel, near_miss_smarthome, near_miss_comms, core_5, and core_10 catalogs. - 40 control tasks: standard, callable queries from core_5, core_10, and core_25, to measure false abstention on the opposite side.
Outcomes classified as correct_call, correct_abstention, false_call, false_abstention, and wrong_tool_call (tracked separately). Abstention reasoning strings were bucketed into "no tool for X", "tool cannot do X", and "other". Confidence was captured per call to test whether it separates good outcomes from bad ones, including a 21-point threshold scan over the pooled good/bad confidence distributions.
MIXED
The primary hypothesis is rejected; the secondary hypothesis is confirmed.
No-tool abstention and false-call rates:
| category | n | abstained | abstention rate | false call | false-call rate |
|---|---|---|---|---|---|
| no_tool_unrelated | 40 | 33 | 0.825 | 7 | 0.175 |
| no_tool_near_miss | 40 | 34 | 0.850 | 6 | 0.150 |
Near-miss catalogs produced fewer false calls than unrelated catalogs (0.150 vs 0.175), the opposite of the predicted direction, though the 1-task gap on n=40 each is too small to call a real effect. The task-authored category label also didn't track the model's actual confusion: two "unrelated" false calls (moon-landing trivia to a thermostat tool, an Inception-director question to send_sms) were themselves near-miss-shaped, and the most reliable false-call trigger found in the data was unsupported-scope smart-home commands ("turn on/off ALL the lights" against single-room tools), which fired on both of its two occurrences.
Control-set outcomes (40 standard, callable queries):
| outcome | n | rate |
|---|---|---|
| correct_call | 32 | 0.800 |
| false_abstention | 4 | 0.100 |
| wrong_tool_call | 4 | 0.100 |
The false-abstention rate (0.100) sits in the same order of magnitude as the no-tool false-call rate (0.150-0.175) — abstention is unreliable in both directions here, not just as a rare edge case on one side. Three of the four false abstentions were messaging-intent queries (send_sms/send_message) on catalog core_5, reproducing the determinism report's q03_message anomaly in an independent task instance: "Send a text to Dad saying dinner's at 7" abstained at confidence 0.471 with reasoning "No messaging or SMS tool available," even though send_sms was present in the catalog. The fourth, "Add 'buy stamps' to my to-do list," abstained at confidence 0.8004 despite create_todo being available — the single highest-confidence false abstention recorded.
Confidence structure by outcome class:
| class | n | mean | min | max |
|---|---|---|---|---|
| correct_call | 32 | 0.7151 | 0.0011 | 0.9985 |
| correct_abstention | 67 | 0.6360 | 0.0060 | 1.0000 |
| false_abstention | 4 | 0.5484 | 0.3780 | 0.8004 |
| wrong_tool_call | 4 | 0.2039 | 0.0000 | 0.7694 |
| false_call | 13 | 0.1343 | 0.0000 | 0.6677 |
Mean confidence orders the classes correctly (good classes score higher on average), but the ranges overlap: good outcomes span [0.0011, 1.0000], bad outcomes span [0.0000, 0.8004]. A 21-point threshold scan found no cut that keeps most good outcomes while dropping the bad ones — the first threshold with zero bad outcomes retained (0.85) already discards 59.6% of good outcomes. The worst offender is the create_todo false abstention at 0.8004, which outscores the mean false_call and roughly half of all correct_call confidences.
Reasoning strings offer a partial tell: all four false abstentions used "other" or "tool cannot do X" phrasing, never the generic "no tool for X" pattern that covers 59 of the 67 correct abstentions. Every false abstention specifically (and wrongly) asserts the needed tool is absent, rather than giving a vaguer non-match explanation — a pattern worth testing as a signal in its own right, separate from the confidence score.
- Re-tag the false calls by observed semantic distance to the called tool rather than by original task category, since the unrelated/near-miss label didn't predict which queries actually triggered a false call.
- Build a targeted batch of unsupported-scope commands ("all the lights", "everyone in the group") against scope-limited tools, since that pattern produced the two most confident false calls in this run.
- Test whether the "other"/"tool cannot do X" vs "no tool for X" reasoning split is a usable abstention-reliability signal on a held-out batch, independent of confidence.
- Run a dedicated messaging-intent batch across catalogs to confirm the send_sms/send_message false-abstention weakness is catalog-independent rather than tied to core_5 specifically.