The confidence head is calibrated — accuracy rises monotonically with confidence threshold, and expected calibration error (ECE) is low.
Pooled analysis, no new inference. Inference was already confirmed deterministic in needle-determinism-and-paraphrase-sensitivity, so every raw record from five prior runs is a reusable sample: needle-catalog-size-scaling (240), needle-argument-complexity (270), needle-abstention-no-tool-queries (120), and two consistency-series files (200 + 120). 950 raw records, deduplicated by (query, catalog_id), collapsed to 593 unique pooled points.
Each point scored two ways: tool-level correctness (right tool, or correctly abstained) and full-call correctness (tool-level AND exact argument match). Outcome classes reused the 5-way taxonomy from the abstention scratchpad.
Computed per correctness definition, over the full pool and three slices (by call-vs-abstain, by source experiment, and a pooled escalation scan):
- Reliability diagram: 10 equal-width confidence bins, mean confidence vs empirical accuracy per bin.
- ECE: bin-count-weighted mean absolute gap between confidence and accuracy.
- Accuracy-vs-threshold curve, 21 points from 0.00 to 1.00.
- 0.0-floor analysis: fraction of points at or near zero confidence, their accuracy, and a 20-bin histogram for bimodality.
- Escalation simulation: for thresholds 0.1-0.9, fraction of calls escalated (confidence below threshold) and error rate among calls kept on-device.
REJECTED
Neither monotonicity nor low ECE holds. Only a coarse three-way ordering (low bins < mid bins < top bin) survives.
Overall accuracy and ECE (593 pooled points):
| correctness definition | accuracy | ECE |
|---|---|---|
| tool-level | 0.8297 | 0.1782 |
| full-call (tool + args) | 0.6560 | 0.1310 |
Both ECE values sit well above the 0.05-0.10 range typically called "well-calibrated." The reliability diagram is non-monotonic at the bin level for both definitions: accuracy dips at the [0.4, 0.5) bin in both curves, and again across [0.6, 0.8) for full-call correctness, before climbing at the top bin.
Confidence distribution is bimodal, not continuous. The first 0.05-wide bin holds 12.1% of the pool (72/593 points); the last 0.05-wide bin holds 29.2% (173/593). Everything between is a trough. A single linear threshold has to cross that trough, which is where the reliability diagram's local dips sit.
| metric | value |
|---|---|
| n at exactly 0.0 confidence | 19/593 (3.20%) |
| n at or below 0.01 | 49/593 (8.26%) |
| accuracy at 0.0, tool-level | 0.3158 |
| accuracy at 0.0, full-call | 0.0000 |
| overall accuracy, tool-level (reference) | 0.8297 |
The zero-floor is disproportionately wrong on average but not a hard gate — nearly a third of exactly-0.0-confidence calls are still tool-correct, consistent with the 0.0-floor-with-a-call-still-emitted behavior seen in needle-runtime-feasibility.
Escalation simulation (escalate below threshold, keep at or above):
| threshold | escalated fraction | kept error, tool-level | kept error, full-call |
|---|---|---|---|
| 0.1 | 13.32% | 12.26% | 27.04% |
| 0.3 | 18.72% | 11.41% | 25.73% |
| 0.5 | 26.98% | 10.16% | 23.79% |
| 0.7 | 43.84% | 9.01% | 19.82% |
| 0.9 | 64.25% | 4.25% | 9.43% |
No threshold from 0.1 to 0.9 hits under-5% kept-error at under-50% escalation, for either correctness definition. Tool-level error only drops below 5% at threshold=0.9, where 64.25% of traffic is already escalated. Full-call error never drops below 5% anywhere in the scan; its floor is 9.43%, also at threshold=0.9.
Accuracy and ECE also shift by source experiment: tool-level accuracy ranges from 0.6867 (argument-complexity tasks) to 0.9469 (paraphrase tasks), and ECE ranges from 0.1602 to 0.2830 across sources with n≥80. Pooling five experiments is not sampling one stationary distribution — task mix changes the calibration profile, so a threshold tuned on one mix over-trusts the model on a harder one.
The direction of miscalibration is not consistent either: confidence understates full-call accuracy in the [0.5, 0.6) bin but overstates it across [0.6, 0.8). A uniform bias could be corrected by shifting the threshold; a bias that flips direction across the range cannot.
- Test whether a two-threshold policy (escalate below the floor trough, escalate above nothing, keep the rest) beats a single linear threshold on this same pooled dataset.
- Fit calibration on one source experiment's task mix and test transfer to a different mix, to quantify the cross-source miscalibration directly rather than inferring it from separate ECE numbers.
- Repeat the escalation simulation using full-call correctness as the deployment gate by default, since it is the metric that determines whether an automated action executes correctly, not just whether the right tool was named.
- Investigate whether the [0.4, 0.5) dip and the [0.6, 0.8) trough correspond to identifiable task properties (e.g. argument count, catalog size) rather than noise.