# Confidence Calibration for Edge-Cloud Escalation (cactus-needle 2.0.5)

> The confidence head is calibrated — accuracy rises monotonically with confidence threshold, and expected calibration error (ECE) is low.

Date: 2026-08-17



## Test

Pooled analysis, no new inference. Inference was already confirmed deterministic in [needle-determinism-and-paraphrase-sensitivity](/scratchpads/needle-determinism-and-paraphrase-sensitivity), so every raw record from five prior runs is a reusable sample: [needle-catalog-size-scaling](/scratchpads/needle-catalog-size-scaling) (240), [needle-argument-complexity](/scratchpads/needle-argument-complexity) (270), [needle-abstention-no-tool-queries](/scratchpads/needle-abstention-no-tool-queries) (120), and two consistency-series files (200 + 120). 950 raw records, deduplicated by `(query, catalog_id)`, collapsed to 593 unique pooled points.

Each point scored two ways: **tool-level** correctness (right tool, or correctly abstained) and **full-call** correctness (tool-level AND exact argument match). Outcome classes reused the 5-way taxonomy from the abstention scratchpad.

Computed per correctness definition, over the full pool and three slices (by call-vs-abstain, by source experiment, and a pooled escalation scan):
- Reliability diagram: 10 equal-width confidence bins, mean confidence vs empirical accuracy per bin.
- ECE: bin-count-weighted mean absolute gap between confidence and accuracy.
- Accuracy-vs-threshold curve, 21 points from 0.00 to 1.00.
- 0.0-floor analysis: fraction of points at or near zero confidence, their accuracy, and a 20-bin histogram for bimodality.
- Escalation simulation: for thresholds 0.1-0.9, fraction of calls escalated (confidence below threshold) and error rate among calls kept on-device.


## Result

**REJECTED**

Neither monotonicity nor low ECE holds. Only a coarse three-way ordering (low bins < mid bins < top bin) survives.

**Overall accuracy and ECE (593 pooled points):**

| correctness definition | accuracy | ECE |
|---|:-:|:-:|
| tool-level | 0.8297 | 0.1782 |
| full-call (tool + args) | 0.6560 | 0.1310 |

Both ECE values sit well above the 0.05-0.10 range typically called "well-calibrated." The reliability diagram is non-monotonic at the bin level for both definitions: accuracy dips at the [0.4, 0.5) bin in both curves, and again across [0.6, 0.8) for full-call correctness, before climbing at the top bin.

**Confidence distribution is bimodal, not continuous.** The first 0.05-wide bin holds 12.1% of the pool (72/593 points); the last 0.05-wide bin holds 29.2% (173/593). Everything between is a trough. A single linear threshold has to cross that trough, which is where the reliability diagram's local dips sit.

| metric | value |
|---|:-:|
| n at exactly 0.0 confidence | 19/593 (3.20%) |
| n at or below 0.01 | 49/593 (8.26%) |
| accuracy at 0.0, tool-level | 0.3158 |
| accuracy at 0.0, full-call | 0.0000 |
| overall accuracy, tool-level (reference) | 0.8297 |

The zero-floor is disproportionately wrong on average but not a hard gate — nearly a third of exactly-0.0-confidence calls are still tool-correct, consistent with the 0.0-floor-with-a-call-still-emitted behavior seen in [needle-runtime-feasibility](/scratchpads/needle-runtime-feasibility).

**Escalation simulation** (escalate below threshold, keep at or above):

| threshold | escalated fraction | kept error, tool-level | kept error, full-call |
|:-:|:-:|:-:|:-:|
| 0.1 | 13.32% | 12.26% | 27.04% |
| 0.3 | 18.72% | 11.41% | 25.73% |
| 0.5 | 26.98% | 10.16% | 23.79% |
| 0.7 | 43.84% | 9.01% | 19.82% |
| 0.9 | 64.25% | 4.25% | 9.43% |

No threshold from 0.1 to 0.9 hits under-5% kept-error at under-50% escalation, for either correctness definition. Tool-level error only drops below 5% at threshold=0.9, where 64.25% of traffic is already escalated. Full-call error never drops below 5% anywhere in the scan; its floor is 9.43%, also at threshold=0.9.

Accuracy and ECE also shift by source experiment: tool-level accuracy ranges from 0.6867 (argument-complexity tasks) to 0.9469 (paraphrase tasks), and ECE ranges from 0.1602 to 0.2830 across sources with n≥80. Pooling five experiments is not sampling one stationary distribution — task mix changes the calibration profile, so a threshold tuned on one mix over-trusts the model on a harder one.

The direction of miscalibration is not consistent either: confidence understates full-call accuracy in the [0.5, 0.6) bin but overstates it across [0.6, 0.8). A uniform bias could be corrected by shifting the threshold; a bias that flips direction across the range cannot.


## Next

1. Test whether a two-threshold policy (escalate below the floor trough, escalate above nothing, keep the rest) beats a single linear threshold on this same pooled dataset.
2. Fit calibration on one source experiment's task mix and test transfer to a different mix, to quantify the cross-source miscalibration directly rather than inferring it from separate ECE numbers.
3. Repeat the escalation simulation using full-call correctness as the deployment gate by default, since it is the metric that determines whether an automated action executes correctly, not just whether the right tool was named.
4. Investigate whether the [0.4, 0.5) dip and the [0.6, 0.8) trough correspond to identifiable task properties (e.g. argument count, catalog size) rather than noise.