# Tool-Selection Accuracy and Latency vs Catalog Size (cactus-needle 2.0.5)

> Tool-selection accuracy degrades as the tool catalog grows from 5 to 50 tools, and per-call latency grows with catalog size due to prefill cost.

Date: 2026-08-17



## Test

Builds on the [runtime feasibility](/scratchpads/needle-runtime-feasibility) and [determinism](/scratchpads/needle-determinism-and-paraphrase-sensitivity) findings. Uses the nested catalog tiers from the [task dataset construction](/scratchpads/needle-task-dataset-construction) run: core_5 subset core_10 subset core_25 subset core_50.

Model: cactus-needle 2.0.5, local CPU inference. Harness: shared needle_harness (runner/scoring/catalog/tasks modules).

Two task framings, 240 total (task, catalog) pairs, one call each, no repeats (inference confirmed deterministic in a prior run):

- **Matched subset**: the 16 tasks whose gold tool lives in core_5, cloned to run at all four sizes (16 x 4 = 64 tasks). Holds query identity constant across sizes.
- **All-eligible set**: every remaining standard task cloned to every catalog size at or above its native tier. n grows with size (16 / 32 / 72 / 120), so the query mix differs per size, not just the catalog.

Metrics per size: tool-selection accuracy, exact-argument accuracy, per-field argument accuracy, JSON validity, false-abstention rate, wrong-tool rate and target breakdown, confidence distribution, and wall_ms / prefill_tps / decode_tps from raw records. The run was the sole experiment on the machine, but run_batch's own 4 worker subprocesses executed all four catalog sizes concurrently, competing for the same CPU cores — a latency confound noted below.


## Result

**REJECTED** (as stated) — but a clean secondary effect emerged that the hypothesis didn't ask about.

Neither half of the original claim holds. Accuracy does not decline monotonically with size, and wall-clock latency does not rise with size.

**Tool-selection accuracy by catalog size:**

| Size | Matched (n=16) | All-eligible (n varies) |
|------|:-:|:-:|
| core_5  | 13/16 = 0.8125 | 13/16 = 0.8125 |
| core_10 | 14/16 = 0.875  | 30/32 = 0.9375 |
| core_25 | 16/16 = 1.000  | 62/72 = 0.8611 |
| core_50 | 15/16 = 0.9375 | 101/120 = 0.8417 |

The matched subset peaks at core_25, not core_5. The all-eligible set peaks at core_10, and core_5 — the smallest catalog — has the second-lowest accuracy of the four, below core_25. Neither framing shows the predicted decline.

**Latency:** wall_ms rises from core_5 to core_10, then falls at core_25 and core_50 in both subsets (matched: 2811 to 5798 to 5616 to 4888 ms). The engine's own prefill_tps, a more trustworthy per-size signal since it isn't confounded by the harness's worker concurrency, rises monotonically with catalog size instead (matched: 60.6 to 170.3 to 172.3 to 173.1 tok/s; all-eligible: 60.6 to 166.5 to 237.6 to 263.3 tok/s). Bigger catalogs make the engine faster per token on this run, not slower — the opposite of the "prefill cost grows with catalog size" prediction.

**What did shift cleanly with size is the composition of errors:**

| Size | Total errors (all-eligible) | Wrong-tool share | False-abstention share |
|------|:-:|:-:|:-:|
| core_5  | 3/16  | 0.0%  | 100.0% |
| core_10 | 2/32  | 50.0% | 50.0%  |
| core_25 | 10/72 | 80.0% | 20.0%  |
| core_50 | 19/120 | 73.7% | 26.3% |

At core_5, every error is a false abstention. By core_25 and core_50, roughly three-quarters of errors are wrong-tool calls. This tracks the dataset design directly: core_5's five tools have no confusable partner inside the catalog, so there's nothing to misfire on besides declining to act. core_25 and core_50 add the nine built-in near-name-collision pairs (turn_on/off_light, lock/unlock_door, add_reminder vs create_calendar_event/set_recurring_reminder, call_contact vs video_call/play_music, adjust_brightness vs set_light_color, check_security_camera vs get_thermostat_status, play_podcast vs play_music), and every single wrong-tool miss at core_25/core_50 lands on exactly one of those pairs — none is a random or unrelated tool.

A separate, recurring failure sits outside the size effect: "Send a text to Dad saying dinner's at 7." abstained identically at both core_5 and core_10, echoing the messaging-abstention anomaly from the [determinism experiment](/scratchpads/needle-determinism-and-paraphrase-sensitivity) (q03, "send/text ... saying ..." phrasing). That looks like a model-level phrasing quirk, not a catalog-size artifact, since it fails the same way regardless of which catalog carries it.

JSON validity held at 1.0 across all 240 responses at every size — no malformed output at any catalog scale. Exact-argument accuracy and per-field accuracy were noisy but non-monotonic too; the weakest fields (date, source_language, a two-value door enum) were weak wherever they appeared, not specifically worse at large catalogs.


## Next

1. Re-run with n_workers=1 (serial, one catalog size at a time) to remove the worker-concurrency confound on wall_ms and check whether prefill_tps still dominates the latency story once that confound is gone.
2. Build a matched-subset variant that isolates confusable-pair introduction directly — same 16 queries, but catalogs constructed to add confusable partners one pair at a time, to test whether wrong-tool rate tracks pair count rather than raw catalog size.
3. Push past core_50 (core_100, core_200) to see whether the prefill throughput gain keeps climbing or whether a size threshold eventually reverses it, and whether accuracy stays flat or finally shows a real decline.
4. Target the recurring "send/text ... saying ..." abstention directly, isolated from catalog-size runs, to confirm it as a deterministic phrasing-level failure across a larger set of messaging paraphrases.