SCRATCHPADS-Experiment

Builds on the runtime feasibility and determinism findings. Uses the nested catalog tiers from the task dataset construction run: core_5 subset core_10 subset core_25 subset core_50.

Model: cactus-needle 2.0.5, local CPU inference. Harness: shared needle_harness (runner/scoring/catalog/tasks modules).

Two task framings, 240 total (task, catalog) pairs, one call each, no repeats (inference confirmed deterministic in a prior run):

• Matched subset: the 16 tasks whose gold tool lives in core_5, cloned to run at all four sizes (16 x 4 = 64 tasks). Holds query identity constant across sizes.
• All-eligible set: every remaining standard task cloned to every catalog size at or above its native tier. n grows with size (16 / 32 / 72 / 120), so the query mix differs per size, not just the catalog.

Metrics per size: tool-selection accuracy, exact-argument accuracy, per-field argument accuracy, JSON validity, false-abstention rate, wrong-tool rate and target breakdown, confidence distribution, and wall_ms / prefill_tps / decode_tps from raw records. The run was the sole experiment on the machine, but run_batch's own 4 worker subprocesses executed all four catalog sizes concurrently, competing for the same CPU cores — a latency confound noted below.

REJECTED (as stated) — but a clean secondary effect emerged that the hypothesis didn't ask about.

Neither half of the original claim holds. Accuracy does not decline monotonically with size, and wall-clock latency does not rise with size.

Tool-selection accuracy by catalog size:

Size	Matched (n=16)	All-eligible (n varies)
core_5	13/16 = 0.8125	13/16 = 0.8125
core_10	14/16 = 0.875	30/32 = 0.9375
core_25	16/16 = 1.000	62/72 = 0.8611
core_50	15/16 = 0.9375	101/120 = 0.8417

The matched subset peaks at core_25, not core_5. The all-eligible set peaks at core_10, and core_5 — the smallest catalog — has the second-lowest accuracy of the four, below core_25. Neither framing shows the predicted decline.

Latency: wall_ms rises from core_5 to core_10, then falls at core_25 and core_50 in both subsets (matched: 2811 to 5798 to 5616 to 4888 ms). The engine's own prefill_tps, a more trustworthy per-size signal since it isn't confounded by the harness's worker concurrency, rises monotonically with catalog size instead (matched: 60.6 to 170.3 to 172.3 to 173.1 tok/s; all-eligible: 60.6 to 166.5 to 237.6 to 263.3 tok/s). Bigger catalogs make the engine faster per token on this run, not slower — the opposite of the "prefill cost grows with catalog size" prediction.

What did shift cleanly with size is the composition of errors:

Size	Total errors (all-eligible)	Wrong-tool share	False-abstention share
core_5	3/16	0.0%	100.0%
core_10	2/32	50.0%	50.0%
core_25	10/72	80.0%	20.0%
core_50	19/120	73.7%	26.3%

At core_5, every error is a false abstention. By core_25 and core_50, roughly three-quarters of errors are wrong-tool calls. This tracks the dataset design directly: core_5's five tools have no confusable partner inside the catalog, so there's nothing to misfire on besides declining to act. core_25 and core_50 add the nine built-in near-name-collision pairs (turn_on/off_light, lock/unlock_door, add_reminder vs create_calendar_event/set_recurring_reminder, call_contact vs video_call/play_music, adjust_brightness vs set_light_color, check_security_camera vs get_thermostat_status, play_podcast vs play_music), and every single wrong-tool miss at core_25/core_50 lands on exactly one of those pairs — none is a random or unrelated tool.

A separate, recurring failure sits outside the size effect: "Send a text to Dad saying dinner's at 7." abstained identically at both core_5 and core_10, echoing the messaging-abstention anomaly from the determinism experiment (q03, "send/text ... saying ..." phrasing). That looks like a model-level phrasing quirk, not a catalog-size artifact, since it fails the same way regardless of which catalog carries it.

JSON validity held at 1.0 across all 240 responses at every size — no malformed output at any catalog scale. Exact-argument accuracy and per-field accuracy were noisy but non-monotonic too; the weakest fields (date, source_language, a two-value door enum) were weak wherever they appeared, not specifically worse at large catalogs.

1. Re-run with n_workers=1 (serial, one catalog size at a time) to remove the worker-concurrency confound on wall_ms and check whether prefill_tps still dominates the latency story once that confound is gone.
2. Build a matched-subset variant that isolates confusable-pair introduction directly — same 16 queries, but catalogs constructed to add confusable partners one pair at a time, to test whether wrong-tool rate tracks pair count rather than raw catalog size.
3. Push past core_50 (core_100, core_200) to see whether the prefill throughput gain keeps climbing or whether a size threshold eventually reverses it, and whether accuracy stays flat or finally shows a real decline.
4. Target the recurring "send/text ... saying ..." abstention directly, isolated from catalog-size runs, to confirm it as a deterministic phrasing-level failure across a larger set of messaging paraphrases.