SCRATCHPADS-Experiment

Shared build for five planned Needle 2 experiments (catalog-size scaling, argument complexity, no-tool abstention, confidence calibration, noisy input), so their numbers stay comparable.

Tool pool: 65 hand-written ToolSpecs modeled on real smart-home, phone/assistant, and utility APIs. 22 enum parameters, 9 confusable pairs (set_alarm/set_timer, send_sms/send_email, lock_door/unlock_door, get_weather/get_forecast, and five others). Description length is mixed by design: most tools get one line, about a dozen get 2-3 sentences that spell out an implementation nuance (e.g. set_thermostat notes it stores Celsius internally regardless of the unit argument).

Catalogs: 7 total. Four nested size tiers (core_5/core_10/core_25/core_50, each a strict subset of the next, domains mixed at every prefix rather than grouped) plus three catalogs for the no-tool experiment (near_miss_smarthome, near_miss_comms built entirely from confusable pairs; unrelated_finance_travel off-topic to most queries). Nesting and the held-out 15 tools were verified programmatically.

Task records: 470 across four files — 120 standard tasks tier-tagged 1-3 by argument count, 40 + 40 no-tool tasks (unrelated vs. near-miss, with the tempting wrong tool named in notes), 120 noisy-input tasks (one typo + one STT-style corruption per standard-task parent), 150 paraphrase tasks (5 hand-written variants each for 30 parents). A normalization policy (NORMALIZATION.md) was written before authoring and amended twice after the pilot. validate_dataset.py cross-checks every task against its catalog — gold tool present, argument keys match declared params, required params present, enum values valid — and all 470 pass.

Pilot: 40 tasks sampled stratified across all four categories and all three standard-task tiers, run serially against the real cactus-needle==2.0.5 engine on CPU. Every mismatch was manually classified as either a genuine model error or a label/normalization bug. No controls — this is label quality control, not an experiment.

CONFIRMED

The dataset validates internally with zero errors and the pilot found exactly two systematic label bugs, both fixed dataset-wide and re-validated. No other mismatch turned out to be a dataset defect.

Metric	Result
Tool-selection accuracy	22/30 (73.3%)
Argument accuracy (tool-correct calls)	14/22 (63.6%), up from 13/22 pre-fix
Abstention on no-tool tasks	10/10 (100%), 0 false calls

Label fixes applied:

1. get_directions article-stripping. The engine strips leading "the"/"my" from origin/destination (e.g. "my hotel" to "hotel"), which is correct grounding behavior, not an error. 13 gold values fixed across the standard, noisy, and paraphrase files, scoped only to those two fields — other free-text fields keep their articles since there the article is literal content.
2. STT-variant casing. The STT noise transform lowercases the whole query, so a Title-Case gold value copied from a standard-task parent could never match. 31 string-valued gold args lowercased across all 60 STT-noise tasks.

After these fixes, every remaining mismatch (8 tool-selection, 8 argument, out of 40 pilot tasks) was manually classified as genuine engine behavior rather than a labeling problem. The recurring patterns are worth carrying into the downstream experiments rather than re-diagnosing: the engine invents values for unmentioned optional parameters but flags them ungrounded (a candidate signal for confidence calibration); it failed a straightforward minutes-to-seconds conversion while reporting low confidence on that same call; all 9 confusable pairs produced at least one genuine confusion under clean input; two typo cases flipped to the wrong tool entirely, and an inserted filler word split a compound action ("video call") into the wrong tool under STT noise; and a combined date+time field was mishandled two different ways on two different tasks, with no single canonical form that would have caught both — reserved for argument complexity rather than folded into the normalization policy.

One result stood out as a genuine data point rather than noise: a to-do-list query with unusually high wording overlap against its target tool still triggered an explicit no-call in the 25-tool catalog, suggesting false-negative abstention is not confined to sparse catalogs.

1. Run catalog-size scaling across the four nested core tiers using this pool.
2. Run argument complexity, with the date+time inconsistency as a specific case to track.
3. Run no-tool abstention on the near-miss and unrelated catalogs, checking whether the 25-tool false-negative case generalizes.
4. Run noisy input robustness and confidence calibration using the ungrounded flag as a starting signal for over-filled optional parameters.