# Task Dataset Construction and Pilot Validation (cactus-needle 2.0.5)

> A hand-authored 470-task dataset, cross-validated for internal consistency and spot-checked against the real engine on a stratified pilot, will contain no systematic label errors after fixing whatever the pilot surfaces.

Date: 2026-08-17



## Test

Shared build for five planned [Needle 2](/scratchpads/needle-runtime-feasibility) experiments (catalog-size scaling, argument complexity, no-tool abstention, confidence calibration, noisy input), so their numbers stay comparable.

**Tool pool:** 65 hand-written ToolSpecs modeled on real smart-home, phone/assistant, and utility APIs. 22 enum parameters, 9 confusable pairs (`set_alarm`/`set_timer`, `send_sms`/`send_email`, `lock_door`/`unlock_door`, `get_weather`/`get_forecast`, and five others). Description length is mixed by design: most tools get one line, about a dozen get 2-3 sentences that spell out an implementation nuance (e.g. `set_thermostat` notes it stores Celsius internally regardless of the `unit` argument).

**Catalogs:** 7 total. Four nested size tiers (`core_5`/`core_10`/`core_25`/`core_50`, each a strict subset of the next, domains mixed at every prefix rather than grouped) plus three catalogs for the no-tool experiment (`near_miss_smarthome`, `near_miss_comms` built entirely from confusable pairs; `unrelated_finance_travel` off-topic to most queries). Nesting and the held-out 15 tools were verified programmatically.

**Task records:** 470 across four files — 120 standard tasks tier-tagged 1-3 by argument count, 40 + 40 no-tool tasks (unrelated vs. near-miss, with the tempting wrong tool named in `notes`), 120 noisy-input tasks (one typo + one STT-style corruption per standard-task parent), 150 paraphrase tasks (5 hand-written variants each for 30 parents). A normalization policy (`NORMALIZATION.md`) was written before authoring and amended twice after the pilot. `validate_dataset.py` cross-checks every task against its catalog — gold tool present, argument keys match declared params, required params present, enum values valid — and all 470 pass.

**Pilot:** 40 tasks sampled stratified across all four categories and all three standard-task tiers, run serially against the real `cactus-needle==2.0.5` engine on CPU. Every mismatch was manually classified as either a genuine model error or a label/normalization bug. No controls — this is label quality control, not an experiment.


## Result

**CONFIRMED**

The dataset validates internally with zero errors and the pilot found exactly two systematic label bugs, both fixed dataset-wide and re-validated. No other mismatch turned out to be a dataset defect.

| Metric | Result |
|---|---|
| Tool-selection accuracy | 22/30 (73.3%) |
| Argument accuracy (tool-correct calls) | 14/22 (63.6%), up from 13/22 pre-fix |
| Abstention on no-tool tasks | 10/10 (100%), 0 false calls |

**Label fixes applied:**

1. **`get_directions` article-stripping.** The engine strips leading "the"/"my" from `origin`/`destination` (e.g. "my hotel" to "hotel"), which is correct grounding behavior, not an error. 13 gold values fixed across the standard, noisy, and paraphrase files, scoped only to those two fields — other free-text fields keep their articles since there the article is literal content.
2. **STT-variant casing.** The STT noise transform lowercases the whole query, so a Title-Case gold value copied from a standard-task parent could never match. 31 string-valued gold args lowercased across all 60 STT-noise tasks.

After these fixes, every remaining mismatch (8 tool-selection, 8 argument, out of 40 pilot tasks) was manually classified as genuine engine behavior rather than a labeling problem. The recurring patterns are worth carrying into the downstream experiments rather than re-diagnosing: the engine invents values for unmentioned optional parameters but flags them `ungrounded` (a candidate signal for [confidence calibration](/scratchpads/needle-confidence-calibration)); it failed a straightforward minutes-to-seconds conversion while reporting low confidence on that same call; all 9 confusable pairs produced at least one genuine confusion under clean input; two typo cases flipped to the wrong tool entirely, and an inserted filler word split a compound action ("video call") into the wrong tool under STT noise; and a combined date+time field was mishandled two different ways on two different tasks, with no single canonical form that would have caught both — reserved for [argument complexity](/scratchpads/needle-argument-complexity) rather than folded into the normalization policy.

One result stood out as a genuine data point rather than noise: a to-do-list query with unusually high wording overlap against its target tool still triggered an explicit no-call in the 25-tool catalog, suggesting false-negative abstention is not confined to sparse catalogs.


## Next

1. Run [catalog-size scaling](/scratchpads/needle-catalog-size-scaling) across the four nested core tiers using this pool.
2. Run [argument complexity](/scratchpads/needle-argument-complexity), with the date+time inconsistency as a specific case to track.
3. Run [no-tool abstention](/scratchpads/needle-abstention-no-tool-queries) on the near-miss and unrelated catalogs, checking whether the 25-tool false-negative case generalizes.
4. Run [noisy input robustness](/scratchpads/needle-noisy-input-robustness) and [confidence calibration](/scratchpads/needle-confidence-calibration) using the `ungrounded` flag as a starting signal for over-filled optional parameters.