SCRATCHPADS-Experiment

270 tasks against cactus-needle 2.0.5 (local CPU inference, deterministic decoding, no repeats needed — determinism confirmed in needle-determinism-and-paraphrase-sensitivity): 120 standard tasks across three complexity tiers plus 150 paraphrase tasks (5 paraphrases each of 30 standard-task parents).

Tiers: tier 1 = single string arg, tier 2 = 2-3 mixed-type args, tier 3 = 4+ args including enums, dates, and optionals.

Per-field argument accuracy, the error taxonomy, and the ungrounded-predictor check are all scored only on tasks where the tool itself was correctly selected (370 scored field events across 200 tool-correct tasks), so wrong-tool cases don't double-penalize argument metrics.

Error taxonomy categories: wrong_value, hallucinated_optional (unmentioned optional filled anyway), out_of_enum (value outside the schema's enum list), missing_required, omitted_optional_specified, type_error.

PARTIALLY CONFIRMED

Tool selection and exact-argument accuracy by tier (exact-arg accuracy denominator is tool-correct tasks only):

Tier	n	Tool selection acc	Exact-arg acc
1	125	0.648	0.889
2	111	0.775	0.698
3	34	0.971	0.576

Tool-selection accuracy rises with tier, the opposite of what the hypothesis's first clause predicted. That rise is a composition artifact, not evidence complexity helps: tier-3 tools (book_flight, create_calendar_event, set_thermostat) are less confusable than the near-miss-heavy tier-1 catalog, and paraphrase wording specifically drags down tier-1 selection (paraphrase tier-1 tool accuracy 0.554 vs core tier-1 0.750) — a phrasing effect already documented separately, not a complexity effect.

Exact-argument accuracy falls monotonically with tier as hypothesized (0.889 → 0.698 → 0.576), but per-field macro accuracy does not fall — it rises slightly (0.750 → 0.767 → 0.875):

Tier	Events	Macro field acc	Micro field acc
1	77	0.750	0.883
2	186	0.767	0.844
3	107	0.875	0.813

Individual fields are filled about as well or better at tier 3. There are just more of them, so the joint probability of getting every field right in one call drops even though no single field got harder. This is the clean confirmation of the hypothesis's second half: argument-count multiplication, not per-field difficulty, drives tier 3's low exact-match rate.

By parameter type, accuracy is highest for bool (1.000, n=7) and float (0.958, n=24), lowest for str (0.822, n=191) and enum (0.844, n=77), with date_string mid-pack (0.870, n=23). Enum failures are mostly cosmetic: 4 of 9 out_of_enum errors are the model predicting the humanized form ("living room") instead of the catalog's snake_case token ("living_room") — it identified the right value, just not the right string.

Error taxonomy over 58 incorrect fields (of 370 scored):

Error type	Count
wrong_value	30
hallucinated_optional	19
out_of_enum	9
missing_required	0
omitted_optional_specified	0
type_error	0

The model never drops a required argument and never emits the wrong type for a field it does fill. Every error is about which value or which token, not whether or what shape. Hallucinated-optional errors cluster at tier 1-2 (7 and 12 cases) and vanish at tier 3 (0 cases) — tier-3 tools apparently leave less room to invent unmentioned optional values, or tier-3 queries tend to specify more of what they use.

Secondary hypothesis — REJECTED. validation.ungrounded as a per-argument error detector scores precision 0.600, recall 0.259 (15 true positives, 10 false positives, 43 false negatives, 302 true negatives). It misses three of every four real argument errors and is wrong two out of five times it does fire. All 10 false positives are exact-match-correct values the engine still flagged ungrounded (e.g. duration_seconds gold 300, predicted 300), confirming the "normalized-but-correct" failure mode is real, but that only explains the false-positive side — it does nothing for the much larger recall gap.

Paraphrase splits by argument kind: constrained fields (int/float/bool/enum/date_string) hold up much better under paraphrasing (0.932 aggregate accuracy) than free-text fields (0.752 aggregate). See needle-determinism-and-paraphrase-sensitivity for the tool-selection-level version of this finding; this run decomposes it down to the argument-kind level.

1. Test whether case/whitespace-normalized enum matching recovers most out_of_enum failures, since 4 of 9 are the same "living room" vs "living_room" formatting slip.
2. Run a dedicated multi-lingual argument test on translate_text, whose source_language/target_language fields scored 0.0 accuracy everywhere they appeared with n≥4 in this run.
3. Investigate the combined date+time field (create_calendar_event.date) specifically — it dropped the time component in 3 of 12 scored events, all at tier 2.
4. Re-run the ungrounded-as-predictor check at higher n to see whether precision improves once the normalized-but-correct false positives are filtered out structurally.