Accuracy degrades under realistic input noise, and degrades faster for argument filling than for tool selection. STT-style noise (homophones, no punctuation, fillers, lowercasing) hurts more than character-level typos.
Paired design: 120 noisy tasks (60 character-typo variants, 60 STT-style variants) of 60 standard parent tasks from needle-argument-complexity, run once each against their native catalogs on cactus-needle 2.0.5 (local CPU, deterministic decoding, per needle-determinism-and-paraphrase-sensitivity). Each noisy record was paired against its clean parent's existing record rather than re-run, since determinism makes the old record a valid control.
Fields were split into free_text (title, message, text, and similar) versus constrained (int/float/bool/enum/date_string), reusing the classification from the argument-complexity experiment. Scored: paired tool-selection accuracy, paired exact-argument accuracy (restricted to pairs where both clean and noisy runs picked the correct tool), per-field accuracy by kind, confidence shift, and a manual failure-anatomy pass on clean-right-to-noisy-wrong flips.
PARTIALLY CONFIRMED
Tool selection barely moves under noise. Argument filling collapses.
Tool-selection accuracy (paired, n=120):
| group | clean_acc | noisy_acc | shift |
|---|---|---|---|
| overall | 0.817 | 0.833 | +0.016 |
| typo | 0.817 | 0.800 | -0.017 |
| stt | 0.817 | 0.867 | +0.050 |
Exact-argument accuracy (pairs where both sides picked the correct tool, n=90):
| group | clean_acc | noisy_acc | drop |
|---|---|---|---|
| overall | 0.733 | 0.444 | -0.289 |
| typo | 0.721 | 0.465 | -0.256 |
| stt | 0.745 | 0.426 | -0.319 |
Per-field accuracy by kind:
| kind | clean_acc | noisy_acc | drop |
|---|---|---|---|
| free_text | 0.974 | 0.553 | -0.421 |
| constrained | 0.870 | 0.792 | -0.078 |
Splitting further by noise type: free-text fields drop 33.3pt under typo versus 50.0pt under STT; constrained fields drop 2.7pt under typo versus 12.5pt under STT. STT is worse than typo in both field kinds.
Of the 58 clean-right-to-noisy-wrong flips, 50 (86%) are argument-value flips and only 6 (10%) are tool-choice flips — an 8:1 ratio. When noise breaks a previously correct call, it overwhelmingly breaks an argument value, not the tool itself.
A representative STT failure, noisy_stt_010 (parent std_019): the clean query "Translate 'good morning' into Spanish" filled text="good morning". The noisy transform inserted a filler mid-phrase — "translate good you know, morning into spanish" — and the model copied the filler verbatim into the argument: text="good you know, morning". Confidence dropped from 0.614 to 0.166. A matching typo case on the same parent, noisy_typo_010 ("Transllate 'ogod morning' into Spanish"), filled text="ogod morning" — reproducing the typo character-for-character instead of normalizing it, unlike the canonical-form normalization the model applies to dates and times elsewhere in the dataset.
Confidence fell under noise even net of correctness: mean shift -0.193 (typo -0.205, stt -0.181, both n≈60) across engine-successful pairs. The model's own confidence tracks the input getting harder, independent of whether it still lands on the right answer.
The mechanism is a free-text versus constrained-field split, not a noise-type split. When the model has to reproduce or reconstruct a string under noise, it mostly fails to normalize the noise away and instead copies it forward. When it only has to select from a small fixed set of legal values, noise barely registers — the model still recognizes the underlying intent even when the surface form is garbled. Tool selection sits close to this fixed-value regime: it is a choice among a bounded catalog, not a reconstruction task, and the results treat it that way, moving by only a couple of points in either direction depending on noise type.
One tool_choice flip stands out as a genuine intent break rather than argument corruption: noisy_typo_022 (parent std_043), where two typos landing inside a quoted search term ("soudough starter troubleshootting") pushed the model from search_web (confidence 0.951 clean) to find_nearby (confidence 0.006 noisy). This suggests tool-selection robustness holds on average but is not uniform — noise concentrated inside a value that also carries intent signal can still tip the classification.
- Run the same noisy-input design against a stronger and a weaker model to check whether the free-text-versus-constrained split is architecture-dependent or a general property of tool-calling under noise.
- Test whether explicit normalization instructions in the system prompt ("correct obvious typos and disfluencies before filling arguments") close the free-text gap, since the model already normalizes canonical date/time forms but not free text.
- Isolate STT filler words as their own noise dimension (fillers only, no lowercasing or homophone substitution) to see how much of the STT-versus-typo gap on free-text fields is filler contamination specifically.
- Investigate the noisy_typo_022 tool-choice flip class further: does typo density inside a quoted argument value predict tool-selection failure, distinct from typo density elsewhere in the query?