I ran eleven experiments on Needle 2 (cactus-needle 2.0.5), a small tool-calling model that runs locally on CPU, to find out where it breaks. Tool selection came out in the low eighties on most runs, but that number turned out to be the least useful thing I measured, because the model is deterministic: a query that fails will fail identically on every retry. So most of the series went into finding which queries fail and what they have in common.
The first one I found was "Send a message to Alice saying the meeting is at 3pm." With a send_message tool in the catalog, the model abstained, with reasoning "No messaging or communication tool is available" and confidence 0.9779, and it produced that same abstention on all 60 repeats across two serial runs and a parallel one. Rewordings of the same request worked, so the failure belongs to that specific phrasing, and variants of it turned up again in two later experiments.
Everything ran on a Windows machine, CPU inference, no API keys, around 400 ms per call. Eleven experiments and roughly 1,800 engine calls, all drawing on a purpose-built 470-task dataset over a 65-tool pool with nine deliberately confusable tool pairs and four nested catalog tiers (core_5 up to core_50). The dataset, including the noisy, paraphrase, and extraction subsets, is published on Hugging Face. Two of the eleven ran no new inference at all, for reasons covered in the next section. The feasibility probe and the dataset build have the setup detail, and each section links its scratchpad for the full data.
The first thing I checked is whether there's a sampling knob at all, and there isn't: none of the five public methods expose temperature, seed, or top-k, and those keywords only appear in a training code path that inference never touches. So instead of assuming determinism I measured it. Ten queries, twenty repeats each, over three separate runs (one split across four worker processes) produced exactly ten unique outputs, with confidence floats matching to four decimals. That's an observation about one build on one CPU rather than a guarantee, but it held over 600 calls.
With no sampling there's no retrying a bad answer, and the only remaining input that can change the output is the query text, so reliability testing here means paraphrase testing. Across 15 intents with 8 hand-written paraphrases each, tool agreement averaged 0.942 and argument agreement 0.892, with the worst intent at 0.625. The averages hide cases like the Alice message: five or six of the seven rewordings of that intent worked, and the one that failed was the plain imperative, while the deliberately awkward variants went through. A bad phrasing is easy to work around once you know about it, but nothing in the model's output points you at which phrasings to check, so the only way to find them is to test paraphrases in bulk.
Determinism also simplified the rest of the series. Repeat runs add no information, so their budget went to more paraphrases, and two experiments (calibration and the schema census) ran entirely on pooled records from earlier runs.
I expected tool selection to decay as the catalog grew from 5 to 50 tools, and it didn't: accuracy came back non-monotonic in both experimental arms (one holds the same 16 queries fixed across all four sizes and peaked at 25 tools, the other lets the task mix grow with the catalog), latency didn't rise either, and JSON validity sat at 1.0 across all 240 responses. What did change with size is the composition of the errors:
| Catalog | False-abstention share | Wrong-tool share |
|---|---|---|
| core_5 | 100% | 0% |
| core_10 | 50% | 50% |
| core_25 | 20% | 80% |
| core_50 | 26.3% | 73.7% |
The five-tool catalog produced three errors, all of them abstentions on queries it should have handled, which is the harmless direction. The 25- and 50-tool catalogs mostly produced wrong-tool calls instead, and those went to one of the nine near-name pairs in the dataset (turn_on_light/turn_off_light, lock_door/unlock_door, play_podcast/play_music, and so on) in every case, never to an unrelated tool. Since I designed those pairs into the pool myself, the fair reading is that the variable is confusable-pair density rather than catalog size as such. The two are correlated here, and probably in most real catalogs since new tools tend to overlap with existing ones, but if your fiftieth tool resembles nothing else in the catalog, this result gives you no reason to expect trouble from size alone. Full numbers in catalog size scaling.
Exact-argument accuracy drops with task complexity, from 0.889 on single-argument calls to 0.576 at four or more arguments, while per-field accuracy over the same calls goes up:
| Tier | Exact-argument accuracy | Per-field macro accuracy |
|---|---|---|
| 1 | 0.889 | 0.750 |
| 2 | 0.698 | 0.767 |
| 3 | 0.576 | 0.875 |
So individual fields get filled as well or better on the complex calls, and the exact-match number sinks anyway because a call with five fields has five chances for one of them to be wrong. Tier 3 has only 34 tasks, which makes the 0.576 itself shaky, but the gap between the two columns doesn't depend on the tier ordering.
Of 370 scored fields, 58 were wrong, and all 58 were content errors: wrong values (30), values invented for optional parameters the query never mentioned (19), and enum tokens formatted as English words (9). None was a missing required argument or a type error, and the pooled census over 893 deduplicated records from six experiments repeats that shape: zero nonexistent tools, zero missing required arguments, zero type errors, against 6.83% out-of-enum values and 1.12% invented argument names.
Most of the enum failures are the same substitution, room="living room" where the catalog wanted "living_room", which by itself covers 4.48% of all records in the pool. My scoring is textual and that cuts the other way too: alarm_0730 came back as 07:30, 7:30 am and 7:30 in the morning across paraphrases, all correct in substance and scored as disagreement. I still count both kinds as failures, since whatever consumes the call receives the wrong string, but they're the kind a normalizer fixes cheaply. Two of my own gold labels needed the same normalization treatment, and the 40-task pilot is the only reason I caught them.
The extract() path (Pydantic schemas over free text) has the same conservative bias. I expected it to hallucinate values for fields with no evidence in the text, and it almost never did: 3.8% on absent fields, 0% once a single noisy case is excluded. The actual problem is under-extraction, since it returned nothing for 28.3% of fields that had clear textual evidence. See argument complexity, schema violations and structured extraction.
I ran 120 noisy variants of standard tasks (60 with character typos, 60 with STT-style corruption: lowercasing, filler words, homophones) and paired each against its clean parent. Tool selection moved a point or two in either direction. Exact-match argument accuracy dropped from 0.733 to 0.444, with nearly all of the damage in free-text fields:
| Field kind | Clean | Noisy | Delta |
|---|---|---|---|
| Free text | 0.974 | 0.553 | -42 pts |
| Constrained | 0.870 | 0.792 | -8 pts |
The model copies noise instead of normalizing it. "translate good you know, morning into spanish" got its argument filled as text="good you know, morning", filler included, and a typo'd "ogod morning" came through character for character, from the same model that converts "7:30 in the morning" into a canonical time without being asked. Constrained fields hold up because picking from a small set of legal values doesn't require reconstructing a garbled string, and tool selection is a choice from a fixed set too, which I'd guess is why it holds up as well. There was one exception: a task where both typos landed inside a quoted search term flipped the model from search_web to find_nearby, so noise that corrupts the intent-carrying words can still break selection. The noise transforms are synthetic rather than sampled from a real recognizer, so I'd trust the free-text/constrained split more than the exact magnitudes. Details in noisy input robustness.
AI-generated phrasing damages selection instead. I took 30 tasks the model handled correctly and rewrote each in six AI-typical styles, holding intent and gold arguments fixed. Four styles changed little. Markdown fragments, verb-less lines like **Set volume**: \35``, produced seven false abstentions out of 30, one at confidence 0.92. My best guess is that the model wants an imperative verb before it commits to a call, but with n=30 I don't treat that as settled, especially since an earlier, smaller run in this series pointed at em-dash phrasing as a problem and the 30-intent re-test showed no effect at all. Two styles helped: terse list formatting and typographic punctuation both improved argument accuracy over the clean baseline. If your queries come from an upstream LLM, strip the markdown and leave the punctuation alone. See AI phrasing artifacts.
Both perturbation classes above produced false abstentions at high confidence, which is what the last two experiments looked at directly: whether the model can signal its own failures.
Abstention itself works. An unmatched query returns an empty call with a reasoning string rather than a call to the nearest tool, and 67 of 80 no-tool queries abstained correctly. It's unreliable in both directions at similar rates though, about 16% false calls on no-tool queries against 10% false abstentions on plainly callable ones, and my hypothesis about the trigger was wrong: near-miss catalogs (a semantically close but wrong tool present) produced slightly fewer false calls than unrelated ones. The trigger that did show up was scope, where "turn on all the lights" against single-room tools produced the most confident false calls in the set.
The confidence float turned out to be unusable as a gate. On average it orders the outcomes correctly, good outcomes scoring higher than bad ones, but good outcomes span nearly the whole [0, 1] range and bad ones reach up to 0.80, so the first threshold that excludes every bad outcome also throws away 60% of the good ones. Formal calibration over 593 pooled records agrees: the expected calibration error (ECE), the average gap between the model's stated confidence and its actual accuracy, comes out at 0.178 at tool level, the reliability curve is non-monotonic, and no escalation threshold between 0.1 and 0.9 keeps on-device error under 5% while escalating less than half the traffic.
A fair objection is that pooling five experiments' task mixes isn't a realistic calibration test, since production traffic is one stationary mix. Two things in the data answer it: calibration never gets good on any single source either (ECE 0.16 at best), and the direction of the miscalibration flips across the range, understating accuracy in one band and overstating it in the next, which no threshold shift can correct. Sample size doesn't rescue it either, since individual wrong answers arrived at confidence 0.80 and 0.98, and a gate exists to catch exactly those.
The backup signals do no better. The engine's validation.ungrounded flag catches one in four real argument errors and is wrong 40% of the time it fires, and the extract() path returns no confidence at all, just the parsed object or Python None, with None coming back on 8% of calls, two of them on complete, unambiguous input. One pattern I want to test further: all four false abstentions justified themselves with "this tool can't do X" phrasing, while correct abstentions almost always used a generic "no tool for X". Four cases are too few to build on. Data in abstention and confidence calibration.
Since confidence can't gate calls, gate on schema validation instead: shape violations never occurred across 893 records, but checking argument names and enum membership against the catalog catches a real 8% of calls, which is more than any confidence threshold removes safely.
A few cheaper fixes follow from the same data. Normalize enum values instead of comparing them for equality, since the space-for-underscore substitution alone covers 4.48% of all records. Prefer enums and typed parameters over free text where the schema allows it, because constrained fields lost 8 points to noise where free-text fields lost 42. Clean STT input before the model and strip markdown after an upstream LLM; these are separate failures, and fixing one does nothing for the other.
On the runtime: complete() is stateful by default, so call reset() before every independent query, and use one engine process per worker rather than threads, since the engine holds process-global state and a 4-thread probe failed 15 calls out of 20. When you test, use paraphrase batches rather than repeat runs, because repeats of an identical query agree with themselves by construction.
All of this is one model version, one machine, and one hand-authored dataset written by the same person who drew the conclusions, and that dataset had two label bugs only the pilot run caught, so the residual label-error rate is probably not zero. Four of the eleven hypotheses came back rejected. What I'd carry to another model is the method more than the numbers: check for determinism first, since it changes what every experiment after it costs; track the composition of the errors along with the rate; and treat selection failures and argument failures as separate problems, since they respond to different fixes.
The scratchpads with full data: runtime feasibility, determinism, task dataset, catalog size, argument complexity, abstention, confidence calibration, noisy input, structured extraction, AI phrasing, schema violations.