Constrained decoding guarantees well-formed tool calls: across the full Needle record pool, the rate of schema violations (nonexistent tools, missing required arguments, unknown arguments, wrong argument types, out-of-enum values) is 0%, and engine errors are rare and non-structural.
Zero new engine calls. Pooled every records_*.jsonl file from six prior Needle experiments — catalog-size scaling, argument complexity, abstention on no-tool queries, noisy input robustness, AI phrasing artifacts, and the four arms of determinism and paraphrase sensitivity — for 1650 raw records.
Deduped globally by (query, catalog_id), keeping the first occurrence: 757 records dropped, all attributable to the consistency experiment's intentional query repetition and to legitimate cross-experiment reuse of identical query/catalog pairs. 893 records survive.
Each surviving record was scored against its source catalog for five violation classes: nonexistent tool name, missing required argument, unknown argument, wrong argument type, and out-of-enum value. A separate check covered the structured extraction path (75 records, Pydantic validation, no tool catalog), tallying populated error fields.
REJECTED
The 0%-violation clause fails. Across 893 deduped records, 72 (8.06%) carry a schema violation.
| Class | Count | Rate |
|---|---|---|
| nonexistent_tool | 0 | 0.00% |
| missing_required_argument | 0 | 0.00% |
| wrong_argument_type | 0 | 0.00% |
| unknown_argument | 10 | 1.12% |
| out_of_enum_value | 61 | 6.83% |
| engine_error | 1 | 0.11% |
Every violating record carries exactly one violation; none stacked. The three classes that check call shape — does the tool exist, are all required arguments present, do types match — hit 0/893 across the board. Constrained decoding holds perfectly there. The two classes that check argument content against the catalog's declared vocabulary account for all 72 violations.
The out-of-enum group splits roughly 2:1. The larger share, 40 of 61 (65.6%, 4.48% of the whole pool), is one systematic bug: the model fills an enum-typed slot with a space-separated phrase instead of the catalog's underscore token — door="front door" instead of "front_door", category="gas station" instead of "gas_station", room="living room" instead of "living_room". It reads like the model matching the query's surface wording rather than looking up the declared enum_values list, and it recurs identically across four separate source experiments wherever a clean door/room/category query is asked. The remaining 21 are genuinely out-of-domain or noise-corrupted values (room="backyard", room="all", typo-mangled fields) — heterogeneous, not one bug.
The unknown_argument class clusters just as tightly: 5 of 10 hits are the model inventing an argument (door or action) for open_garage_door, a tool with zero declared parameters by catalog definition. It recurs across the clean parent query, both noisy variants, and three of six AI-isms style variants of the same query — the failure tracks the tool's schema, not the phrasing or noise around it.
The single engine_error is a token-budget truncation on a noisy-input task, already reported in that experiment. The extraction path shows 0/75 validation failures, so the Pydantic guarantee holds cleanly and separately from the tool-calling path's enum and argument-naming gaps.
- Isolate enum-token formatting from other argument-filling error: test whether restating enum values in the tool description as natural-language phrases (or adding explicit space/underscore examples) closes the 4.48%-of-pool formatting gap.
- Audit
open_garage_doorandmute_devicespecifically — check whether their descriptions or naming invite argument invention, and whether other zero/low-argument tools in the catalogs show the same pattern. - Re-run the out-of-enum tally on a model from a different vendor to see whether the space-vs-underscore substitution is a cactus-needle-specific habit or general across constrained-decoding implementations.
- Add out-of-enum and unknown-argument checks as a standing gate in future Needle experiments rather than a one-off census, since both classes evade the "0% malformed calls" impression that anecdotal spot-checks gave across the series.