# Schema Violation Census Across All Series Records (cactus-needle 2.0.5)

> Constrained decoding guarantees well-formed tool calls: across the full Needle record pool, the rate of schema violations (nonexistent tools, missing required arguments, unknown arguments, wrong argument types, out-of-enum values) is 0%, and engine errors are rare and non-structural.

Date: 2026-08-17



## Test

Zero new engine calls. Pooled every records_*.jsonl file from six prior [Needle](/scratchpads/needle-runtime-feasibility) experiments — [catalog-size scaling](/scratchpads/needle-catalog-size-scaling), [argument complexity](/scratchpads/needle-argument-complexity), [abstention on no-tool queries](/scratchpads/needle-abstention-no-tool-queries), [noisy input robustness](/scratchpads/needle-noisy-input-robustness), [AI phrasing artifacts](/scratchpads/needle-ai-phrasing-artifacts), and the four arms of [determinism and paraphrase sensitivity](/scratchpads/needle-determinism-and-paraphrase-sensitivity) — for 1650 raw records.

Deduped globally by `(query, catalog_id)`, keeping the first occurrence: 757 records dropped, all attributable to the consistency experiment's intentional query repetition and to legitimate cross-experiment reuse of identical query/catalog pairs. 893 records survive.

Each surviving record was scored against its source catalog for five violation classes: nonexistent tool name, missing required argument, unknown argument, wrong argument type, and out-of-enum value. A separate check covered the [structured extraction](/scratchpads/needle-structured-extraction) path (75 records, Pydantic validation, no tool catalog), tallying populated `error` fields.


## Result

**REJECTED**

The 0%-violation clause fails. Across 893 deduped records, 72 (8.06%) carry a schema violation.

| Class | Count | Rate |
|---|---:|---:|
| nonexistent_tool | 0 | 0.00% |
| missing_required_argument | 0 | 0.00% |
| wrong_argument_type | 0 | 0.00% |
| unknown_argument | 10 | 1.12% |
| out_of_enum_value | 61 | 6.83% |
| engine_error | 1 | 0.11% |

Every violating record carries exactly one violation; none stacked. The three classes that check call *shape* — does the tool exist, are all required arguments present, do types match — hit 0/893 across the board. Constrained decoding holds perfectly there. The two classes that check argument *content* against the catalog's declared vocabulary account for all 72 violations.

The out-of-enum group splits roughly 2:1. The larger share, 40 of 61 (65.6%, 4.48% of the whole pool), is one systematic bug: the model fills an enum-typed slot with a space-separated phrase instead of the catalog's underscore token — `door="front door"` instead of `"front_door"`, `category="gas station"` instead of `"gas_station"`, `room="living room"` instead of `"living_room"`. It reads like the model matching the query's surface wording rather than looking up the declared `enum_values` list, and it recurs identically across four separate source experiments wherever a clean door/room/category query is asked. The remaining 21 are genuinely out-of-domain or noise-corrupted values (`room="backyard"`, `room="all"`, typo-mangled fields) — heterogeneous, not one bug.

The unknown_argument class clusters just as tightly: 5 of 10 hits are the model inventing an argument (`door` or `action`) for `open_garage_door`, a tool with zero declared parameters by catalog definition. It recurs across the clean parent query, both noisy variants, and three of six AI-isms style variants of the same query — the failure tracks the tool's schema, not the phrasing or noise around it.

The single engine_error is a token-budget truncation on a noisy-input task, already reported in that experiment. The extraction path shows 0/75 validation failures, so the Pydantic guarantee holds cleanly and separately from the tool-calling path's enum and argument-naming gaps.


## Next

1. Isolate enum-token formatting from other argument-filling error: test whether restating enum values in the tool description as natural-language phrases (or adding explicit space/underscore examples) closes the 4.48%-of-pool formatting gap.
2. Audit `open_garage_door` and `mute_device` specifically — check whether their descriptions or naming invite argument invention, and whether other zero/low-argument tools in the catalogs show the same pattern.
3. Re-run the out-of-enum tally on a model from a different vendor to see whether the space-vs-underscore substitution is a cactus-needle-specific habit or general across constrained-decoding implementations.
4. Add out-of-enum and unknown-argument checks as a standing gate in future Needle experiments rather than a one-off census, since both classes evade the "0% malformed calls" impression that anecdotal spot-checks gave across the series.