Field-level accuracy of needle.extract() (Pydantic schemas) is high on clean text, but the model fills values for fields absent from the text instead of leaving them empty — hallucination at the field level, mirroring the call-level false-call behavior found in the no-tool experiment.
Companion to the no-tool abstention experiment, which measured call-level false-call rate. This test measures the same discipline at field level, using needle.extract(text, PydanticModel, max_new_tokens=256), which exposes no sampling parameters (consistent with the determinism findings in needle-determinism-and-paraphrase-sensitivity).
75 single-call cases across 5 Pydantic schema families (ContactInfo, CalendarEvent, OrderReceipt, TravelBooking, MessageNote), each mixing required and Optional fields, one Literal enum field, and one date-string field.
60 base cases (12 per family): 4 complete (every field present), 4 absent-optional (1-3 Optional fields have no textual evidence, gold=None), 2 distractor (two plausible values for one field, gold is the contextually correct one), 2 normalization (date/time written non-canonically, gold normalized to ISO). Plus 15 noisy variants (typo/STT-style transforms, per needle-noisy-input-robustness) of 3 cases per family.
Each field scored into: correct_value, wrong_value, hallucinated (gold=None, model filled), missed (gold has a value, model returned None), correct_absent (gold=None, model returned None). agent.reset() before every case; value matching case-insensitive with substring fallback.
REJECTED
The predicted failure mode (hallucination on absent fields) did not dominate. Instead the model under-fills fields that have clear textual evidence.
| category | field events | correct | wrong | missed | hallucinated | correct absent | acc on populated |
|---|---|---|---|---|---|---|---|
| complete | 120 | 81 | 5 | 34 | 0 | 0 | 0.675 |
| absent_optional | 120 | 45 | 0 | 12 | 0 | 63 | 0.789 |
| distractor | 59 | 44 | 4 | 7 | 1 | 3 | 0.800 |
| normalization | 60 | 43 | 5 | 9 | 0 | 3 | 0.754 |
| complete_noisy | 30 | 17 | 2 | 11 | 0 | 0 | 0.567 |
| absent_optional_noisy | 30 | 9 | 4 | 2 | 3 | 12 | 0.600 |
| distractor_noisy | 29 | 16 | 8 | 3 | 0 | 2 | 0.593 |
Absent-optional hallucination rate: 3/78 gold-None fields (3.8%), against a call-level false-call rate of 16.25% (13/80) in the no-tool baseline — per-field abstention is roughly 4x more reliable than per-call abstention. All 3 hallucinations came from a single noisy case (noisy_stt_11); excluding it, the rate over the other 76 gold-None fields is 0.0%.
Clean-field accuracy on "complete" cases (every field has evidence) is only 0.675, driven by missed fields (34/120, 28.3%), not wrong values (5/120, 4.2%) or hallucination (0% by category definition). Required fields outperform Optional fields on populated-gold events (0.833 vs 0.616). Field-type accuracy ranks enum (0.732) ~ text (0.727) > date (0.600). Distractor win rate is 6/10 (0.60). Noisy input drops correct-value rate on scoreable fields from 0.708 (clean) to 0.583 (noisy), a comparable degradation to the argument-filling results in needle-noisy-input-robustness.
A third failure mode surfaced outside the per-field scoring: needle.extract() returned Python None for the whole call (full extraction abstention, zero function calls emitted) on 6/75 calls (8.0%), including two "complete" cases with unambiguous, fully-present data. needle.extract() also discards the response envelope — it returns only the parsed Pydantic instance or None, never confidence, reasoning, or validation.ungrounded. Extraction mode has no confidence or grounding signal available through the public API, so the confidence-gating approach explored in needle-confidence-calibration does not carry over to this code path.
The model's conservative bias shows up again in how it handles conflict: on distractor cases it either picks the correct value, picks the wrong-but-plausible decoy, or misses the field outright — it never invents a third, unrelated value. On non-canonical dates it more often copies the raw phrasing verbatim than converts it, another form of declining to commit rather than fabricating.
- Investigate the 28.3% missed-field rate on complete cases directly — test whether prompting for explicit per-field confirmation, or splitting extraction into smaller schemas, recovers the fields the model currently drops despite clear textual evidence.
- Run a larger noisy-vs-clean sample (the current 15-pair comparison is underpowered relative to the 120-pair sample in the tool-argument noise experiment) to confirm the -12.5pt degradation and characterize the field-slot-confusion pattern seen in
noisy_stt_11andtravel_c1. - Characterize the 8.0% full-extraction-abstention rate: does it correlate with schema family, field count, or text length, and can it be reduced without inflating hallucination.
- Test date-field extraction in isolation across more phrasing variants, since dates are the weakest field type (0.600) both inside and outside the dedicated normalization category.