# Structured Extraction Field Accuracy and Absent-Field Hallucination (cactus-needle 2.0.5)

> Field-level accuracy of needle.extract() (Pydantic schemas) is high on clean text, but the model fills values for fields absent from the text instead of leaving them empty — hallucination at the field level, mirroring the call-level false-call behavior found in the no-tool experiment.

Date: 2026-08-17



## Test

Companion to the [no-tool abstention experiment](/scratchpads/needle-abstention-no-tool-queries), which measured call-level false-call rate. This test measures the same discipline at field level, using `needle.extract(text, PydanticModel, max_new_tokens=256)`, which exposes no sampling parameters (consistent with the determinism findings in [needle-determinism-and-paraphrase-sensitivity](/scratchpads/needle-determinism-and-paraphrase-sensitivity)).

75 single-call cases across 5 Pydantic schema families (ContactInfo, CalendarEvent, OrderReceipt, TravelBooking, MessageNote), each mixing required and Optional fields, one Literal enum field, and one date-string field.

60 base cases (12 per family): 4 complete (every field present), 4 absent-optional (1-3 Optional fields have no textual evidence, gold=None), 2 distractor (two plausible values for one field, gold is the contextually correct one), 2 normalization (date/time written non-canonically, gold normalized to ISO). Plus 15 noisy variants (typo/STT-style transforms, per [needle-noisy-input-robustness](/scratchpads/needle-noisy-input-robustness)) of 3 cases per family.

Each field scored into: `correct_value`, `wrong_value`, `hallucinated` (gold=None, model filled), `missed` (gold has a value, model returned None), `correct_absent` (gold=None, model returned None). `agent.reset()` before every case; value matching case-insensitive with substring fallback.


## Result

**REJECTED**

The predicted failure mode (hallucination on absent fields) did not dominate. Instead the model under-fills fields that have clear textual evidence.

| category | field events | correct | wrong | missed | hallucinated | correct absent | acc on populated |
|---|---|---|---|---|---|---|---|
| complete | 120 | 81 | 5 | 34 | 0 | 0 | 0.675 |
| absent_optional | 120 | 45 | 0 | 12 | 0 | 63 | 0.789 |
| distractor | 59 | 44 | 4 | 7 | 1 | 3 | 0.800 |
| normalization | 60 | 43 | 5 | 9 | 0 | 3 | 0.754 |
| complete_noisy | 30 | 17 | 2 | 11 | 0 | 0 | 0.567 |
| absent_optional_noisy | 30 | 9 | 4 | 2 | 3 | 12 | 0.600 |
| distractor_noisy | 29 | 16 | 8 | 3 | 0 | 2 | 0.593 |

Absent-optional hallucination rate: 3/78 gold-None fields (3.8%), against a call-level false-call rate of 16.25% (13/80) in the no-tool baseline — per-field abstention is roughly 4x more reliable than per-call abstention. All 3 hallucinations came from a single noisy case (`noisy_stt_11`); excluding it, the rate over the other 76 gold-None fields is 0.0%.

Clean-field accuracy on "complete" cases (every field has evidence) is only 0.675, driven by missed fields (34/120, 28.3%), not wrong values (5/120, 4.2%) or hallucination (0% by category definition). Required fields outperform Optional fields on populated-gold events (0.833 vs 0.616). Field-type accuracy ranks enum (0.732) ~ text (0.727) > date (0.600). Distractor win rate is 6/10 (0.60). Noisy input drops correct-value rate on scoreable fields from 0.708 (clean) to 0.583 (noisy), a comparable degradation to the argument-filling results in [needle-noisy-input-robustness](/scratchpads/needle-noisy-input-robustness).

A third failure mode surfaced outside the per-field scoring: `needle.extract()` returned Python `None` for the whole call (full extraction abstention, zero function calls emitted) on 6/75 calls (8.0%), including two "complete" cases with unambiguous, fully-present data. `needle.extract()` also discards the response envelope — it returns only the parsed Pydantic instance or `None`, never `confidence`, `reasoning`, or `validation.ungrounded`. Extraction mode has no confidence or grounding signal available through the public API, so the confidence-gating approach explored in [needle-confidence-calibration](/scratchpads/needle-confidence-calibration) does not carry over to this code path.

The model's conservative bias shows up again in how it handles conflict: on distractor cases it either picks the correct value, picks the wrong-but-plausible decoy, or misses the field outright — it never invents a third, unrelated value. On non-canonical dates it more often copies the raw phrasing verbatim than converts it, another form of declining to commit rather than fabricating.


## Next

1. Investigate the 28.3% missed-field rate on complete cases directly — test whether prompting for explicit per-field confirmation, or splitting extraction into smaller schemas, recovers the fields the model currently drops despite clear textual evidence.
2. Run a larger noisy-vs-clean sample (the current 15-pair comparison is underpowered relative to the 120-pair sample in the tool-argument noise experiment) to confirm the -12.5pt degradation and characterize the field-slot-confusion pattern seen in `noisy_stt_11` and `travel_c1`.
3. Characterize the 8.0% full-extraction-abstention rate: does it correlate with schema family, field count, or text length, and can it be reduced without inflating hallucination.
4. Test date-field extraction in isolation across more phrasing variants, since dates are the weakest field type (0.600) both inside and outside the dedicated normalization category.