# Determinism and Paraphrase Sensitivity (cactus-needle 2.0.5)

> Needle 2 inference is deterministic — repeated identical queries yield identical tool, arguments, and confidence — so prediction variance in future experiments should come only from input wording, making paraphrase spread the correct measure of reliability rather than repeat sampling.

Date: 2026-08-17



## Test

Follow-up to [needle-runtime-feasibility](/scratchpads/needle-runtime-feasibility), which flagged that confidence can floor at 0.0 without gating whether a call happens. This experiment checks whether Needle 2 output is deterministic at all, and if so, whether paraphrasing (not resampling) is the right lever for reliability testing.

Model: cactus-needle 2.0.5, local CPU inference, no API keys. Shared 8-tool catalog (`set_alarm`, `get_weather`, `send_message`, `create_calendar_event`, `play_music`, `translate_text`, `convert_currency`, `book_flight`).

1. **Sampling-control probe:** inspected `Needle.__init__/.complete/.reset/.run/.extract` signatures and grepped the installed package for temperature/seed/top_k/top_p/sampling/greedy/do_sample.
2. **Determinism at scale:** 10 distinct queries (7 standard/easy, 1 standard/ambiguous, 1 no_tool_unrelated, 1 no_tool_near_miss) × 20 identical repeats = 200 tasks, run three times (`serial_a`, `serial_b`, and a `run_batch(n_workers=4)` parallel arm) — 600 calls total. Compared `(tool, arguments, confidence, reasoning)` signatures within and across runs.
3. **Paraphrase consistency:** 15 intents × 8 hand-written paraphrases each (terse command, polite request, rambling speech, shorthand/em-dash, direct question, imperative) = 120 tasks, one serial run. Argument values kept canonical across paraphrases of the same intent. Agreement scored as (paraphrases matching the modal tool/args) / 8 per intent.


## Result

**CONFIRMED**

**Sampling controls:** none of the five public methods (`complete`, `run`, `reset`, `extract`, `__init__`) expose temperature, seed, top_k, or top_p. Those keywords only appear in an unrelated JAX training/finetune code path never invoked by inference. There is no knob to hold at zero because there is no knob.

**Determinism at scale (600 calls, 200 tasks × 3 run arms):**

| run | records written | queries fully deterministic (20/20 repeats identical) |
|---|---|---|
| serial_a | 200/200 | 10/10 |
| serial_b | 200/200 | 10/10 |
| parallel | 200/200 | 10/10 |

Cross-run match (serial_a vs serial_b vs parallel, same signature): 10/10 queries. All 600 calls collapsed to exactly 10 unique signatures, confidence floats matching to the 4th decimal in every case (e.g. `q01_alarm`: confidence=0.7567 on all 60 calls).

One result stands out: `q03_message` ("Send a message to Alice saying the meeting is at 3pm.") abstained on all 60 calls (confidence=0.9779, reasoning "No messaging or communication tool is available") despite `send_message` being present in the catalog. This is not noise — it reproduced identically across both serial runs and the parallel arm. The paraphrase run later showed it is specific to that exact phrasing: the same wording reused as paraphrase index 0 for both `message_alice` and `message_bob` abstained identically, while 5-6 of the other 7 paraphrases of the same intent correctly called `send_message`. The cause is not observable from the outside — no debug field exposes retrieval internals, consistent with the opacity noted in [needle-runtime-feasibility](/scratchpads/needle-runtime-feasibility).

**Paraphrase agreement (120 calls, 15 intents × 8 paraphrases):**

| metric | mean | stdev | min | max |
|---|---:|---:|---:|---:|
| tool agreement | 0.942 | 0.133 | 0.625 | 1.000 |
| args agreement | 0.892 | 0.176 | 0.500 | 1.000 |

11 of 15 intents scored perfect (1.000) tool agreement, and the modal tool matched the intended gold tool in all 15 — paraphrasing never flipped the majority answer to the wrong tool family. Argument-string agreement was looser: `alarm_0730`'s 8 paraphrases split into `{"07:30": 1, "7:30 in the morning": 2, "7:30 am": 5}`, all correct in substance but scored as 0.625 modal agreement because the comparison is textual. The three lowest-scoring intents (`message_alice`, `message_bob`, `music_jazz`, tool agreement 0.625-0.875) shared a common trigger: the one shorthand/em-dash paraphrase in each group ("Alice, meeting at 3pm -- send that.") drove confidence toward 0 and, in two of three cases, produced a wrong or truncated call.

With no sampling parameters exposed and confirmed bit-identical output across independent processes, prediction variance in Needle has exactly one source: how the query is worded. Repeat runs of an identical query are redundant for measuring reliability — they will always agree with themselves. Paraphrase spread is the only lever that surfaces real instability, and it did so twice here: once for shorthand/em-dash phrasing (a general pattern across three intents) and once for the surprising case where the plain, most literal phrasing was the outlier that failed.


## Next

1. Targeted follow-up on why `send_message` abstains specifically on the "Send a message to X saying Y" phrasing, isolating which token or structure triggers it.
2. Extend paraphrase testing to noisy/vague argument values (this experiment kept them canonical) to see whether disagreement compounds — see [needle-noisy-input-robustness](/scratchpads/needle-noisy-input-robustness).
3. Use the confirmed determinism result to drop repeat-run budgets from future Needle experiments and reallocate call budget to more paraphrases per intent, especially for free-text arguments like message bodies and event titles.
4. Cross-reference the abstention anomaly against [needle-abstention-no-tool-queries](/scratchpads/needle-abstention-no-tool-queries) to check whether other tool-present-but-abstains cases exist beyond `send_message`.