SCRATCHPADS-Experiment

Follow-up to needle-runtime-feasibility, which flagged that confidence can floor at 0.0 without gating whether a call happens. This experiment checks whether Needle 2 output is deterministic at all, and if so, whether paraphrasing (not resampling) is the right lever for reliability testing.

Model: cactus-needle 2.0.5, local CPU inference, no API keys. Shared 8-tool catalog (set_alarm, get_weather, send_message, create_calendar_event, play_music, translate_text, convert_currency, book_flight).

1. Sampling-control probe: inspected Needle.__init__/.complete/.reset/.run/.extract signatures and grepped the installed package for temperature/seed/top_k/top_p/sampling/greedy/do_sample.
2. Determinism at scale: 10 distinct queries (7 standard/easy, 1 standard/ambiguous, 1 no_tool_unrelated, 1 no_tool_near_miss) × 20 identical repeats = 200 tasks, run three times (serial_a, serial_b, and a run_batch(n_workers=4) parallel arm) — 600 calls total. Compared (tool, arguments, confidence, reasoning) signatures within and across runs.
3. Paraphrase consistency: 15 intents × 8 hand-written paraphrases each (terse command, polite request, rambling speech, shorthand/em-dash, direct question, imperative) = 120 tasks, one serial run. Argument values kept canonical across paraphrases of the same intent. Agreement scored as (paraphrases matching the modal tool/args) / 8 per intent.

CONFIRMED

Sampling controls: none of the five public methods (complete, run, reset, extract, __init__) expose temperature, seed, top_k, or top_p. Those keywords only appear in an unrelated JAX training/finetune code path never invoked by inference. There is no knob to hold at zero because there is no knob.

Determinism at scale (600 calls, 200 tasks × 3 run arms):

run	records written	queries fully deterministic (20/20 repeats identical)
serial_a	200/200	10/10
serial_b	200/200	10/10
parallel	200/200	10/10

Cross-run match (serial_a vs serial_b vs parallel, same signature): 10/10 queries. All 600 calls collapsed to exactly 10 unique signatures, confidence floats matching to the 4th decimal in every case (e.g. q01_alarm: confidence=0.7567 on all 60 calls).

One result stands out: q03_message ("Send a message to Alice saying the meeting is at 3pm.") abstained on all 60 calls (confidence=0.9779, reasoning "No messaging or communication tool is available") despite send_message being present in the catalog. This is not noise — it reproduced identically across both serial runs and the parallel arm. The paraphrase run later showed it is specific to that exact phrasing: the same wording reused as paraphrase index 0 for both message_alice and message_bob abstained identically, while 5-6 of the other 7 paraphrases of the same intent correctly called send_message. The cause is not observable from the outside — no debug field exposes retrieval internals, consistent with the opacity noted in needle-runtime-feasibility.

Paraphrase agreement (120 calls, 15 intents × 8 paraphrases):

metric	mean	stdev	min	max
tool agreement	0.942	0.133	0.625	1.000
args agreement	0.892	0.176	0.500	1.000

11 of 15 intents scored perfect (1.000) tool agreement, and the modal tool matched the intended gold tool in all 15 — paraphrasing never flipped the majority answer to the wrong tool family. Argument-string agreement was looser: alarm_0730's 8 paraphrases split into {"07:30": 1, "7:30 in the morning": 2, "7:30 am": 5}, all correct in substance but scored as 0.625 modal agreement because the comparison is textual. The three lowest-scoring intents (message_alice, message_bob, music_jazz, tool agreement 0.625-0.875) shared a common trigger: the one shorthand/em-dash paraphrase in each group ("Alice, meeting at 3pm -- send that.") drove confidence toward 0 and, in two of three cases, produced a wrong or truncated call.

With no sampling parameters exposed and confirmed bit-identical output across independent processes, prediction variance in Needle has exactly one source: how the query is worded. Repeat runs of an identical query are redundant for measuring reliability — they will always agree with themselves. Paraphrase spread is the only lever that surfaces real instability, and it did so twice here: once for shorthand/em-dash phrasing (a general pattern across three intents) and once for the surprising case where the plain, most literal phrasing was the outlier that failed.

1. Targeted follow-up on why send_message abstains specifically on the "Send a message to X saying Y" phrasing, isolating which token or structure triggers it.
2. Extend paraphrase testing to noisy/vague argument values (this experiment kept them canonical) to see whether disagreement compounds — see needle-noisy-input-robustness.
3. Use the confirmed determinism result to drop repeat-run budgets from future Needle experiments and reallocate call budget to more paraphrases per intent, especially for free-text arguments like message bodies and event titles.
4. Cross-reference the abstention anomaly against needle-abstention-no-tool-queries to check whether other tool-present-but-abstains cases exist beyond send_message.