SCRATCHPADS-Experiment

Gate check for a planned Needle scratchpad series (catalog size, argument complexity, abstention, confidence calibration, noisy input). Installed cactus-needle==2.0.5 via uv add in a fresh uv init project (Python 3.13.7, Windows, no build-system section), then ran it through six checks:

1. Install footprint and first-run engine download.
2. Single-call validation on a 3-tool catalog (set_alarm, get_weather, send_message).
3. Throughput: 20 serial single-call queries against the 3-tool catalog, agent.reset() before each call, wall-clock via time.perf_counter() plus engine-reported prefill_tps/decode_tps.
4. API-shape probes: abstention (2 queries), confidence (1 easy, 1 hard query), an 8-tool catalog (2 queries), Pydantic structured extraction (1 query), multi-turn behavior (3 calls), and a concurrency probe (4 threads x 5 queries, run in a subprocess to isolate any native-library crash).
5. No controls or baselines — this is a feasibility probe, not a benchmark.

Full per-call data in data/needle_feasibility/latency.jsonl and data/needle_feasibility/api_probes.jsonl.

CONFIRMED

Needle 2 installs cleanly on Windows, calls tools correctly on the first attempt, and runs fast enough on CPU for the planned series. Every open API question got a direct answer.

Check	Result
Install	uv add cactus-needle resolved 44 packages in ~15s; no build errors
Engine download	14 MB libneedle.dll, one-time, cached to ~/.cache/cactus-needle/2.0.2/
First call	correct tool, correct argument, init() 4.6s, complete() 0.16s
Throughput (n=20, 3-tool catalog)	mean 400.8 ms/call, stdev 151.2 ms, min 211.1 / max 677.8 ms
Engine-reported speed	prefill 225.8 tok/s, decode 95.4 tok/s (mean)
Plausible tool calls	18/20 (2 were abstentions, not errors)
8-tool catalog	2/2 correct (confidence 0.7727, 0.9947)
Concurrency (4 threads x 5 queries)	5/20 succeeded, 15/20 clean RuntimeError, no crash

The response envelope is {type, success, error, error_code, reason, function_calls, reasoning, confidence, prefill_tps, decode_tps, peak_ram_mb, validation}. Abstention is explicit rather than forced: a query with no matching tool returns type='call' with an empty function_calls list and a reasoning string, not a call to the nearest tool. Confidence is a single float in the 0.0–0.9947 range observed here; on a deliberately rambling query the model still emitted a call but confidence floored at exactly 0.0, and a separate validation.ungrounded field flags which arguments were inferred rather than copied from the text — so confidence and argument grounding are two distinct signals, worth checking further before the calibration experiment leans on confidence as continuous. Structured extraction via needle.extract(text, PydanticModel) returns a populated Pydantic instance on one shot, internally converting the model's JSON schema into a single-tool catalog.

Two findings constrain every downstream experiment in the series. First, complete() is stateful by default: a second call on the same Needle instance inherits context from the first even with no explicit history, so any single-shot design must call agent.reset() before each independent query or Needle.run() for genuine multi-turn. Second, the engine holds process-global state behind ctypes, so concurrent Needle instances across threads in one process are unsafe — the 4-thread probe saw a 75% failure rate (clean RuntimeErrors, not crashes). Parallelism, if needed later, has to go through separate processes rather than threads.

1. Check confidence's floor-at-0.0 behavior against a larger sample before the confidence calibration experiment treats it as continuous.
2. Run the planned catalog size scaling sweep now that tool-selection stays correct through 8 tools, to find where it degrades.
3. Build the task dataset with reset() per query baked into the harness, per the statefulness finding here.
4. Verify whether Needle(weights=...) fine-tuned mode (confidence reported as None) matters for any experiment in the series before it starts.