# Runtime Feasibility and API Surface (cactus-needle 2.0.5)

> Needle 2 installs on Windows, completes a correct tool call, delivers usable throughput on CPU, and exposes the API features the planned Needle experiment series depends on.

Date: 2026-08-17



## Test

Gate check for a planned Needle scratchpad series (catalog size, argument complexity, abstention, confidence calibration, noisy input). Installed `cactus-needle==2.0.5` via `uv add` in a fresh `uv init` project (Python 3.13.7, Windows, no build-system section), then ran it through six checks:

1. Install footprint and first-run engine download.
2. Single-call validation on a 3-tool catalog (`set_alarm`, `get_weather`, `send_message`).
3. Throughput: 20 serial single-call queries against the 3-tool catalog, `agent.reset()` before each call, wall-clock via `time.perf_counter()` plus engine-reported `prefill_tps`/`decode_tps`.
4. API-shape probes: abstention (2 queries), confidence (1 easy, 1 hard query), an 8-tool catalog (2 queries), Pydantic structured extraction (1 query), multi-turn behavior (3 calls), and a concurrency probe (4 threads x 5 queries, run in a subprocess to isolate any native-library crash).
5. No controls or baselines — this is a feasibility probe, not a benchmark.

Full per-call data in `data/needle_feasibility/latency.jsonl` and `data/needle_feasibility/api_probes.jsonl`.


## Result

**CONFIRMED**

Needle 2 installs cleanly on Windows, calls tools correctly on the first attempt, and runs fast enough on CPU for the planned series. Every open API question got a direct answer.

| Check | Result |
|---|---|
| Install | `uv add cactus-needle` resolved 44 packages in ~15s; no build errors |
| Engine download | 14 MB `libneedle.dll`, one-time, cached to `~/.cache/cactus-needle/2.0.2/` |
| First call | correct tool, correct argument, `init()` 4.6s, `complete()` 0.16s |
| Throughput (n=20, 3-tool catalog) | mean 400.8 ms/call, stdev 151.2 ms, min 211.1 / max 677.8 ms |
| Engine-reported speed | prefill 225.8 tok/s, decode 95.4 tok/s (mean) |
| Plausible tool calls | 18/20 (2 were abstentions, not errors) |
| 8-tool catalog | 2/2 correct (confidence 0.7727, 0.9947) |
| Concurrency (4 threads x 5 queries) | 5/20 succeeded, 15/20 clean `RuntimeError`, no crash |

The response envelope is `{type, success, error, error_code, reason, function_calls, reasoning, confidence, prefill_tps, decode_tps, peak_ram_mb, validation}`. Abstention is explicit rather than forced: a query with no matching tool returns `type='call'` with an empty `function_calls` list and a `reasoning` string, not a call to the nearest tool. Confidence is a single float in the 0.0–0.9947 range observed here; on a deliberately rambling query the model still emitted a call but confidence floored at exactly 0.0, and a separate `validation.ungrounded` field flags which arguments were inferred rather than copied from the text — so confidence and argument grounding are two distinct signals, worth checking further before the calibration experiment leans on confidence as continuous. Structured extraction via `needle.extract(text, PydanticModel)` returns a populated Pydantic instance on one shot, internally converting the model's JSON schema into a single-tool catalog.

Two findings constrain every downstream experiment in the series. First, `complete()` is stateful by default: a second call on the same `Needle` instance inherits context from the first even with no explicit history, so any single-shot design must call `agent.reset()` before each independent query or `Needle.run()` for genuine multi-turn. Second, the engine holds process-global state behind ctypes, so concurrent `Needle` instances across threads in one process are unsafe — the 4-thread probe saw a 75% failure rate (clean `RuntimeError`s, not crashes). Parallelism, if needed later, has to go through separate processes rather than threads.


## Next

1. Check confidence's floor-at-0.0 behavior against a larger sample before the [confidence calibration](/scratchpads/needle-confidence-calibration) experiment treats it as continuous.
2. Run the planned [catalog size scaling](/scratchpads/needle-catalog-size-scaling) sweep now that tool-selection stays correct through 8 tools, to find where it degrades.
3. Build the [task dataset](/scratchpads/needle-task-dataset-construction) with `reset()` per query baked into the harness, per the statefulness finding here.
4. Verify whether `Needle(weights=...)` fine-tuned mode (confidence reported as `None`) matters for any experiment in the series before it starts.