SCRATCHPADS-Experiments
Short lab reports: each entry is one controlled experiment with a falsifiable hypothesis.
Abstention on No-Tool Queries (cactus-needle 2.0.5) Effect of AI-Style Phrasing Artifacts on Tool Calls (cactus-needle 2.0.5) Argument Accuracy vs Argument Complexity (cactus-needle 2.0.5) Tool-Selection Accuracy and Latency vs Catalog Size (cactus-needle 2.0.5) Confidence Calibration for Edge-Cloud Escalation (cactus-needle 2.0.5) Determinism and Paraphrase Sensitivity (cactus-needle 2.0.5) Robustness to Typos and STT-Style Noise (cactus-needle 2.0.5) Runtime Feasibility and API Surface (cactus-needle 2.0.5) Schema Violation Census Across All Series Records (cactus-needle 2.0.5) Structured Extraction Field Accuracy and Absent-Field Hallucination (cactus-needle 2.0.5) Task Dataset Construction and Pilot Validation (cactus-needle 2.0.5) Output Diversity vs Generation Parameters (gpt-oss-120b) Passphrase Seeding — Random Noise (gpt-oss-120b) Passphrase Seeding — Themed Words (gpt-oss-120b) Character Count Range Compliance (gpt-oss-120b) Compound Length Constraints — Words + Characters (gpt-oss-120b) Word Count Range Compliance (gpt-oss-120b) Character Count Compliance (gpt-oss-120b) Default Generation Length Without Constraints (gpt-oss-120b) Do LLMs Actually Hit Requested Word Counts? Division Accuracy on Integers (gpt-oss-120b) Addition Accuracy on Negative Floats (gpt-oss-120b) Addition Accuracy on Positive Floats (gpt-oss-120b) Multiplication Accuracy on Negative Integers (gpt-oss-120b) Multiplication Accuracy on Positive Integers (gpt-oss-120b) Addition Accuracy on Negative Integers (gpt-oss-120b) Addition Accuracy on Positive Integers (gpt-oss-120b)
Abstention on No-Tool Queries (cactus-needle 2.0.5)
Hypothesis
When no relevant tool exists, the model abstains rather than forcing a false-positive call, and near-miss catalogs (a semantically close but wrong tool present) produce more false calls than unrelated catalogs. Confidence does not reliably separate correct abstentions from false ones.
needletool-callingabstentionfalse-positiveconfidence-calibrationlocal-model