PUBLICATIONS-Artifacts
Datasets and other artifacts produced by the experiments on this blog, published for reuse under the license stated on each entry.
Synthetic Tool-Calling Eval
A synthetic benchmark for tool selection, argument filling, and abstention in language models.
What it is
A synthetic benchmark for tool-calling (function-calling) in language models. It tests three behaviors: picking the right tool from a catalog, filling the arguments correctly, and declining to call a tool when nothing in the catalog applies to the request.
Contents
126 tool-spec rows across 8 catalogs, covering 57 distinct tools. Catalog sizes are nested at 5, 10, 25, and 50 tools, plus two "near-miss" catalogs built from 9 confusable tool-name pairs and one unrelated catalog used for abstention testing. On top of these catalogs, the dataset has 770 tasks split across 6 subsets: standard, no-tool, noisy (typo and speech-to-text variants), paraphrase, AI-ism phrasing styles, and paraphrase-consistency. It also includes 75 structured-extraction cases across 5 schema families.
Provenance and license
The dataset is machine-generated by Anthropic Claude models during automated experiments. Gold labels were validated against Needle 2 (cactus-needle 2.0.5), a 45M-parameter local tool-calling model, across the blog's Needle experiment series. It is released under CC-BY-4.0. The Hugging Face copy is loadable via the datasets library; the GitHub copy holds the raw JSONL plus documentation.