https://www.glitchytales.com AI research, odd experiments, and the stories that come out of them. https://www.glitchytales.com/articles/testing-needle-a-local-tool-calling-model https://www.glitchytales.com/articles/testing-needle-a-local-tool-calling-model Wed, 26 Aug 2026 00:00:00 GMT Eleven experiments and roughly 1,800 calls against cactus-needle 2.0.5. The accuracy number turned out to matter much less than which errors it makes, and where. https://www.glitchytales.com/scratchpads/needle-abstention-no-tool-queries https://www.glitchytales.com/scratchpads/needle-abstention-no-tool-queries Mon, 17 Aug 2026 00:00:00 GMT When no relevant tool exists, the model abstains rather than forcing a false-positive call, and near-miss catalogs (a semantically close but wrong tool present) produce more false calls than unrelated catalogs. Confidence does not reliably separate correct abstentions from false ones. https://www.glitchytales.com/scratchpads/needle-ai-phrasing-artifacts https://www.glitchytales.com/scratchpads/needle-ai-phrasing-artifacts Mon, 17 Aug 2026 00:00:00 GMT AI-writing artifacts in queries — em-dash constructions, hedging preambles, unicode punctuation, markdown fragments, list-style phrasing, over-polite boilerplate — degrade tool calling even when the intent is perfectly clear. https://www.glitchytales.com/scratchpads/needle-argument-complexity https://www.glitchytales.com/scratchpads/needle-argument-complexity Mon, 17 Aug 2026 00:00:00 GMT Tool selection stays accurate as argument complexity rises, but argument-filling accuracy falls with argument count and type complexity (enums, dates, optionals). Secondary: the validation.ungrounded field predicts which arguments are wrong. https://www.glitchytales.com/scratchpads/needle-catalog-size-scaling https://www.glitchytales.com/scratchpads/needle-catalog-size-scaling Mon, 17 Aug 2026 00:00:00 GMT Tool-selection accuracy degrades as the tool catalog grows from 5 to 50 tools, and per-call latency grows with catalog size due to prefill cost. https://www.glitchytales.com/scratchpads/needle-confidence-calibration https://www.glitchytales.com/scratchpads/needle-confidence-calibration Mon, 17 Aug 2026 00:00:00 GMT The confidence head is calibrated — accuracy rises monotonically with confidence threshold, and expected calibration error (ECE) is low. https://www.glitchytales.com/scratchpads/needle-determinism-and-paraphrase-sensitivity https://www.glitchytales.com/scratchpads/needle-determinism-and-paraphrase-sensitivity Mon, 17 Aug 2026 00:00:00 GMT Needle 2 inference is deterministic — repeated identical queries yield identical tool, arguments, and confidence — so prediction variance in future experiments should come only from input wording, making paraphrase spread the correct measure of reliability rather than repeat sampling. https://www.glitchytales.com/scratchpads/needle-noisy-input-robustness https://www.glitchytales.com/scratchpads/needle-noisy-input-robustness Mon, 17 Aug 2026 00:00:00 GMT Accuracy degrades under realistic input noise, and degrades faster for argument filling than for tool selection. STT-style noise (homophones, no punctuation, fillers, lowercasing) hurts more than character-level typos. https://www.glitchytales.com/scratchpads/needle-runtime-feasibility https://www.glitchytales.com/scratchpads/needle-runtime-feasibility Mon, 17 Aug 2026 00:00:00 GMT Needle 2 installs on Windows, completes a correct tool call, delivers usable throughput on CPU, and exposes the API features the planned Needle experiment series depends on. https://www.glitchytales.com/scratchpads/needle-schema-violation-census https://www.glitchytales.com/scratchpads/needle-schema-violation-census Mon, 17 Aug 2026 00:00:00 GMT Constrained decoding guarantees well-formed tool calls: across the full Needle record pool, the rate of schema violations (nonexistent tools, missing required arguments, unknown arguments, wrong argument types, out-of-enum values) is 0%, and engine errors are rare and non-structural. https://www.glitchytales.com/scratchpads/needle-structured-extraction https://www.glitchytales.com/scratchpads/needle-structured-extraction Mon, 17 Aug 2026 00:00:00 GMT Field-level accuracy of needle.extract() (Pydantic schemas) is high on clean text, but the model fills values for fields absent from the text instead of leaving them empty — hallucination at the field level, mirroring the call-level false-call behavior found in the no-tool experiment. https://www.glitchytales.com/scratchpads/needle-task-dataset-construction https://www.glitchytales.com/scratchpads/needle-task-dataset-construction Mon, 17 Aug 2026 00:00:00 GMT A hand-authored 470-task dataset, cross-validated for internal consistency and spot-checked against the real engine on a stratified pilot, will contain no systematic label errors after fixing whatever the pilot surfaces. https://www.glitchytales.com/articles/a-field-guide-to-ai-benchmarks https://www.glitchytales.com/articles/a-field-guide-to-ai-benchmarks Fri, 07 Aug 2026 00:00:00 GMT A reference catalog of the benchmarks used to evaluate LLMs: who made each one, what it measures, and which numbers to look at when you want a model that's good at a specific thing. https://www.glitchytales.com/articles/gepa-prompt-optimization-tiny-local-models https://www.glitchytales.com/articles/gepa-prompt-optimization-tiny-local-models Sun, 10 May 2026 00:00:00 GMT Running DSPy's GEPA optimizer on SmolLM2 135M and 1.7B for classification tasks improves performance compared to basic prompts, but optimization process and model capabilities hit a wall fast. https://www.glitchytales.com/articles/clir-a-fixed-vocabulary-ir-for-llm-code-generation https://www.glitchytales.com/articles/clir-a-fixed-vocabulary-ir-for-llm-code-generation Thu, 07 May 2026 00:00:00 GMT Instead of a smaller generalist, a 103-token fixed vocabulary that eliminates naming entirely. The model only handles computation structure while a co-designed harness owns everything else: workspace assembly, validation, expansion to target code. https://www.glitchytales.com/articles/replicant-giving-ai-agents-gpu-access https://www.glitchytales.com/articles/replicant-giving-ai-agents-gpu-access Wed, 08 Apr 2026 00:00:00 GMT An MCP server in Rust that lets Claude run GPU experiments on Colab autonomously. Sessions persist on Drive, code runs asynchronously, and the agent never needs to touch a browser. https://www.glitchytales.com/scratchpads/output-diversity-vs-generation-parameters https://www.glitchytales.com/scratchpads/output-diversity-vs-generation-parameters Thu, 05 Mar 2026 00:00:00 GMT Higher temperature and higher top_p values produce more diverse outputs when repeating the same creative prompt, as measured by vocabulary richness, n-gram overlap, semantic similarity, and token entropy. https://www.glitchytales.com/scratchpads/passphrase-seeding-random-noise https://www.glitchytales.com/scratchpads/passphrase-seeding-random-noise Thu, 05 Mar 2026 00:00:00 GMT Prepending random passphrases (random words, random numbers, random alphanumeric strings) before a creative prompt increases output diversity compared to baseline. https://www.glitchytales.com/scratchpads/passphrase-seeding-themed-words https://www.glitchytales.com/scratchpads/passphrase-seeding-themed-words Thu, 05 Mar 2026 00:00:00 GMT Thematic passphrases affect output diversity differently from random passphrases: themed seeds may bleed into story content, produce different entropy profiles, and steer outputs in measurably different directions. https://www.glitchytales.com/articles/how-well-do-llms-follow-length-instructions https://www.glitchytales.com/articles/how-well-do-llms-follow-length-instructions Fri, 20 Feb 2026 00:00:00 GMT Six experiments, 573 API calls, and three different ways to phrase a length constraint. The answer depends entirely on how you ask. https://www.glitchytales.com/scratchpads/character-range-compliance https://www.glitchytales.com/scratchpads/character-range-compliance Tue, 10 Feb 2026 00:00:00 GMT Character count ranges ('between X and Y characters') produce more reliable compliance than exact or approximate point targets, particularly at 5000 characters where exact targeting broke down catastrophically. https://www.glitchytales.com/scratchpads/compound-length-constraints https://www.glitchytales.com/scratchpads/compound-length-constraints Tue, 10 Feb 2026 00:00:00 GMT Providing both word count and character count constraints simultaneously degrades compliance on both metrics compared to single-constraint baselines, and incompatible pairs force the model to prioritize one constraint over the other. https://www.glitchytales.com/scratchpads/word-count-range-compliance https://www.glitchytales.com/scratchpads/word-count-range-compliance Tue, 10 Feb 2026 00:00:00 GMT When given a word count range ('between X and Y words'), LLMs produce output within the range more reliably than with exact or approximate point targets, and output clusters near the midpoint. https://www.glitchytales.com/scratchpads/character-count-compliance https://www.glitchytales.com/scratchpads/character-count-compliance Fri, 06 Feb 2026 00:00:00 GMT LLMs do not reliably produce text matching a requested character count, with deviation patterns similar to word count compliance — high accuracy with 'exactly' phrasing, poor accuracy with 'approximately' phrasing. https://www.glitchytales.com/scratchpads/default-generation-length https://www.glitchytales.com/scratchpads/default-generation-length Fri, 06 Feb 2026 00:00:00 GMT Without a length constraint, LLM outputs cluster around a 'natural' default length that varies by topic type and prompt framing. https://www.glitchytales.com/scratchpads/llm-length-compliance https://www.glitchytales.com/scratchpads/llm-length-compliance Tue, 03 Feb 2026 00:00:00 GMT LLMs don't reliably produce text matching a requested word count, with deviation increasing for longer targets and varying by phrasing. https://www.glitchytales.com/scratchpads/division-accuracy-integers https://www.glitchytales.com/scratchpads/division-accuracy-integers Thu, 22 Jan 2026 00:00:00 GMT LLMs will show lower accuracy on division compared to addition and multiplication, particularly for non-terminating decimals and division by zero. https://www.glitchytales.com/scratchpads/addition-accuracy-negative-floats https://www.glitchytales.com/scratchpads/addition-accuracy-negative-floats Sun, 18 Jan 2026 00:00:00 GMT LLMs will show lower accuracy on float addition involving negative numbers, combining decimal handling with sign handling challenges. https://www.glitchytales.com/scratchpads/addition-accuracy-positive-floats https://www.glitchytales.com/scratchpads/addition-accuracy-positive-floats Sun, 18 Jan 2026 00:00:00 GMT LLMs will show lower accuracy on float addition compared to integer addition, due to decimal place handling. https://www.glitchytales.com/scratchpads/multiplication-accuracy-negative-integers https://www.glitchytales.com/scratchpads/multiplication-accuracy-negative-integers Sun, 11 Jan 2026 00:00:00 GMT LLMs can reliably perform simple multiplication involving negative integers below 100 in absolute value. https://www.glitchytales.com/scratchpads/multiplication-accuracy-positive-integers https://www.glitchytales.com/scratchpads/multiplication-accuracy-positive-integers Sun, 11 Jan 2026 00:00:00 GMT LLMs will show lower accuracy on multiplication of positive integers below 100 compared to addition, especially for larger operands. https://www.glitchytales.com/scratchpads/simple-addition-negative-integers https://www.glitchytales.com/scratchpads/simple-addition-negative-integers Thu, 08 Jan 2026 00:00:00 GMT LLMs can reliably perform simple addition involving negative integers below 100 in absolute value. https://www.glitchytales.com/scratchpads/simple-addition-positive-integers https://www.glitchytales.com/scratchpads/simple-addition-positive-integers Thu, 08 Jan 2026 00:00:00 GMT LLMs can reliably perform simple addition of two positive integers below 100. https://www.glitchytales.com/articles/automating-ai-experiments-with-claude-code https://www.glitchytales.com/articles/automating-ai-experiments-with-claude-code Fri, 05 Dec 2025 00:00:00 GMT Automating LLM experiments with a Director/Contributor multi-agent setup in Claude Code, from hypothesis to raw data. https://www.glitchytales.com/articles/what-if-agent-memory-didnt-need-a-database https://www.glitchytales.com/articles/what-if-agent-memory-didnt-need-a-database Fri, 14 Nov 2025 00:00:00 GMT Testing whether a single fixed-size matrix can replace a vector database for agent memory, using superposition and SONAR embeddings. https://www.glitchytales.com/articles/fabula-building-a-bookkeeper-for-stories https://www.glitchytales.com/articles/fabula-building-a-bookkeeper-for-stories Sat, 18 Oct 2025 00:00:00 GMT Building a Prolog system that tracks narrative state so LLMs can write stories without losing track of their own plot.