LLMs do not reliably produce text matching a requested character count, with deviation patterns similar to word count compliance — high accuracy with 'exactly' phrasing, poor accuracy with 'approximately' phrasing.
Follow-up to the word count compliance scratchpad. Character count is arguably harder — characters are harder to count than words, but real-world limits (tweets at 280, SMS at 160) make it practically relevant.
150 API calls to Cerebras (gpt-oss-120b), varying:
Character targets: 160 (SMS), 280 (tweet), 500 (short), 1500 (medium), 5000 (long)
Topics: factual, creative, argumentative
Phrasing: "Write exactly X characters" vs "Write approximately X characters"
5 runs per combination (5 × 3 × 2 × 5 = 150 total).
- Temperature: 1.0, top_p: 0.95, max_completion_tokens: 65000
- Character count: len(text) on raw completion
PARTIALLY CONFIRMED
The model is surprisingly accurate at small targets but completely breaks down at 5000 characters.
By phrasing:
| Phrasing | Mean deviation | Exact matches |
|---|---|---|
| "Exactly" | 125.18% | 39/75 (52.0%) |
| "Approximately" | 18.84% | 4/75 (5.3%) |
The "exactly" mean is inflated by catastrophic failures at 5000 chars. For targets 160-500, "exactly" phrasing yields 0.04-0.07% mean deviation with 36/45 exact matches.
By target × phrasing:
| Target | "Exactly" mean dev | "Approx" mean dev |
|---|---|---|
| 160 | 0.04% | 3.08% |
| 280 | 0.07% | 2.64% |
| 500 | 0.07% | 4.28% |
| 1500 | 0.44% | 24.30% |
| 5000 | 625.26% | 59.91% |
At 5000 chars with "exactly" phrasing, the model either produces catastrophically long outputs (211K, 137K, 88K characters hitting the token limit) or near-empty ones (0 and 86 characters). "Approximately" at 5000 is actually more predictable — consistent ~60% overshoot with no extreme outliers.
The pattern from the word count experiment ("exactly" always outperforms "approximately") holds up to 1500 characters but inverts at 5000, where "exactly" triggers a catastrophic failure mode that "approximately" avoids.
- Test range-based character instructions ("between X and Y characters") to see if explicit slack prevents the 5000-char catastrophe
- Test other models to determine if the 5000-char breakdown is specific to gpt-oss-120b
- Investigate why SMS (160) and tweet (280) targets show near-identical compliance profiles — possible training data calibration