# Character Count Compliance (gpt-oss-120b)

> LLMs do not reliably produce text matching a requested character count, with deviation patterns similar to word count compliance — high accuracy with 'exactly' phrasing, poor accuracy with 'approximately' phrasing.

Date: 2026-02-06



## Test

Follow-up to the [word count compliance scratchpad](/scratchpads/llm-length-compliance). Character count is arguably harder — characters are harder to count than words, but real-world limits (tweets at 280, SMS at 160) make it practically relevant.

150 API calls to Cerebras (gpt-oss-120b), varying:

**Character targets:** 160 (SMS), 280 (tweet), 500 (short), 1500 (medium), 5000 (long)

**Topics:** factual, creative, argumentative

**Phrasing:** "Write exactly X characters" vs "Write approximately X characters"

5 runs per combination (5 × 3 × 2 × 5 = 150 total).

- Temperature: 1.0, top_p: 0.95, max_completion_tokens: 65000
- Character count: len(text) on raw completion


## Result

**PARTIALLY CONFIRMED**

The model is surprisingly accurate at small targets but completely breaks down at 5000 characters.

**By phrasing:**

| Phrasing | Mean deviation | Exact matches |
|----------|:-:|:-:|
| "Exactly" | 125.18% | 39/75 (52.0%) |
| "Approximately" | 18.84% | 4/75 (5.3%) |

The "exactly" mean is inflated by catastrophic failures at 5000 chars. For targets 160-500, "exactly" phrasing yields 0.04-0.07% mean deviation with 36/45 exact matches.

**By target × phrasing:**

| Target | "Exactly" mean dev | "Approx" mean dev |
|-------:|:-:|:-:|
| 160 | 0.04% | 3.08% |
| 280 | 0.07% | 2.64% |
| 500 | 0.07% | 4.28% |
| 1500 | 0.44% | 24.30% |
| 5000 | 625.26% | 59.91% |

At 5000 chars with "exactly" phrasing, the model either produces catastrophically long outputs (211K, 137K, 88K characters hitting the token limit) or near-empty ones (0 and 86 characters). "Approximately" at 5000 is actually more predictable — consistent ~60% overshoot with no extreme outliers.

The pattern from the word count experiment ("exactly" always outperforms "approximately") holds up to 1500 characters but **inverts** at 5000, where "exactly" triggers a catastrophic failure mode that "approximately" avoids.


## Next

1. Test range-based character instructions ("between X and Y characters") to see if explicit slack prevents the 5000-char catastrophe
2. Test other models to determine if the 5000-char breakdown is specific to gpt-oss-120b
3. Investigate why SMS (160) and tweet (280) targets show near-identical compliance profiles — possible training data calibration