LLMs will show lower accuracy on division compared to addition and multiplication, particularly for non-terminating decimals and division by zero.
Division introduces complexity the prior operations didn't have: remainders, repeating decimals, and the undefined case of division by zero. All prior experiments (addition and multiplication, integers and floats, positive and negative) showed 100% accuracy.
100 division problems with integer operands (1-99), stratified into 5 cases:
Clean (25 pairs): a % b == 0, integer result
Terminating (25 pairs): a/b is a terminating decimal
Repeating (25 pairs): a/b is a repeating decimal
Division by one (15 pairs): b = 1
Division by zero (10 pairs): b = 0
Model: gpt-oss-120b via Cerebras API (free tier)
Temperature: 0, top_p: 1, max_completion_tokens: 1024
Prompt (cases 1-4): "What is {a} / {b}? Reply with only the number."
Prompt (case 5): "What is {a} / {b}? Reply with only the number, or 'undefined' if the operation is not possible."
Repeating decimals checked by rounding both answers to 4 decimal places
Parser handles both decimal and fraction responses (e.g. "4/19")
REJECTED
100% accuracy across all 100 problems, including repeating decimals and division by zero.
| Case | Total | Correct | Accuracy |
|---|---|---|---|
| Clean (integer result) | 25 | 25 | 100.0% |
| Terminating decimal | 25 | 25 | 100.0% |
| Repeating decimal | 25 | 25 | 100.0% |
| Division by one | 15 | 15 | 100.0% |
| Division by zero | 10 | 10 | 100.0% |
For repeating decimals, the model sometimes returned simplified fractions (e.g. "4/19" for 8/38, "8/17" for 32/68) rather than decimal approximations — mathematically correct and arguably more precise. Division by zero was consistently identified as "undefined" with no hallucinated numeric values.
Combined with all prior experiments, gpt-oss-120b is now 700/700 across all four basic operations with operands below 100.
- Scale up to 3-digit and 4-digit operands to find where division accuracy breaks down
- Test division with float operands
- Run on a non-reasoning model — the chain-of-thought likely matters most for division, where the model visibly simplifies fractions before answering