What happens when an LLM never sees material beyond fifth grade?
Ranked #5 on Hacker News with 66 points and 46 comments.
The hosted 5B model, live in your browser. Open in a new tab ↗ if the chat doesn’t load below.
A controlled sandbox for studying how models acquire knowledge
Modern LMs are trained on everything at once, so it is hard to tell whether a new skill was learned or merely elicited . We constrain the training distribution itself: an 88B-token corpus filtered to the U.S. elementary-school curriculum, with models trained from scratch on it and matched unfiltered controls.
An 88B-token corpus distilled from FineWeb-Edu through a five-stage filtering pipeline aligned with Common Core standards (K–5). Concepts, facts, and vocabulary taught above Grade 5 are explicitly excluded.
Three scales (0.6B / 1.3B / 5B) trained from scratch on LittleCurriculum: chattable models with an interpretable knowledge boundary. Each ships with a matched Unfiltered control for clean comparison.
In our experiments, scaling, SFT+GRPO post-training, and in-context learning amplify what the curriculum taught, but none meaningfully improves out-of-scope performance, indicating that the pretraining filter sets the effective capability ceiling.