Post

A GPT-2-class language model built entirely from scratch in C/CUDA — no PyTorch, no autograd, no ML libraries. The forward and backward passes are written and verified by hand, and the whole training pipeline lives in this repo: a hand-written byte-level BPE tokenizer , pretraining on a books + web corpus, and supervised fine-tuning into a chat model (RLHF/DPO planned). It runs on CPU ( libm + OpenMP) for a small showcase model, and a full from-scratch CUDA engine — cuBLAS matmuls, a hand-written FlashAttention , validated against a CPU reference by a full-model gradient check — trains a ~116M-parameter model on a single RTX 4070.

Status & honesty. This is a research/educational artifact, built in public. At ~116M parameters trained on a single consumer GPU, it is a text generator in the spirit of GPT-2-small : fluent-ish English, no real world knowledge . It is not a capable assistant — the chat model demonstrates that the pretrain→SFT pipeline works end to end, it is not a useful chatbot. The point of the project is the from-scratch engineering and the complete, understandable training pipeline.

Read it as a step of numerical integration. The forward-Euler method advances an ordinary differential equation dx/dt = f(x) by

With step size Δt = 1 this is exactly the residual update. So a deep residual network is a discretized ODE : depth is integration time , and each layer integrates the hidden state forward by one Euler step. This is the view behind work like Neural ODEs (a ResNet is the Euler discretization of a continuous flow). The project is named after Leonhard Euler , who gave us that integration method.

A sample from the ~116M model after a partial pretraining run on the books + web corpus (prompt Alessandro eat a ):

The content is not meaningful, but notice what it learned on its own: real grammar, long clauses, and an encyclopedic register picked up from the web data. This is the expected behaviour of a small model trained on a single GPU — fluent shape, shallow substance. More training and (far) more data improve fluency; world knowledge needs scale this project does not pretend to have.