slimgpt

most people’s first contact with gpt-2’s architecture is andrej karpathy’s nanoGPT — deliberately a single dense file, because that’s the point of the exercise. slimgpt is what happens when i took that same architecture and recipe and asked a different question: what does this look like if every concern gets its own file, with nothing hidden?

the split is boring on purpose: config.py for architecture hyperparameters, layers.py for layernorm/attention/mlp/block, model.py for the forward pass and generation, data.py for batching, lr.py for the cosine schedule, train_config.py for the training cli. same 12-layer, 12-head, 768-dim, 124m-parameter model as gpt-2 small — just reorganized so you can read one file at a time instead of scrolling through one that does everything.

the part i actually had to solve, not just reorganize, was data. openwebtext is ~54gb raw, and it didn’t fit on the 60gb disk i was training on. prepare.py streams the corpus and writes straight to tokenized .bin files with a capped token budget — the raw text is never stored on disk at all, just tokenized and discarded as it streams through. that one change is what made training on modest hardware possible.

training ran 5,000 iterations over ~5b tokens on a single nvidia l4 (23gb vram) — about 40k tokens/sec, ~13 seconds per iteration, ~18 hours total, with flash attention and torch.compile on and bf16 precision. val loss tracked train loss closely the whole way (generalization gap of +0.0145 at 5k iters — basically no overfitting), landing at a val perplexity of 27.3 against ~22.4 for a fully-trained gpt-2 small. that gap is exactly what you’d expect from 5k iterations instead of the hundreds of thousands a full run uses — the point was never to match the original, it was to watch every part of the process happen somewhere i could see it.

export.py converts the training checkpoint to huggingface format, a slim inference-only .pt, or a single safetensors file, so the result loads with plain transformers code, not just this repo’s own runtime.