slimgpt: what actually happens when you build your own llm

it’s easy to use an llm without ever seeing what’s actually inside one. tokens go in, logits come out, and the 12 transformer blocks in between stay a black box you trust because everyone else trusts it too. slimgpt was me refusing to leave it a black box — training gpt-2 small (124m parameters) from scratch, on real data, with every piece of the pipeline in its own readable file.

this isn’t a novel architecture. it’s the same 12-layer, 12-head, 768-dim model gpt-2 small always was, following andrej karpathy’s “let’s reproduce gpt-2” recipe closely. the value wasn’t in inventing something new — it was in building it myself, end to end, closely enough that nothing in the pipeline was still a black box by the end.

one file per concern, on purpose

nanoGPT — the reference implementation this is built on — is deliberately one dense file. that’s a feature for the audience it’s written for, not a flaw. but reading a dense file and building your own understanding of a training pipeline are different exercises, so i split it up:

slimgpt/
├── config.py          GPTConfig — architecture hyper-parameters
├── layers.py          LayerNorm, CausalSelfAttention, MLP, Block
├── model.py           GPT — forward, from_pretrained, generate, configure_optimizers
├── data.py            DataLoader — batches from tokenised .bin files
├── lr.py              cosine LR schedule with linear warmup
└── train_config.py    TrainConfig — training hyper-parameters + CLI

the data flow across those files is short enough to hold in your head: tokens → embed + positional encoding → 12× (attention → mlp → layernorm) → final layernorm → logits. once every step has its own file and its own name, you stop treating the model as one opaque thing and start being able to point at exactly which file is responsible for exactly which part of what comes out the other end.

the disk problem nobody mentions in the tutorials

openwebtext, uncompressed, is about 54gb of raw text. i was training on a machine with a 60gb disk. that’s not a “reduce your batch size” problem — that’s a “the corpus you want to train on literally does not fit” problem, and it’s the kind of constraint that never shows up until you actually try to run something yourself instead of reading about it.

the fix was to never let the raw text land on disk at all. prepare.py streams the corpus and tokenizes on the fly, writing straight into fixed-size .bin files up to a capped token budget:

# ~5B tokens, ~10 GB (default)
python data/openwebtext/prepare.py

# smaller, ~6 GB
python data/openwebtext/prepare.py --train_tokens 3e9

54gb of source text becomes ~10gb of tokenized .bin on disk, and the intermediate raw form never has to exist as a file at all. it’s a small change in the code and a real constraint solved — the kind of thing you only find by actually running out of disk mid-download.

training on one gpu, for real

no cluster, no multi-node orchestration — a single nvidia l4 (23gb vram), 4 vcpus, 15gb ram, debian 12. flash attention kicks in automatically on torch ≥2.0, torch.compile is on by default, and precision drops to bf16 since the l4 is ampere-class. effective batch size is batch_size × block_size × grad_accum × world_size, tuned to land around 0.5m tokens per iteration — 491,520 tokens/iter exactly, at roughly 40k tokens/sec.

5,000 iterations, ~13 seconds each, ~18 hours wall-clock, covering about 5 billion tokens of openwebtext.

what the loss curves actually said

iteration	train loss	val loss
1,000	4.1839	4.1632
2,000	3.6316	3.6346
3,000	3.4578	3.4535
4,000	3.3594	3.3680
5,000	3.2934	3.3079

val loss tracks train loss the whole way — the gap at 5k iterations is +0.0145, which is about as close to “no overfitting” as a real run gets. best val perplexity came out to 27.3, against roughly 22.4 for a fully-trained gpt-2 small. that gap is not a bug or a disappointing result — a full training run is hundreds of thousands of iterations, and this is 5,000. the numbers landing close, in the right direction, tracking cleanly between train and val, is exactly what “the pipeline works and the model is learning the right thing, just not finished” looks like.

exporting it so it’s actually usable

a training checkpoint carries optimizer state you don’t want at inference time. export.py converts it to a proper huggingface repo (model + tokenizer + model card), a slim inference-only .pt at roughly half the size, or a single safetensors file — so loading it back is just:

from transformers import AutoModelForCausalLM, AutoTokenizer

tok   = AutoTokenizer.from_pretrained("samueljayasingh/slimGPT")
model = AutoModelForCausalLM.from_pretrained("samueljayasingh/slimGPT")

no dependency on this repo’s own code to actually use the result — which was the point. the training pipeline is mine to read end to end; the output plays nicely with the same tools everyone already uses.

obvious credits to andrej karpathy’s “let’s reproduce gpt-2 (124m)” — this exists because that tutorial made the whole pipeline feel buildable instead of theoretical.