rune-r1: build your own reasoning model

slimgpt answered “what does it take to train a gpt from scratch.” rune-r1 answers the next question: what does it take to make one reason — not just autocomplete fluently, but show its work and get rewarded for landing on the right answer. the whole pipeline runs end to end on a single gpu, for under $100.

three stages, each one continuing the previous stage’s weights: pretrain on raw text, sft on chain-of-thought math solutions, then grpo — reinforcement learning from verifiable rewards. each stage teaches something the previous one couldn’t.

each stage continues the previous stage's checkpoint — pretraining produces a fluent base model, sft teaches the response format, grpo teaches the model to actually be right.

the architecture: a specific, named recipe

the model itself is a 351m-parameter decoder-only transformer — 22 layers, 1024 embedding dimension, 16 query heads grouped into 4 kv heads, 1024 token context, gpt-2’s tiktoken vocabulary. every piece of the mechanism — tokens, embeddings, attention, feed-forward, the residual stream — works exactly like how llms actually work describes. what changes here is which specific variant of each piece got picked, and none of it was arbitrary — it’s the same combination qwen3 uses:

rope for position, rotating query/key vectors instead of adding a position embedding
rmsnorm with qk-norm — rmsnorm’s cheaper rescale-only normalization, with an extra norm applied to queries and keys specifically before the attention dot product, which keeps attention logits from growing unbounded as training progresses
swiglu feed-forward instead of plain relu/gelu
grouped-query attention (gqa) — 16 query heads sharing just 4 key/value heads, cutting kv-cache memory for a model this size

the specific recipe — gqa+rope+qk-norm attention, swiglu ffn, rmsnorm throughout — is the same combination qwen3 uses, just at a much smaller scale.

stage one: pretraining is still just next-token prediction

nothing about pretraining changes because a reasoning stage comes later. the model streams fineweb-edu and learns next-token prediction, same as any base llm. 9,624 steps over 5.05b tokens took loss from 10.88 down to 2.999. the output — rune-0.3b-base — is fluent but has no idea it’s supposed to follow instructions. ask it a question and it doesn’t answer it, it continues it, sometimes by looping the same fragment forever:

(0,3)$
Answer: (0,3)$
Answer: (0,3)$
Answer: (0,3)$
...

that’s the base model asked to convert a coordinate to polar form. it’s not wrong, exactly — it just doesn’t know “answer the question” is a thing it’s supposed to do.

stage two: sft teaches the shape of an answer, not correctness

sft fine-tunes the base checkpoint on distilled chain-of-thought math solutions — each one formatted as <think>...</think> followed by a boxed final answer. 5,650 steps, two epochs, validation loss from 2.55 down to 1.400. the result — rune-0.3b-sft — produces real, structured reasoning on the same coordinate problem:

<think>Okay, so I need to convert the rectangular coordinates (0,3) to polar
coordinates. Let me think about how to do this. The problem says that r is 0,
and the point (0,3) is in polar coordinates...

that’s a genuine format improvement — it thinks out loud, it uses the right tags. but sft only ever sees “correct” demonstrations during training; nothing in the loss function tells it whether its own generated answer is actually right. it learned to sound like it’s reasoning. whether the reasoning lands anywhere is a separate question, and on plenty of problems it just doesn’t converge — the model keeps talking without ever closing the <think> tag with a boxed answer.

stage three: grpo is where correctness enters the loss function

this is the part that’s actually new relative to a normal fine-tuning pipeline. grpo — group-relative policy optimization — is reinforcement learning from verifiable rewards (rlvr): for each math problem, sample several responses from the current policy, grade each one with a symbolic verifier (rune/verifier.py, math-style answer extraction) that checks whether the boxed final answer is actually correct, then update the policy toward the responses that got it right.

four sampled responses to one problem, most rewarded zero — only the correct one (shaded) pulls weight. the update is clipped and pulled back toward the frozen sft checkpoint by a kl penalty, so the policy can't drift arbitrarily far chasing reward.

the update itself is a ppo-style clipped objective, plus a kl penalty that anchors the policy back to the frozen sft checkpoint — without that anchor, a policy chasing reward on a narrow verifier can drift into degenerate text that happens to satisfy the verifier without actually reasoning.

the reward signal here is genuinely sparse: over 2,001 steps, mean reward moved from 0.010 to 0.023 comparing the first half of training to the second, and only 152 of those 2,001 steps produced any nonzero reward at all. this is the real, unglamorous texture of rlvr on a small model — most rollouts get nothing, and the whole training run is trying to nudge the policy using a tiny fraction of signal that does land.

what that sparse signal buys you shows up clearly on a problem sft alone couldn’t close out — “what is √(√(235423523))”:

GRPO: ...The square root of 1 is 1, so the answer should be 1.</think> \boxed{1}. Correct.

sft, on this same kind of problem, tends to keep reasoning without ever closing the tag. grpo commits to a boxed answer — because looping forever gets zero reward every single time, and committing (even imperfectly) is the only path to reward at all. that’s the clearest evidence that grpo added something sft structurally couldn’t: the loss function never saw “correct” until this stage.

what doesn’t change: inference cost

throughput is ~73-75 tokens/sec, kv-cached greedy decoding on one mi300x — essentially identical across base, sft, and grpo checkpoints. reasoning-tuning changes behavior, not the architecture, so it doesn’t change inference cost either. the same forward pass that used to loop on a fragment now runs the same speed while producing a boxed answer. weights shrink from 1.4gb (fp32, training) down to 351mb at int8 for deployment.

the honest cost

all three stages — pretrain, sft, grpo — ran on one rented mi300x (192gb vram) for well under $100 total. that number is the actual point: none of this pipeline requires a lab budget. a from-scratch base model, reasoning-tuned with real rlvr, on hardware anyone can rent by the hour.

full sample transcripts across all three checkpoints are in the repo’s evals/results.json, and the complete write-up, references, and code — including the qwen3 paper and the deepseekmath/deepseek-r1 lineage grpo is built on — are linked there.