how llms actually work

modern llms are mostly built by stacking transformer blocks over and over, so understanding the transformer machinery gets you most of the way there. this is a full walkthrough of how that machinery actually works — the core mechanisms inside modern transformer-based llms, without the sticky math. you should still learn the math eventually. this is the introduction that makes the math worth learning.

most modern llms share the same transformer-family skeleton. the differences come from what each one was trained on, the scale and configuration choices, and the post-training done on top. by the end, you should be able to read a modern llm paper or model card and know which piece of the architecture each section is talking about.

here’s the path: tokens → embeddings → positional encoding → attention → multi-head attention → the feed-forward network → the residual stream and layer normalization → next-token prediction → architecture vs trained weights.

tokenization

models don’t read text directly. they read integer ids. the step that converts your prompt into a sequence of those integers is called tokenization. a tokenizer takes a string and produces a sequence of integers, where each integer points to an entry in a fixed vocabulary. modern llm vocabularies usually contain tens of thousands to a few hundred thousand entries.

token id — the integer the model uses for one vocabulary entry. the model works with the number, not the written word itself.

tokens aren’t usually whole words. they’re usually subword pieces. “tokenization” might split into [“token”, “ization”]. “running” might split into [“run”, “ning”]. the reason is efficiency: whole-word vocabularies are too big and don’t generalize to new words, character-level vocabularies are too small and force the model to learn even the simplest patterns from scratch. subword tokenization sits in the middle — the most common pieces become single tokens, and rare or novel words get composed from smaller pieces.

vocabulary — the tokenizer’s fixed list of pieces. each piece has an id, and the model can only directly receive ids from that list.

the trade-off shows up in places people don’t expect. the classic example: ask an llm how many r’s are in “strawberry.” llms used to get it wrong. that’s not the model failing at counting — it’s the model not operating on letters directly, only on token ids that happen to spell out a word a human would split letter by letter.

different model families use different tokenizers — gpt models use byte pair encoding variants, sentencepiece is common in llama-style models. the choice matters for compute (fewer tokens means less work) and for things like multilingual coverage, but the basic shape is the same: text in, integers out.

now that the prompt is a sequence of integers, the next step is to give those integers meaning.

embeddings

a token id like 1024 is just a row index. it doesn’t mean anything by itself. the thing that gives it meaning is a giant table called the embedding matrix.

every model has one. it has one row per vocabulary entry, and each row is a long vector of numbers. the length of each row is the model’s hidden size — in many 7b-class models, that’s 4,096 numbers per token. larger models usually use wider vectors.

vector — a list of numbers. in a transformer, each token becomes a vector so the model can do math with it.

when the tokenizer hands the model an integer, the model looks up that row and uses the vector instead. that vector is the token’s embedding — the model’s representation of what that token “means,” learned during training.

embedding matrix — a lookup table. token id in, learned vector out.

the interesting property of these embeddings is that semantically similar tokens end up with similar vectors. the vector for “king” is close in space to the vector for “queen,” and “paris” is close to “france.” none of this is hard-coded — it emerges from training on enough text, because those positions are what let the model predict text well.

you can do arithmetic on embeddings and it sometimes works. the famous example: king − man + woman ≈ queen. the geometry of embedding space carries real semantic structure, even though nobody told the model to build it that way.

worth being clear on: at this stage every token has been replaced by its embedding, but the embedding alone says nothing about where the token sits in the sequence. the vector for “dog” is the same vector whether “dog” is the first word in your prompt or the fifth. that’s a problem, and it’s the gap positional encoding fills.

positional encoding

plain self-attention doesn’t have a built-in representation of word order. without some positional signal, it has no direct way to know “dog” came before “bites” instead of after it. word order changes meaning, so the model needs a way to inject the position of each token into the math.

positional encoding — how the model gets order information. it tells the model where each token sits in the sequence.

the original transformer paper (vaswani et al., 2017) did this by giving each position its own pattern of numbers and adding it directly to each token’s embedding before any other processing. position 1 had one pattern, position 5 a different one, position 100 another — generated from sine and cosine waves at different frequencies. now the embedding for “dog” at position 1 was different from “dog” at position 5, purely because of the position pattern added to it.

that worked, and sinusoidal encodings were chosen partly because they can extrapolate beyond the exact sequence lengths seen during training. but additive position schemes had two problems that mattered more as models scaled up. first, the embedding had to carry both meaning and position in the same set of numbers — there’s only so much you can pack in. second, learned absolute position embeddings in particular don’t generalize cleanly: if you trained on prompts up to 2,048 tokens, the model never saw position 5,000 during training, and that embedding was never learned properly.

modern models mostly use a different scheme called rotary position embeddings (rope), introduced by su et al. in 2021 and now used in llama, mistral, gemma, qwen, and most other open-weight families. the intuition: instead of adding position info to each token’s vector, rope rotates the query and key vectors by an angle that depends on the token’s position. a token at position 1 gets a small turn, a token at position 100 gets a bigger turn. when two tokens are later compared during attention, what matters is the difference between their rotations — which encodes how far apart they are.

rope — rotary position embeddings. instead of adding a position vector, it rotates query and key vectors so relative distance shows up during attention.

the practical advantages are real: rope encodes relative position naturally (closer to what attention actually wants), it generalizes better to longer contexts, and it doesn’t add new parameters to the model.

even with good positional encoding, modern llms have a documented “lost in the middle” problem (liu et al., 2023) — they use information at the start and end of long prompts more reliably than information buried in the middle. that’s why prompt engineering tips like “put important context first” or “repeat key info at the end” actually help. the model isn’t using every part of your prompt equally well.

with token meaning and position both encoded, the next question is how tokens actually exchange information.

attention

this is the mechanism that gave the architecture its name. inside every transformer layer, attention does one thing: it lets each token look at the other tokens it’s allowed to see and decide which ones matter for what comes next.

it does this by giving each token three roles at once. each token gets transformed into three new vectors — query, key, and value (q, k, v).

q, k, v — query means “what am i looking for,” key means “what do i match with,” and value is the information that gets copied when the match is strong.

the query asks, “what am i looking for from other tokens?” the key says, “this is what i offer to tokens looking at me.” the value carries, “this is what gets passed along when a match happens.” the same token plays all three roles at once — the q, k, v transformations are learned matrices, so the model figures out during training what each token should look for and what it should offer.

matching happens through a similarity score: each token’s query is compared against the key of each token it’s allowed to see, using a scaled dot product — a measure of how much the two vectors line up. the scaling keeps the numbers stable before softmax.

dot product — a simple way to score how aligned two vectors are. higher alignment means a stronger match.

the match scores get turned into weights using softmax, which takes any set of numbers and turns them into a probability-like distribution that sums to 1. tokens with higher match scores get higher weights, and those weights are then used to take a weighted average of the value vectors.

softmax — turns raw scores into weights that add up to 1. big scores get big weights, small scores get small weights.

an example: “the cat that i saw yesterday was sleeping.” when the model processes “was,” it needs to figure out what’s doing the sleeping. the query vector for “was” gets compared against the key vectors of the tokens it’s allowed to see. the dot product with “cat” is high, because the model has learned that verbs like “was” need a subject and subjects like “cat” produce keys that line up well. the dot product with “yesterday” is low. softmax turns those scores into weights — “cat” gets a high weight, “yesterday” gets a low one — and the model takes a weighted sum of the value vectors, so the value for “cat” dominates the result. the new representation of “was” is now mostly shaped by the value of “cat.” that’s how a token several positions back becomes the referent.

there’s a constraint specific to gpt-style language models: they generate text left to right. a token at position 5 can only attend to positions 1 through 5 — it cannot attend to positions 6, 7, 8, because those haven’t been generated yet. this is causal masking, implemented simply: future tokens get match scores so low they end up with effectively zero weight after softmax.

causal masking — hides future tokens. it keeps a decoder-only language model from looking ahead while predicting the next token.

one of the most interesting findings in interpretability research is about specialized attention heads called induction heads, found by anthropic in 2022. these learn to spot patterns of the form “a b … a” and predict that b comes next — when the model sees “a” the second time, the induction head looks back to where “a” appeared before, sees what came after, and copies that. they’re one of the clearest known mechanisms behind in-context learning: the ability of an llm to pick up a pattern from your prompt and continue it.

induction head — an attention head that notices repeated patterns in the prompt and helps continue them.

attention has one big cost: in full attention, each token compares against all the tokens it’s allowed to see, so doubling the prompt length roughly quadruples the work. that’s why long prompts are expensive to run, and why a lot of recent research is about making attention more efficient (flash attention, sparse attention, linear attention).

but one attention head only gives the model one learned view of those relationships.

multi-head attention

a single attention pass gives the model one way of deciding which tokens matter to which other tokens. that’s not enough — language has many relationships happening at once. subject and verb agreement. pronouns and the names they refer to. long-range references between sentences. word order and local phrases.

multi-head attention solves this by running attention many times in parallel, with each parallel pass — called a head — operating in its own smaller space.

attention head — one independent attention pass with its own learned projections.

the part that’s often described wrong, including in plenty of tutorials: each head doesn’t get a literal slice of the original token vector. each head has its own learned projection matrices that map the full token vector down to its own smaller q, k, v vectors. if a model has 4,096 numbers per token and 32 heads, each head usually works in a 128-dimensional space — but those 128 numbers are a learned projection of the full 4,096, not a fixed slice. different views of the same token, not different chunks of it.

each head runs its attention pass independently. then the outputs of all heads get concatenated and passed through a final linear layer that mixes them back into one full-size vector — the model learns that final mixing too.

what makes this interesting is that different heads often end up partially specialized, without ever being told what to do. specialization emerges naturally during training. researchers have found heads that track grammar (linking verbs to their objects, articles to their nouns), heads that figure out which pronoun refers to which name, heads that track positional patterns, induction heads, and many more. a single transformer layer might have 32 heads; a modern frontier model has dozens of layers — so a typical llm has thousands of attention heads in total, each adding its own learned view.

there’s a practical cost concern that drove a recent architectural change. each head needs to keep its key and value vectors in memory for every token already generated, so the model doesn’t have to recompute everything from scratch when it generates the next one. this is the kv cache, and it’s the main memory cost of running an llm at long context lengths.

kv cache — stores old key and value vectors during generation. it saves the model from recomputing the whole prompt every time it adds a token.

modern decoder-only llms mostly use a variant called grouped-query attention (gqa). instead of every head having its own keys and values, groups of heads share the same key/value heads. llama-2 70b has 64 query heads but only 8 key/value heads. mistral 7b has 32 query heads and 8 key/value heads. the result is nearly the same accuracy as full multi-head attention with much less memory pressure and inference cost.

gqa — grouped-query attention lets multiple query heads share fewer key/value heads. that cuts kv-cache memory while keeping many query views.

feed-forward network

after attention finishes mixing information between tokens, every layer has a second step that nobody talks about as much: the feed-forward network.

where attention is about tokens talking to each other, the feed-forward network is about each token, on its own, doing more processing. it runs on every token’s vector independently, with no cross-token mixing. it does three things in order: expand the token’s vector to a larger size (the original transformer used 4×, modern swiglu models often use different expansion sizes), apply a non-linear function, then compress the vector back down to its original size.

that non-linear step in the middle is doing something specific worth understanding. a non-linearity is a function that bends its input — the simplest one, relu, outputs zero for any negative number and passes positive numbers through unchanged.

non-linearity — a function that prevents the network from collapsing into one big linear transformation.

without it, the ffn would just be two linear layers stacked together, and stacking pure linear math collapses — two linear layers in a row are mathematically equivalent to a single linear layer, and a hundred linear layers in a row are still equivalent to one. the non-linearity is what stops that collapse, and it’s the reason the ffn can do something richer than a single matrix multiplication. the original transformer used relu, gpt and bert moved to gelu, modern models like llama, mistral, and palm use swiglu. the expand-then-compress structure stayed the same — the non-linearity itself is what’s been iterated on.

most of the parameters in a dense transformer model live in the ffn, not in attention. and those parameters aren’t generic — they’re where much of the model’s stored factual and semantic structure lives. researchers have found that some neurons inside the ffn are strongly associated with specific concepts: one neuron might activate strongly on eiffel-tower-related text, another on programming languages, another on past-tense verbs. when a model “knows” paris is the capital of france, that fact is represented across ffn weights and activations in specific layers.

this stored-memory property has an interesting consequence. researchers have figured out how to directly edit some facts in a trained model without retraining it. methods like rome (rank-one model editing) can change “the eiffel tower is in paris” to “the eiffel tower is in rome” by making a targeted low-rank edit to a specific ffn weight matrix. the model then tends to generate text consistent with the edited association.

some modern frontier models have started replacing the dense ffn with mixture of experts (moe). instead of one feed-forward network per layer, the model has many parallel ffns (experts) and a tiny router network that picks which experts process each token. mixtral 8x7b has 8 experts per layer, only 2 activated for any given token. total parameter count goes up substantially, but compute per token grows much more slowly since only a few experts run — that’s how you scale parameter count without scaling inference cost in proportion.

moe — mixture of experts means the model has several feed-forward networks and routes each token through only a few of them.

mixtral 8x7b has 46.7 billion total parameters but uses about 12.9 billion per token. this has become a common option for very large models because it lets you keep growing parameter count without making inference cost grow in proportion.

residual stream and layer normalization

the residual stream is what makes the model “additive” instead of “replacing.” after attention runs, or after the feed-forward network runs, the result usually doesn’t replace the token’s vector — it gets added to it, position by position. the new vector equals the old vector plus the sub-block’s output.

residual connection — adds a block’s output back to the vector it started from. it gives information and gradients a shortcut through the network.

across thirty or fifty or a hundred layers, each layer’s contribution accumulates instead of overwriting the previous vector. that running sum is the residual stream, and it has a strange property: the original input embeddings still have a direct additive path into late layers, mixed together with every sub-block’s contribution along the way.

residual connections weren’t invented for transformers — they came from resnet (he et al., 2015), originally for image recognition. the motivation was that deep networks were impossible to train: the training signal got too weak (or too strong) by the time it traveled back through many layers, so the model couldn’t actually learn from its own mistakes. adding a shortcut path let the signal flow directly back from output to input, and suddenly you could train networks with hundreds of layers. transformers inherited the same trick. in modern interpretability research, the residual stream has become the central object — every component, every head, every ffn, even the unembedding step at the end, reads from it and writes back to it.

layer normalization exists for a much more practical reason. without it, the residual stream wouldn’t stay stable — numbers flowing through dozens of additions tend to either explode upward or collapse toward zero, and either way, training fails. layer normalization rescales each token’s vector back into a controlled range between sub-blocks.

layer normalization — rescales a token vector so its numbers stay in a stable range while the model trains.

the original 2017 transformer applied normalization after each sub-block (post-norm). that worked for shallow models but became harder to train reliably as depth increased. modern transformers (gpt-2 onward, llama, mistral) commonly apply normalization before each sub-block (pre-norm) — one of the changes that made very deep transformers easier to train.

the function itself has also changed. many modern open models (llama, mistral, gemma, phi) use a simpler variant called rmsnorm. the original layer normalization did two things at once — shift each vector toward zero, then rescale its size. rmsnorm drops the shift step and keeps only the rescaling. empirically, the rescaling carries most of the benefit while being cheaper to compute.

rmsnorm — a cheaper normalization method that rescales vector size without subtracting the mean first.

without residual connections, very deep models become much harder to train. without layer normalization, the running sum can blow up or collapse. with both, you get models hundreds of layers deep.

next-token prediction

after all the layers of attention and feed-forward processing finish, the model has a vector for each token in the sequence. to predict the next word, it takes the final vector of the last token only.

that last vector gets converted into one number per possible next token — if the vocabulary has 100,000 tokens, that’s 100,000 numbers, called logits. they aren’t probabilities yet. they can be any size, positive or negative.

logits — raw scores for each possible next token. they become probabilities only after softmax.

a softmax turns those logits into the model’s probability distribution over possible next tokens — same operation as before, different place in the model.

the model usually doesn’t just pick the highest-probability token every time. decoding settings control how deterministic or varied the output is. temperature changes how sharp the distribution is; top-k and top-p limit the choices to the most plausible next tokens. that’s why the same model can feel precise in one setting and more creative in another.

temperature — controls randomness during sampling. low temperature makes the model more conservative; high temperature makes it more varied.

once a token is picked, it gets added to the input, and the model runs the next step on the longer sequence — usually reusing the kv cache so it doesn’t recompute the whole prefix from scratch. new attention for the new token, new feed-forward, new final vector, new prediction. the loop continues until the model emits an end-of-sequence token or hits a length limit. a whole paragraph is just this loop, one token at a time.

this single objective — predicting the next token — is the core training signal for a base llm. the base model isn’t trained on factual accuracy, conversational ability, reasoning, or coding directly. it’s trained to predict the next token in massive amounts of text. post-training then tunes it for instruction following, preference, safety, and conversational behavior.

there’s a major efficiency innovation worth knowing about: speculative decoding. a small, fast model proposes several tokens ahead; the big model verifies them in parallel. if the proposed tokens are accepted under the big model’s probabilities, they’re kept — if not, it falls back to the big model. done correctly, the output distribution matches running the big model alone, but the loop runs much faster.

speculative decoding — uses a small draft model to guess ahead, then asks the larger model to verify several guessed tokens at once.

the next-token loop is the simplest part of the architecture, but it’s what makes the whole thing work.

architecture vs trained weights

we’ve gone through the core mechanisms: tokens, embeddings, positional encoding, attention, multi-head attention, the feed-forward network, the residual stream and normalization, and the next-token loop on the output side. that’s the basic architecture in one pass.

so what’s actually different between gpt and claude and gemini and llama? public details vary, and proprietary models don’t publish every architectural choice — but at the level this post covers, they broadly sit in the same transformer-family design space. most modern transformer-based llms use the same broad structure: tokenization, embeddings, positional encoding, stacked transformer layers (each with multi-head attention and a feed-forward network), residual streams, layer normalization, and next-token prediction.

what changes between models: the trained weights themselves, learned from different training data at different scales; the configuration — number of layers, vocabulary size, head count, parameter count, moe or dense; and the post-training — instruction tuning, learning from human feedback, safety controls applied on top of the base model.

weights — the learned numbers inside the model. training changes those numbers until the model predicts text well.

the 2023–2025 “modern transformer” stack converged on a common set of choices across many serious frontier and open-weight models, even though different teams arrived at them independently: pre-norm placement, rmsnorm, rope, swiglu, grouped-query attention, mixture of experts in some of the largest models. none of these were invented at once — they accumulated over about five years of refinement on top of the original 2017 design.

where this is going

the convergence around transformer-family architectures is unusual in machine learning history. for most of the field’s life, every problem had its own specialized network — image recognition used one kind, language another, audio a third, and vision/language teams barely shared methods. now transformer-style models show up across language, vision, audio, and multimodal systems. the transformer absorbed a huge part of the field.

that could change. mamba and other state-space models are credible alternatives, especially for very long sequences. hybrid architectures are being explored. mixture-of-experts has already shifted what “the architecture” means at the frontier in ways that would have been considered exotic five years ago.

but the core mechanisms in this post — tokens, embeddings, positional encoding, attention, the feed-forward network, the residual stream and normalization, and next-token prediction — are the durable parts. even when the architecture changes, these are the problems any sequence model has to solve in some form.

if you’ve made it this far, you can read a modern transformer paper or model card and know which piece each section is talking about. that’s the goal.