The complete progression
Karpathy's makemore README names seven model families from bigrams to Transformers. This review orders them by the problem each architecture solves: first expand context, then preserve information through time, then remove recurrent training bottlenecks.
Bigram Count one-step transitions No learned representation; one-token context. → Bengio MLP Embeddings + fixed window Learns similarity, but context length is fixed. → RNN Reusable hidden state Context becomes dynamic, but memory is repeatedly rewritten.
LSTM / GRU Gate recurrent memory Improves long-range credit assignment; timesteps remain sequential. WaveNet / CNN Causal dilated convolutions Parallel training across positions; finite receptive field and autoregressive decoding. Both paths lead toward attention → Transformer Conceptual progression reconstructed from the papers linked in the makemore roadmap. WaveNet is not a descendant of LSTM; it is the major convolutional alternative that attacks the same sequence-modelling problem without recurrent state.
1. Bigram: language modelling as counting
A bigram model estimates the probability of the next token from exactly one previous token: P(xₜ xₜ₋₁). For a character model, every row of a count matrix corresponds to the current character and every column corresponds to the next character.
counts[previoustoken, nexttoken] += 1 probabilities = counts / counts.sum(dim=1, keepdim=True) nexttoken = sample(probabilities[currenttoken])
The model is transparent and useful as a baseline. It also exposes the central problem immediately: the next-token distribution cannot depend on anything earlier than xₜ₋₁. A neural language model must both represent tokens and combine a longer context.
2. Bengio's MLP: learn representations and use more context
Bengio et al. (2003) replaced sparse transition tables with distributed representations. Each token indexes an embedding vector. The embeddings from a fixed number of previous positions are concatenated, transformed by an MLP, and projected to vocabulary logits.
contextids # (B, Tcontext) embeddings # (B, Tcontext, C) flatten # (B, Tcontext C) hidden = tanh(flatten @ Wh + bh) logits = hidden @ Wout + bout # (B, vocabsize)
Previous tokens xₜ₋₃, xₜ₋₂, xₜ₋₁ → Shared lookup embeddings → concatenate → Prediction MLP → logits → softmax The key change in Bengio et al. is distributed representation: related tokens can share statistical strength through nearby embedding vectors instead of requiring independent count-table entries.
This solves two bigram weaknesses: it uses several previous tokens and it generalizes through learned geometry. The remaining constraint is architectural. The context length is chosen in advance, and the first MLP layer expects exactly that many concatenated embeddings. The natural next question is whether one reusable state can summarize an arbitrarily long prefix.
3. Vanilla RNN: replace a fixed window with recurrent state
A recurrent neural network reuses the same transition at every timestep. Instead of concatenating a predetermined number of token embeddings, it combines the current input xₜ with the previous hidden state hₜ₋₁.
ht = tanh(xt @ Wx + hprev @ Wh + bh) logitst = ht @ Wy + by
Wx learns how new evidence should affect the state. Wh learns how the existing state should be transformed before it continues. The state remains fixed-size—usually (B, H)—regardless of sequence length. Its values change; its shape does not.
Timestep 1 x1 + h0 → shared RNN cell → h1 Timestep 2 x2 + h1 → shared RNN cell → h2 Timestep 3 x3 + h2 → shared RNN cell → h3 An unrolled recurrent computation based on Mikolov et al. (2010) . The boxes are repeated uses of one cell with shared parameters, not separate layers with separate weights.
Minimal cell in PyTorch
class RNNCell(nn.Module): def init(self, inputsize, hiddensize): super().init() self.Wx = nn.Parameter(torch.randn(inputsize, hiddensize)) self.Wh = nn.Parameter(torch.randn(hiddensize, hiddensize)) self.b = nn.Parameter(torch.zeros(hiddensize))
def forward(self, xt, hprev): return torch.tanh(xt @ self.Wx + hprev @ self.Wh + self.b)
4. BPTT: backpropagation through the unrolled sequence
Backpropagation through time is ordinary backpropagation applied to the unrolled recurrent graph. The loss at a later timestep can influence earlier states because the chain rule follows every recurrent dependency backward.
∂ht / ∂hk = (∂ht / ∂h{t-1}) (∂h{t-1} / ∂h{t-2}) ... (∂h{k+1} / ∂hk)
This is the same global gradient machinery found across deep neural networks. Recurrence does not introduce a different learning algorithm; it makes the computational graph deeper by reusing the same transition through time.
Vanishing gradients Repeated factors below one shrink distant credit signals. 0.8 × 0.8 × ... × 0.8 → 0 Exploding gradients Repeated factors above one amplify them. 1.2 × 1.2 × ... × 1.2 → ∞ The difficulty is the repeated product of recurrent Jacobians. The chain rule is universal; the long, weight-shared path is specific to recurrence.
Saturated tanh derivatives contribute to vanishing gradients, but the recurrent matrix Wh also matters. Gradient clipping limits explosion; it cannot recreate information that has already vanished.
5. LSTM: create a controlled memory path
A vanilla RNN rewrites one hidden state at every timestep. LSTM adds a cell state cₜ and learns three control decisions: what old memory to keep, what candidate information to write, and what memory to expose as the hidden state.
Forget fₜ ⊙ cₜ₋₁ Retain or erase each component of old memory. Write iₜ ⊙ gₜ Control how much candidate content enters memory. Reveal oₜ ⊙ tanh(cₜ) Expose selected memory as the hidden state. Gate-level redraw based on the LSTM formulation used in Graves's sequence-generation paper , with the original architecture credited to Hochreiter and Schmidhuber (1997) .
zt = concat(xt, hprev)
ft = sigmoid(zt @ Wf + bf) # forget gate it = sigmoid(zt @ Wi + bi) # input gate gt = tanh( zt @ Wg + bg) # candidate content ot = sigmoid(zt @ Wo + bo) # output gate
ct = ft cprev + it gt ht = ot tanh(ct)
The gates are element-wise soft masks. The important structural change is the additive cell update. Ignoring indirect gate dependencies, ∂cₜ/∂cₜ₋₁ = fₜ. When fₜ stays near one, the cell offers a much cleaner path for information and gradients than repeatedly rebuilding all memory through tanh(Wh hₜ₋₁ + ...).
Why implementations fuse the four projections
gates = linear(torch.cat([xt, hprev], dim=-1)) # (B, 4H) f, i, g, o = gates.chunk(4, dim=-1) # each (B, H)
f = torch.sigmoid(f) i = torch.sigmoid(i) g = torch.tanh(g) o = torch.sigmoid(o)
ct = f cprev + i g ht = o torch.tanh(ct)
Concatenating the gate weight matrices does not reduce the mathematical operation count substantially. It improves execution by replacing several smaller GPU operations with one larger matrix multiplication, reducing kernel-launch and memory overhead.
6. GRU: compress gated memory into one state
GRU is a sibling architecture, not a later version of LSTM. It removes the separate cell state and uses update and reset gates to control one recurrent hidden state.
zt = sigmoid(xt @ Wz + hprev @ Uz + bz) # update rt = sigmoid(xt @ Wr + hprev @ Ur + br) # reset nt = tanh(xt @ Wn + (rt hprev) @ Un + bn)
ht = zt hprev + (1 - zt) nt
The update gate interpolates between preserved state and candidate state. The reset gate controls how much history participates in constructing that candidate. GRU keeps the main lesson of LSTM—memory changes should be learned selectively—while using fewer states and projections.
7. WaveNet and causal CNNs: model sequences without recurrence
Gating improves recurrent memory, but it does not remove the dependency between adjacent timesteps during training: hₜ still requires hₜ₋₁. WaveNet takes a different route. It predicts autoregressively with causal one-dimensional convolutions, so the representation at position t can use only positions at or before t.
xₜ₋₇ · xₜ₋₆ · xₜ₋₅ · xₜ₋₄ · xₜ₋₃ · xₜ₋₂ · xₜ₋₁ · xₜ dilation 1 dilation 2 dilation 4 dilation 8 stacked causal layers → large past receptive field → next-token distribution A causal dilated convolution grows its receptive field without a recurrent state. Dilation rates 1, 2, 4, and 8 let later layers reach exponentially farther into the past.
Conceptual causal convolution block
filterpath = tanh(causaldilatedconv(x)) gatepath = sigmoid(causaldilatedconv(x)) gated = filterpath gatepath x = x + residualprojection(gated) skip = skip + skipprojection(gated)
Three ideas matter:
Causality: no output can depend on future tokens, so the model remains a valid autoregressive predictor.
Dilation: spaced convolutional filters expand the receptive field quickly without requiring an extremely deep stack.
Residual and skip paths: deep convolutional stacks become easier to optimize, while gated activations control signal flow.
Unlike an RNN, a causal CNN can compute all training positions in parallel because every position depends on known input tokens rather than a newly computed hidden state. Generation is still autoregressive—one new sample or token at a time—but training no longer has the same timestep-by-timestep recurrent bottleneck.
RNNs carry the past forward in a state. WaveNet reaches backward through a finite causal receptive field. Both model ordered context; they organize the computation differently.
Architecture comparison
Model How context is represented Training across positions Main limitation Bigram One previous token Parallel counts No learned representation or long context Bengio MLP Fixed window of embeddings Parallel Context size fixed by architecture RNN Single recurrent hidden state Sequential Vanishing/exploding gradients and state overwrite LSTM Hidden state plus gated cell state Sequential More compute; recurrence remains GRU One gated hidden state Sequential Recurrence remains WaveNet / causal CNN Finite dilated receptive field Parallel Context depends on depth, kernel, and dilation schedule
Why Transformers come next
The pre-Transformer architectures divide into two strategies. Recurrent models compress the prefix into a state. Causal CNNs retain a parallel path but communicate through a fixed receptive field. Self-attention gives every token a learned, content-dependent route to other permitted positions, and the Transformer builds the full model around that operation.
That transition deserves its own implementation-level treatment. The next post starts at (B, T, C) and reconstructs queries, keys, values, scaled dot-product attention, masking, multi-head attention, residual paths, LayerNorm, feed-forward layers, logits, loss, and generation.
TRANSFORMERS FROM SCRATCH →
What to retain
Bigram: establishes the autoregressive objective through local counts.
Bengio MLP: introduces learned embeddings and nonlinear generalization over a fixed context.
RNN: replaces the fixed window with a reusable state carried through time.
BPTT: applies ordinary backpropagation to the unrolled recurrent graph.
LSTM and GRU: learn selective memory updates to improve long-range credit assignment.
WaveNet: removes recurrent training dependencies with causal dilated convolutions.
Transformer: replaces both recurrent state and fixed convolutional routing with content-dependent attention.
References
The model families follow Karpathy's makemore roadmap. The diagrams above are original redraws of the mechanisms described in the cited papers rather than copied figures.
Karpathy's makemore model roadmap
The educational implementation sequence: Bigram, MLP, CNN, RNN, LSTM, GRU, and Transformer.
A Neural Probabilistic Language Model — Bengio et al. (2003)
Learned distributed word representations with a fixed-context feed-forward language model.
Recurrent Neural Network Based Language Model — Mikolov et al. (2010)
A compact recurrent language model with a hidden state carried through the sequence.
Long Short-Term Memory — Hochreiter and Schmidhuber (1997)
The original LSTM paper and its controlled error-flow argument.
Generating Sequences With Recurrent Neural Networks — Graves (2013/2014)
The LSTM sequence-generation reference linked from makemore.
On the Properties of Neural Machine Translation — Cho et al. (2014)
Karpathy's linked GRU encoder-decoder reference.
WaveNet: A Generative Model for Raw Audio — van den Oord et al. (2016)
Causal dilated convolutions as a non-recurrent autoregressive sequence model.
Attention Is All You Need — Vaswani et al. (2017)
The Transformer architecture that replaces recurrent and convolutional sequence paths with attention.