Transformer Architecture

Before Transformers, sequence models like RNNs and LSTMs processed text one word at a time, left to right. This was slow and made it hard to capture long-range dependencies — the model would 'forget' information from earlier in the sequence.

The 2017 paper 'Attention Is All You Need' by Vaswani et al. introduced the Transformer, which processes all tokens in parallel using self-attention. Each token computes attention scores with every other token, creating a rich understanding of context. A sentence about 'the bank of the river' correctly interprets 'bank' differently than 'the bank account.'

Transformers consist of encoder and decoder stacks (though most modern LLMs use decoder-only architectures). Each layer contains multi-head attention (looking at the input from multiple perspectives) and feed-forward networks (processing the attended information).

Understanding Transformers isn't just academic — it explains practical limitations and capabilities. The quadratic cost of attention (every token attends to every other token) is why context windows were initially limited and why longer contexts cost more. Innovations like Flash Attention, sliding window attention, and mixture-of-experts address these costs.

The architecture's flexibility is why it's been adapted for images (Vision Transformers), audio (Whisper), video, and even protein folding (AlphaFold 2).