Why transformers were a breakthrough
Earlier NLP systems relied on recurrent networks and LSTMs. Sequential processing limited parallel training and made long-range dependencies difficult. Transformers removed recurrence and made attention primary operation.
- Faster parallel training
- Stronger long-context modeling
- Architecture reusable across tasks
Attention over sequence
Self-attention lets each token weigh every other token. Multiple heads learn different syntactic, semantic, and long-range relationships at same time. Positional encodings preserve order that recurrence used to provide.
Core architecture
Transformer block combines token embeddings, positional information, multi-head attention, feed-forward transformations, residual paths, and normalization.
- Encoder-only systems suit classification, retrieval, and embeddings
- Decoder-only systems generate text token by token
- Encoder-decoder systems suit translation and sequence transformation
Advantages and limits
Transformers scale, transfer, and parallelize well, but attention cost grows quickly with sequence length. They remain compute- and memory-intensive, especially for long documents and low-resource environments.
- Benchmark smaller or sparse alternatives
- Measure memory and latency, not only accuracy
- Treat retrieval, evaluation, and serving as part of system