Stephen Cheng
A research paper titled Attention Is All You Need in 2017 introduced a revolutionary mechanism that would fundamentally reshape artificial intelligence. At the heart of this breakthrough was self-attention, a simple yet powerful idea that enables models to understand relationships within data by determining which elements deserve focus.
Today, self-attention powers everything from ChatGPT’s conversational abilities to advanced image recognition systems. It is definitely one of the most transformative innovations in machine learning history.
Self-attention is a mechanism that weighs the importance of different elements in a sequence relative to each other. When processing the sentence “The animal didn’t cross the road because it was too tired,” self-attention helps the model understand that “it” refers back to “animal” rather than “road”, a relationship that spans multiple words.

What makes self-attention particularly powerful is its ability to compute these relationships for all positions in a sequence simultaneously. Unlike recurrent neural networks that process sequences step-by-step, self-attention examines every element in relation to every other element in a single forward pass. This parallel computation dramatically reduces training time while capturing dependencies regardless of their distance in the sequence.
It’s important to distinguish self-attention from related mechanisms. In self-attention, a sequence attends to itself. Each element queries information from all positions within the same sequence. Cross-attention, by contrast, allows one sequence to attend to another, as when a decoder attends to encoder outputs during machine translation. Masked self-attention adds an additional constraint: it prevents positions from attending to subsequent positions, ensuring that predictions depend only on known outputs, essential for autoregressive text generation.
Recurrent neural networks, including LSTMs and GRUs, process sequences sequentially, creating a fundamental bottleneck: They struggled with long-range dependencies because information had to pass through many intermediate states, leading to vanishing gradients and forgotten context.
The evolution toward self-attention began with attention mechanisms in RNN-based encoder-decoder architectures. In 2014, an attention mechanism that allowed decoders to access all encoder hidden states rather than relying on a single fixed-length context vector was introduced. This breakthrough addressed the information compression problem, where variable-length input sequences were squeezed into fixed-length representations regardless of their complexity.

Later, the simplified attention mechanisms were proposed, including the scaled dot-product approach that would later influence transformer design. While these early attention mechanisms improved performance, they still depended on sequential RNN processing.

The real turning point arrived with a bold question: Could we abandon recurrence completely and run on attention alone? That single idea sparked the transformer architecture, putting self-attention at the center of how AI understands sequences.
Self-attention transforms input sequences through three learned projections: queries, keys, and values. Think of these as different “views” of the same information, each serving a distinct purpose in the attention computation. For an input sequence with embedding dimension d_model, we create three weight matrices: W_Q, W_K, and W_V. When we multiply our input matrix X by these weights, we obtain:
Typically, these matrices project to a smaller dimension d_k (often d_model divided by the number of attention heads), which manages computational complexity while maintaining representational capacity. This dimensionality reduction especially becomes important when scaling to longer sequences. With Q, K, and V in hand, we can now compute attention itself.
The attention computation itself follows an elegant formula. First, we calculate attention scores by taking the dot product between queries and keys. This operation measures the compatibility between what each position is looking for and what every other position offers.
However, as dimensionality increases, these dot products can grow very large in magnitude, pushing the softmax function into regions with extremely small gradients. To prevent this numerical instability, we scale by dividing by the square root of the key dimension.

This scaling factor maintains appropriate variance throughout the network, ensuring stable gradients during training. Without it, models with high-dimensional embeddings would struggle to learn effectively, as the softmax would output near-one-hot distributions that fail to distribute attention meaningfully.
Self-attention is completely permutation-invariant. Shuffle the input tokens, and we’ll get the same output. Just shuffled accordingly. The mechanism is powerful, but essentially blind to order. For tasks where order matters (like understanding language), we need to explicitly inject positional information.
The original transformer used sinusoidal position encodings, applying sine and cosine functions of different frequencies to each dimension. This choice wasn’t arbitrary. The wavelike patterns allow models to learn to attend to relative positions, and the encoding extends naturally to unseen sequence lengths.
The original transformer used sinusoidal position encodings, applying sine and cosine functions of different frequencies to each dimension. This choice wasn’t arbitrary. The wavelike patterns allow models to learn to attend to relative positions, and the encoding extends naturally to unseen sequence lengths.

More recent innovations include RoPE (Rotary Position Embedding), which rotates query and key vectors in a multi-dimensional space by angles proportional to their positions. After rotation, the dot product between queries and keys naturally encodes relative distance.
Another approach, ALiBi (Attention with Linear Biases), takes a different path by adding position-dependent penalties directly to attention scores. Rather than modifying embeddings, ALiBi biases attention weights to favor nearby tokens, with the bias intensity varying across different attention heads.
After computing scaled attention scores, we apply softmax across the key dimension. This normalization converts raw scores into a probability distribution, ensuring that attention weights sum to one for each query position.
The softmax function emphasizes the highest-scoring relationships while suppressing irrelevant connections. However, this can sometimes lead to overly peaked distributions, particularly during early training when the model hasn’t yet learned nuanced attention patterns. To mitigate this, practitioners often apply dropout to attention weights, randomly zeroing out some connections to encourage robustness.
With attention weights, we can compute the final output as a weighted sum of the value vectors. Each position receives a context-aware representation that aggregates information from across the entire sequence, with contributions proportional to the attention weights. This weighted combination allows information to flow dynamically based on learned relationships rather than fixed patterns.

While single-head attention is powerful, it has a fundamental limitation: it can only capture one type of relationship at a time. Multi-head attention addresses this by running several attention computations in parallel, each focusing on different aspects of the input relationships. For instance, there might be one head each specializing in:
This specialization often emerges organically during training. We don’t design it, which one of the most fascinating aspects of the whole mechanism. The implementation involves splitting the Q, K, and V matrices along the dimension axis, processing each split through separate attention heads, and concatenating the results.

If we have 8 heads and 512-dimensional embeddings, each head operates on 64-dimensional subspaces. This parallel processing doesn’t increase the complexity of the sequence length but does multiply the model’s capacity to learn diverse patterns.
In translation models, some heads consistently attend to positional relationships, focusing on neighboring words regardless of content. Other heads specialize in syntactic roles, identifying subject-verb or verb-object relationships. A third category attends to rare or high-information tokens like proper nouns and technical terms.
Interestingly, interpretability studies show that many heads appear redundant, and models can retain performance even when a significant fraction of heads are pruned. This redundancy likely contributes to robustness, ensuring that critical patterns have multiple pathways through the network. In encoder-decoder attention, heads in later decoder layers prove most essential for translation quality, while encoder self-attention heads can often be reduced substantially.
In the transformer encoder, self-attention layers allow each position to attend to every other position in the input sequence. This “all-to-all” pattern enables bidirectional context flow, where each token builds a representation informed by the entire sequence.
Each encoder layer combines multi-head self-attention with a position-wise feed-forward network. The feed-forward network processes each position independently but identically, applying the same learned transformation across all tokens.

Residual connections and layer normalization surround both sub-layers, stabilizing gradients and enabling deeper architectures. Stacking multiple encoder layers creates increasingly abstract representations, with early layers capturing surface patterns and deeper layers encoding semantic relationships.
The decoder modifies self-attention to respect autoregressive constraints. When generating text, the model can only condition on tokens it has already produced. Peeking at future tokens would constitute cheating during training.

Masked self-attention accomplishes this by setting attention weights to negative infinity (which becomes zero after softmax) for all positions following the current one. This ensures that predictions depend only on previous context, and the necessary causal ordering for generation tasks is maintained. During inference, this causality becomes natural: we literally don’t have future tokens to attend to yet.
Between the decoder’s self-attention layers lies cross-attention, which enables the decoder to condition on encoder outputs. Here, queries come from the decoder (representing “what I need to generate”), while keys and values come from the encoder (representing “what the input provides”).

This cross-attention mechanism is essential for sequence-to-sequence tasks like translation, where the decoder must align output words with relevant input words. Unlike self-attention, cross-attention creates dependencies between two different sequences, allowing information to flow from source to target.
Self-attention is one of the most transformative mechanisms in machine learning history. By enabling models to dynamically determine which elements of their input deserve focus, it solved long-standing challenges in capturing long-range dependencies and processing sequences efficiently.
The mathematical elegance of scaled dot-product attention, enhanced through multi-head architectures, provides both theoretical soundness and practical effectiveness. From powering state-of-the-art language models to revolutionizing computer vision and enabling multimodal AI systems, self-attention continues expanding its reach.
AI, Self-Attention, Transformer — Jan 20, 2026
Made with ❤️ and ☀️ on Earth.