AttentionResiduals
Replacing fixed residual connections with learned softmax attention over preceding layer outputs. Selective depth routing, bounded magnitudes, uniform gradients.
Imagine you are writing a novel, and you have 100 advisors — one for each chapter you have already written. Every time you start a new chapter, you walk into a room where all 100 advisors simultaneously shout their suggestions at equal volume. The advisor who wrote chapter 1 is just as loud as the one who wrote chapter 99. You cannot turn anyone down, you cannot focus on the advisors most relevant to what you are writing now. That is how standard residual connections work in every modern Transformer: each layer adds its output to a running sum, and every preceding layer has exactly equal influence on what comes next.
Now imagine instead that you have a spotlight. You can shine it on the two or three advisors whose chapters are most relevant, hear them clearly, and let the rest fade to background murmur. That is Attention Residuals. Each layer gets a learned query vector — its own spotlight — that attends over all preceding layer outputs via softmax attention, producing a weighted combination where the model chooses which past layers matter most.
This seemingly simple change — replacing fixed addition with learned attention — has surprisingly deep consequences. It simultaneously solves three problems that have plagued deep Transformers since their inception, each of which researchers have tried to patch with separate hacks for years.
Selective Access
Learned attention weights let each layer focus on the most relevant predecessors. Layer 50 might strongly attend to layers 3 and 48 while nearly ignoring everything else — targeted information retrieval that standard residuals fundamentally cannot do.
Bounded Outputs
Because softmax weights sum to 1, the output is a convex combination — it can never exceed the largest individual layer output. No more runaway growth with depth. A 100-layer model has the same output scale as a 10-layer model.
Uniform Gradients
The attention mechanism creates direct gradient highways from any layer to any predecessor. Gradients flow directly through attention weights, keeping magnitude uniform across all depths. No vanishing, no exploding.
Why Now? The Historical Context
Residual connections have been the unchanged backbone of deep networks since He et al. introduced them in ResNet (2015). That is nearly a decade of the AI revolution built on the simplest possible information routing: just add. Highway Networks (2015) tried gating, DenseNet (2017) tried dense connections, DenseFormer (2024) tried learned scalars — but none became standard because they either did not scale, added too much overhead, or failed to match the simplicity of addition. AttnRes is the first approach to demonstrably improve on residuals at production scale with negligible overhead. The key insight was reframing residuals as an attention problem — once you see the connection, the solution almost writes itself.