arXiv 2512.19428
Attention Is Not What You Need
By Zhang Chong
Published 2025-12-22
Wiki summary
Explore the paper's summary, context, and related research on Papiers.
We revisit a basic question in sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning? We argue that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining this lifted tensor through gradient descent. This mechanism is extremely expres…