arXiv 2505.06633
Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models
By Isaac Gerber
Published 2025-05-10
Wiki summary
Explore the paper's summary, context, and related research on Papiers.
Decoder-only transformer networks have become incredibly popular for language modeling tasks. State-of-the-art models can have over a hundred transformer blocks, containing billions of trainable parameters, and are trained on trillions of tokens of text. Each transformer block typically consists of a multi-head attention (MHA) mechanism and a two-layer fully connected feedforward network (FFN). In this paper, we exa…