arXiv 2004.05150
Longformer: The Long-Document Transformer
By Iz Beltagy, Matthew E. Peters, et al.
Published 2020-04-10
Wiki summary
Explore the paper's summary, context, and related research on Papiers.
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the st…