arXiv 2004.05150
Longformer: The Long-Document Transformer
By Iz Beltagy, Matthew E. Peters, et al.
Published 2020-04-10
Mindmap
Browse the paper's core ideas, clusters, and relationships in a structured outline.
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the st…