arXiv 2004.05150
Longformer: The Long-Document Transformer
By Iz Beltagy, Matthew E. Peters, et al.
Published 2020-04-10
Discussion
Read the public discussion and references gathered around this paper.
Transformer-based models are unable to process long sequences due to their self-attention operation, which scales quadratically with the sequence length. To address this limitation, we introduce the Longformer with an attention mechanism that scales linearly with sequence length, making it easy to process documents of thousands of tokens or longer. Longformer's attention mechanism is a drop-in replacement for the st…