arXiv 2104.09864
RoFormer: Enhanced Transformer with Rotary Position Embedding
By Jianlin Su, Yu Lu, et al.
Published 2021-04-20
Citation lineage
Review the prior work and downstream research connected to this paper.
Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to e…