arXiv 2305.13245
GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
By Joshua Ainslie, James Lee-Thorp, et al.
Published 2023-05-22
Wiki summary
Explore the paper's summary, context, and related research on Papiers.
Multi-query attention (MQA), which only uses a single key-value head, drastically speeds up decoder inference. However, MQA can lead to quality degradation, and moreover it may not be desirable to train a separate model just for faster inference. We (1) propose a recipe for uptraining existing multi-head language model checkpoints into models with MQA using 5% of original pre-training compute, and (2) introduce grou…