arXiv 2203.15556
Training Compute-Optimal Large Language Models
By Jordan Hoffmann, Sebastian Borgeaud, et al.
Published 2022-03-29
Citation lineage
Review the prior work and downstream research connected to this paper.
We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to…