arXiv 2507.02754
Fast and Simplex: 2-Simplicial Attention in Triton
By Aurko Roy, Timothy Chou, et al.
Published 2025-07-03
Wiki summary
Explore the paper's summary, context, and related research on Papiers.
Recent work has shown that training loss scales as a power law with both model size and the number of tokens, and that achieving compute-optimal models requires scaling model size and token count together. However, these scaling laws assume an infinite supply of data and apply primarily in compute-bound settings. As modern large language models increasingly rely on massive internet-scale datasets, the assumption tha…