arXiv 2507.02754
Fast and Simplex: 2-Simplicial Attention in Triton
By Aurko Roy, Timothy Chou, et al.
Published 2025-07-03
Citation lineage
Review the prior work and downstream research connected to this paper.
Recent work has shown that training loss scales as a power law with both model size and the number of tokens, and that achieving compute-optimal models requires scaling model size and token count together. However, these scaling laws assume an infinite supply of data and apply primarily in compute-bound settings. As modern large language models increasingly rely on massive internet-scale datasets, the assumption tha…