arXiv 2101.03961
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
By William Fedus, Barret Zoph, et al.
Published 2021-01-11
Wiki summary
Explore the paper's summary, context, and related research on Papiers.
In deep learning, models typically reuse the same parameters for all inputs. Mixture of Experts (MoE) defies this and instead selects different parameters for each incoming example. The result is a sparsely-activated model -- with outrageous numbers of parameters -- but a constant computational cost. However, despite several notable successes of MoE, widespread adoption has been hindered by complexity, communication…