arXiv 2006.10413
Are Pretrained Language Models Symbolic Reasoners Over Knowledge?
By Nora Kassner, Benno Krojer, et al.
Published 2020-06-18
Discussion
Read the public discussion and references gathered around this paper.
How can pretrained language models (PLMs) learn factual knowledge from the training set? We investigate the two most important mechanisms: reasoning and memorization. Prior work has attempted to quantify the number of facts PLMs learn, but we present, using synthetic data, the first study that investigates the causal relation between facts present in training and facts learned by the PLM. For reasoning, we show that…