arXiv 2310.15213
Function Vectors in Large Language Models
By Eric Todd, Millicent L. Li, et al.
Published 2023-10-23
Mindmap
Browse the paper's core ideas, clusters, and relationships in a structured outline.
We report the presence of a simple neural mechanism that represents an input-output function as a vector within autoregressive transformer language models (LMs). Using causal mediation analysis on a diverse range of in-context-learning (ICL) tasks, we find that a small number attention heads transport a compact representation of the demonstrated task, which we call a function vector (FV). FVs are robust to changes i…