Newshot

Geometric Factual Recall in Transformers

Summary

arXiv:2605.12426v2 Announce Type: replace Abstract: How do transformer language models memorize factual associations? A common account views internal weight matrices as an associative memory. That account requires parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, \emph{geometric} form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer memorizes random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension is sufficient. Subject embeddings encode \emph{linear superpositions} of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute from the embedding. We extend these results to the multi-hop setting---chains of relational queries such as ``Who is the mother…

This source only provides a headline — there is no body to show. Use “Read the original” below to read it at the source.

Published by arXiv cs.CL
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗