---
title: Memory Mosaics
type: vocabulary
url: "https://www.envisioning.com/vocab/memory-mosaics"
summary: A neural architecture built from a network of associative memories that each store and retrieve key-value pairs, used as an alternative to transformer attention.
year: 2024
generality: 0.40
---

# Memory Mosaics

A neural architecture built from a network of associative memories that each store and retrieve key-value pairs, used as an alternative to transformer attention.
Memory Mosaics is a neural network architecture built from a network of associative memories rather than the attention mechanism used in transformers. Jianyu Zhang, Niklas Nolte, Ranajoy Sadhukhan, Beidi Chen, and Leon Bottou, at FAIR (Meta), Carnegie Mellon, and NYU, introduced it in the 2024 paper "Memory Mosaics" (arXiv:2405.06394), later presented at ICLR 2025. Each associative memory in the network stores and retrieves key-value pairs tied to a specific type of context, such as a persistent fact or a pattern seen earlier in the current input. Many of these memories work together to produce a prediction.

The paper's central claim is that this division of labor across memories, rather than one shared attention computation, gives the network what the authors call predictive disentanglement. Different memories end up specializing in different sub-tasks of the overall prediction automatically, in a way the authors argue is more interpretable than a transformer's distributed attention patterns. On toy tasks and medium-scale (GPT-2-scale) language modeling, the original paper reported that Memory Mosaics matched or exceeded transformer performance while retaining the compositional and in-context learning behavior transformers show.

A 2025 follow-up paper, "Memory Mosaics at scale" (arXiv:2507.03285, NeurIPS 2025), by Zhang and Bottou, scaled the architecture to Llama-8B-sized networks trained on one trillion real-world tokens, with three added architectural changes the authors call Memory Mosaics v2. They reported it matched transformers on learning the training data itself, but outperformed a transformer trained on eight times as many tokens (eight trillion) at storing new knowledge introduced after training and at in-context learning. The authors state these gains could not be reproduced by simply giving the transformer more training data.

---
Source: Envisioning — Technology Research Institute (https://www.envisioning.com/vocab/memory-mosaics)