---
title: LPT (Looped Prediction Transformer)
type: vocabulary
url: "https://www.envisioning.com/vocab/lpt-looped-prediction-transformer"
summary: A looped language model where only the first recursion writes a key-value cache and later recursions read that shared cache instead of each writing their own.
year: 2026
generality: 0.40
---

# LPT (Looped Prediction Transformer)

A looped language model where only the first recursion writes a key-value cache and later recursions read that shared cache instead of each writing their own.
LPT (Looped Prediction Transformer) is a looped language model in which only the first recursion writes a key-value cache. Later recursions read that shared cache instead of each writing their own. Giovanni Monea, Keshav Ramji, Yousef El-Kurdi, Luis A. Lastras, Yoav Artzi, Nathan Godey, and Ramon Fernandez Astudillo, at Cornell University and IBM Research, introduced it in the October 2026 paper "The Surprising Effectiveness of Shared Memory in Looped Transformers" (arXiv:2610.02383). It addresses a memory cost specific to looped transformers: applying the same layers multiple times per token adds compute without adding parameters, but if every recursion writes its own cache, memory still grows with the number of loops.

LPT keeps a short window of each recursion's own local cache but shares one cache, written only by the first pass, across all later recursions. A hybrid variant combines this shared-cache attention with gated DeltaNet layers. At 150 million to 1 billion parameters, the authors report that sharing memory does not cost quality and instead improves it. With five recursions, the hybrid variant lowers validation perplexity on FineWeb-Edu by 1.12 to 1.82 points relative to a same-size standard transformer, while using 76 to 79 percent less context memory.

The paper's analysis finds that shared and local memory develop distinct representations inside the model, and that later recursions attend mostly to the shared memory rather than their own local cache. The authors also find the shared cache acts as a gradient highway back to the first recursion during training. They offer this as part of the explanation for why memory sharing improves rather than merely preserves quality.

---
Source: Envisioning — Technology Research Institute (https://www.envisioning.com/vocab/lpt-looped-prediction-transformer)