There's a scene in Blade Runner 2049 where K sits in a room and recites answers to preset questions. It's a baseline test. They're checking whether his implanted memories are still intact, whether they've drifted or degraded over time. He passes, and life goes on.
I kept thinking about that scene while reading posts about agent memory systems. Everyone's building these elaborate setups with vector databases, retrieval pipelines, chunking strategies. One embedding per memory, stored in a slot, retrieved by similarity search. It works, but it's also kind of... brute force? Every memory gets its own row in a database. Retrieval scales with how much you've stored.
What if instead you could cram hundreds of memories into a single fixed-size matrix, no slots, no indexing, and pull them back out with a single matrix multiply? Google actually explored something adjacent with Infini-attention, compressing KV states into a fixed-size matrix inside the transformer itself. But I wanted to try it as a standalone, modular memory you could plug into anything.
This concept is borrowed from physics and from how associative memories work in old-school neural network research (Hopfield networks, circa 1982). You take a fixed-size matrix, say 1024×1024, and you store facts on top of each other by adding their outer products:
M = M + outer(key, value)
Each fact doesn't get its own slot. They all share the same matrix, layered together in superposition. To retrieve a fact, you just multiply:
retrieved = M @ query
And here's why it works. If you expand what happens when you query a matrix that has multiple facts stored:
M @ q = Σᵢ valueᵢ * dot(keyᵢ, query)
You get back a similarity-weighted sum of all stored values. If your query matches one key well and is orthogonal to everything else, you get back exactly that value. If keys overlap, you get bleed-through from other values. That's the tradeoff: fixed memory footprint, O(1) retrieval, but degradation as the matrix fills up.
No database, no index. Just linear algebra.
I wanted to test this with real text, not just random vectors. That means I needed a way to go from text to a fixed-size vector and back. After some searching, Meta's SONAR turned out to be a good fit. It's a text autoencoder that compresses any sentence into a single 1024-dimensional vector, and its decoder can reconstruct text from that vector. Short declarative sentences come back character-perfect on the round-trip.
Here's the full pipeline:
text → SONAR encode → 1024-dim vector → store in matrix → retrieve → SONAR decode → text
Our memory matrix is 1024×1024 (matching SONAR's embedding dimension), initialized to zeros. About 4MB of float32. That's the entire memory system.
For the first experiments I used auto-associative storage, where the key and value are the same embedding. You store "Thomas is 8 years old" as outer(v, v) where v is the SONAR embedding of that sentence. To retrieve, you query with the same embedding and check what comes back.
Starting with the obvious sanity check. Store one fact, retrieve it:
| Fact | Cosine Similarity | BLEU |
|---|---|---|
| Thomas is 8 years old | 1.0000 | 100 |
| The capital of France is Paris | 1.0000 | 100 |
| Water boils at 100 degrees Celsius | 1.0000 | 100 |
Perfect across the board. Not surprising, but good to confirm the pipeline works end to end.
Now, what happens when you keep adding facts? I stored up to 93 diverse sentences ("The first email was sent in 1971", "Cheetahs can run up to 70 mph", "Pluto has a heart shaped glacier on its surface", that sort of thing) and after each addition, retrieved all stored facts and measured average cosine similarity between the original and retrieved embeddings.
I also tested the delta rule, a smarter write operation. Instead of blindly adding outer(k, v), it first checks what the matrix already returns for that key and only stores the difference:
v_existing = M @ k
delta = v - v_existing
M = M + outer(k, delta)
This avoids doubling up on information that's already stored.
| Facts Stored | Naive | Delta Rule |
|---|---|---|
| 1 | 1.000 | 1.000 |
| 10 | 0.806 | 0.813 |
| 50 | 0.628 | 0.658 |
| 93 | 0.570 | 0.616 |
Degradation is gradual, no sudden cliff where everything collapses. The delta rule helps by about 3-5% across the board, but it doesn't change the overall picture: cosine similarity drops steadily as load increases. At 93 facts, we're well below 0.7, and decoded text starts getting unreliable.
Here's the thing though. A 1024×1024 matrix can theoretically store 1024 orthogonal vectors perfectly. We're hitting trouble at less than 100. Something's eating our capacity.
I computed pairwise cosine similarities for all 93 fact embeddings to see how orthogonal they actually are:
| Statistic | Value |
|---|---|
| Mean pairwise cosine | 0.195 |
| Max | 0.658 |
In a 1024-dimensional space, truly random vectors would have near-zero pairwise cosine similarity. Our embeddings average 0.195. That doesn't sound like much, but it means every sentence shares a decent chunk of structure with every other sentence: common English grammar, similar sentence lengths, shared vocabulary patterns. The matrix can't tell them apart cleanly.
And the worst offenders make intuitive sense. "The first email was sent in 1971" and "The first website went live in 1991" hit 0.658. Same sentence structure, just different nouns and dates.
Before jumping to neural networks, I tried something simpler: PCA whitening. It's a linear transform that decorrelates the embeddings by removing the mean, rotating to principal components, and scaling each dimension to unit variance. No training, just a PCA fit on the corpus of facts.
After whitening, mean pairwise cosine similarity dropped from 0.19 to 0.01.
| Method | Avg Cosine @ 93 facts |
|---|---|
| Naive (raw SONAR) | 0.570 |
| Delta rule (raw SONAR) | 0.616 |
| PCA-whitened + naive | 0.9998 |
Near-perfect retrieval at 93 facts. The capacity problem was almost entirely embedding geometry, not the matrix mechanism.
But can you actually decode the retrieved vectors back to text? PCA whitening produces vectors in a different space than what SONAR's decoder expects. The trick is that the whitening transform is linear with a well-defined inverse, so you un-whiten the retrieved vector before feeding it to the decoder.
Full round-trip: whiten → store in matrix → retrieve → un-whiten → SONAR decode.
All 5 sampled facts at 93 total stored came back character-perfect. Average BLEU across all 93 facts was 93.5. And here's the interesting part: every single BLEU drop turned out to be SONAR's decoder paraphrasing, not memory errors. Every one of these had 1.0 cosine similarity in embedding space, meaning the matrix returned the right vector. The decoder just preferred different phrasing:
| Original | Decoded As | BLEU |
|---|---|---|
| Gravity on the Moon is about one sixth of Earth | ...one sixth of Earth**'s** | 88.0 |
| Iron is the most abundant element in Earth's core | ...in the Earth's core | 70.7 |
| A group of flamingos is called a flamboyance | A group of flamingoes... | 50.0 |
| Honeybees can recognize human faces | Honey bees can recognize... | 50.8 |
| The Olympics originated in ancient Greece | The Olympic Games originated... | 43.5 |
| A sneeze travels at about 100 miles per hour | ...about 160 miles per hour | 59.7 |
| Humans spend about one third of their lives sleeping | ...about a third of their life... | 31.0 |
| Pluto has a heart shaped glacier on its surface | ...a glacier in the form of a heart... | 22.0 |
| More people have cell phones than toilets worldwide | People worldwide have cell phones more than toilets | 27.1 |
Almost all of these are benign: alternate spellings, article insertions, mild rephrasing. One genuine factual error though: the sneeze speed (100 → 160 mph), which is a decoder hallucination, not a memory failure.
The sneeze one still makes me laugh. The matrix did its job perfectly, but SONAR's decoder decided 100 mph wasn't dramatic enough. Can't win them all.
One takeaway: BLEU is the wrong metric for this system. Cosine similarity in embedding space is the reliable measure. BLEU punishes you for things that aren't the memory's fault.
All the results above use auto-associative storage: you store a fact and retrieve with that same fact's embedding. But what you'd actually want for an agent is hetero-associative: store an answer, retrieve it with a question. Store "Thomas is 8 years old" as the value, use the embedding of "How old is Thomas?" as the key.
Unfortunately, this falls apart. 25 QA pairs, average cosine similarity of 0.41, and every query decoded to some variant of "How much of a planet is it?" or "What is the moon?"
In hindsight, the reason is pretty straightforward. Question embeddings all share the "What is...", "How many...", "Where does..." structure, so their keys are even more correlated than declarative facts. The matrix can't distinguish between them and returns a blended average of all stored values. PCA whitening doesn't help either. If you fit it on declarative facts, question embeddings are out-of-distribution and map to garbage.
So this is where learned projections come in. A neural network that maps questions into a key-space where they're actually distinguishable, separate from the value-space where answers live. That's the next step.
A 4MB matrix, a PCA fit, and a pretrained text autoencoder. For auto-associative recall, it works. 93 facts stored and retrieved with near-perfect fidelity, no training required.
I still haven't pushed past 93 facts to find where the whitened curve actually bends, and the QA case needs learned projections to handle the key distribution mismatch. But the core mechanism checks out: superposed storage in a fixed-size matrix, with O(1) write and O(1) read, degrades gradually under load instead of falling over. And the capacity problem turned out to be fixable with a simple linear transform.
Next up is training key/value projectors to crack the hetero-associative case. The PCA-whitened system sets a strong baseline to compare against.
If you want to try this yourself, SONAR is pip install sonar-space (Linux/WSL only, fairseq2 doesn't support Windows), and the matrix operations are literally three lines of PyTorch.