Research preview · operator-run
Which AI memory system actually works?
Memory systems let an AI remember past conversations. Every project claims to be the best. We collect what they publish, show who measured it, and run our own tests in the open.
UPDATED 30 OCTOBER 2026 · MNEMOSYNE RESULTS
LongMemEval-S: latest recorded results
Three complete 500-question runs from the Mnemosyne repository. Totals rechecked against every recorded verdict.
How the memory systems compare
Published results with a source for every figure. Models, datasets, aggregation and retrieval limits vary; hover a score for its setup. BEAM includes nugget scores and answer scores, identified per cell. “Not reported” opens the source coverage for that system.
| System | Measured by | LongMemEval | LoCoMo | BEAM 1M | BEAM 10M | LongMemEval recall | HotpotQA recall | 2Wiki recall | MuSiQue recall |
|---|---|---|---|---|---|---|---|---|---|
| ByteRover | Self-reported | 92.8%overall accuracy | 96.1%overall accuracy | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Chronos | Self-reported | 95.6% | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Hindsight | Self-reported | 94.6% | 92% | 79.1% | 64.1% | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Mem0 | Mixed sources | 94.4% | 92.5% | 64.1% | 48.6% | 71.32%target recall@15 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Mastra | Self-reported | 94.87%task-averaged accuracy | Not reportedin cited sources | 80.6%published BEAM score | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| MnemosyneOperator entry | Mixed sources | 97.2%486 / 500 correct | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 99.7%session recall@10 | 93.45%Recall@5 | 91.7%Recall@5 | 66.72%Recall@5 |
| MemMachine | Mixed sources | 93%optimized retrieval, gpt-5-mini | 91.7%v0.2, gpt-4.1-mini | Not reportedin cited sources | 51.9%nugget score | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| gbrain | Self-reported | 94.8%claude-opus-5-5 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 95.96%strict recall_all@5 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Honcho | Self-reported | 90.4% | 89.9% | 63.1%Hindsight RAG | 40.6%Hindsight RAG | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| ReMe | Self-reported | 89.4%agentic score | Not reportedin cited sources | 65%agentic score | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| MemOS | Mixed sources | 89.2% | 88.83% | Not reportedin cited sources | 56.75% | 52.07%target recall@15 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| SmartSearch | Self-reported | 88.4% | 93.5%indexed, EverMemOS protocol | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| CORE | Self-reported | Not reportedin cited sources | 88.24%average accuracy | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Memora | Self-reported | 87.4% | 86.3%policy retriever, gpt-4.1-mini | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Memori | Mixed sources | 20.8%OmniMemEval reproduced setup | 87%overall accuracy | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| supermemory | Mixed sources | 85.2%Gemini 3 reader | 73.53%OmniMemEval reproduced setup | Not reportedin cited sources | 52.49%nugget score | 95%Recall@15 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| EMem-G | Self-reported | 84.9% | 85.3%gpt-4.1-mini | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Nemori | Self-reported | 74.6% | 83.05%LLM score | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| EverMemOS | Mixed sources | 83% | 93.05%gpt-4.1-mini | Not reportedin cited sources | 47.73%nugget score | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Cognee | Mixed sources | 51.8%OmniMemEval reproduced setup | 83.48%OmniMemEval reproduced setup | Not reportedin cited sources | 67%score (0 to 1 scale) | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| TiMem | Self-reported | 79% | 75.3%gpt-4o-mini | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| CoM | Self-reported | 76.4% | 72.86%gpt-4o-mini | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Memobase | Reported by another party | Not reportedin cited sources | 75.8% | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Zep | Mixed sources | 79.8%OmniMemEval reproduced setup | 75.14%J score (LLM judge) | Not reportedin cited sources | 56.11%nugget score | 62.74%target recall@15 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| HyMem | Self-reported | 78.8%gpt-4.1-mini | 89.55%gpt-4.1-mini | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| Letta | Reported by another party | 77.67%OmniMemEval reproduced setup | 77.12%OmniMemEval reproduced setup | Not reportedin cited sources | 52.3%nugget score | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| LiCoMemory | Self-reported | 73.8% | 67.2%gpt-4o-mini | Not reportedin cited sources | Not reportedin cited sources | 76.63%target recall@15 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| ENGRAM | Self-reported | 71.4% | 77.55%LLM-judge score | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| A-Mem | Reported by another party | 63.4% | 64.16%ACC | Not reportedin cited sources | Not reportedin cited sources | 62.18%target recall@15 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| MemoryOS | Reported by another party | 61.2% | 58.25%ACC | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| LangMem | Reported by another party | 41.6%CoM study, Qwen3-32B | 58.1%J score (LLM judge) | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| OpenAI memory | Reported by another party | Not reportedin cited sources | 52.9%J score (LLM judge) | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| LIGHT | Self-reported | Not reportedin cited sources | Not reportedin cited sources | 33.6% | 26.6% | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| MemoryBank | Reported by another party | 22.9% | 39.77%TiMem study, gpt-4o-mini | Not reportedin cited sources | Not reportedin cited sources | 39.21%target recall@15 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| MemPalace | Self-reported | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 98.4%R@5 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| HippoRAG 2 | Reported by another party | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 96.3%Recall@5 | 90.4%Recall@5 | 74.7%Recall@5 |
| agentmemory | Self-reported | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 95.2%R@5 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| NV-Embed-v2 | Reported by another party | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 94.5%Recall@5 | 76.5%Recall@5 | 69.7%Recall@5 |
| GritLM-7B | Reported by another party | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 92.4%Recall@5 | 76%Recall@5 | 65.9%Recall@5 |
| HippoRAG | Mixed sources | Not reportedin cited sources | 47.1%Memora study | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 77.7%Recall@5 | 90.4%Recall@5 | 53.2%Recall@5 |
| GTE-Qwen2-7B-Instruct | Reported by another party | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 89.1%Recall@5 | 74.8%Recall@5 | 63.6%Recall@5 |
| RAPTOR | Reported by another party | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 86.9%Recall@5 | 66.2%Recall@5 | 57.8%Recall@5 |
| Contriever | Reported by another party | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 76.2%R@5 | 75.3%Recall@5 | 57.5%Recall@5 | 46.6%Recall@5 |
| BM25 | Reported by another party | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 68.3%R@5 | 74.8%Recall@5 | 65.3%Recall@5 | 43.5%Recall@5 |
| GTR (T5-base) | Reported by another party | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 73.9%Recall@5 | 67.9%Recall@5 | 49.1%Recall@5 |
| Stella V5 1.5B | Reported by another party | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | 73.2%R@5 | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
| OpenViking | Reported by another party | 61.07%OmniMemEval reproduced setup | 69.33%OmniMemEval reproduced setup | Not reportedin cited sources | 58.14%nugget score | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources | Not reportedin cited sources |
Answering questions
The two benchmarks everyone quotes, plus how long a query takes.
LongMemEval
Answer accuracy · higher is better
| System | Value | Metric | Source |
|---|---|---|---|
| Mnemosyne | 97.20% | accuracy | Mnemosyne LongMemEval-S results, October 28–29, 2026 |
| Chronos | 95.6% | accuracy | Chronos paper, arXiv 2603.16862 |
| Mastra | 94.87% | task-averaged accuracy | Mastra Observational Memory research |
| gbrain | 94.8% | accuracy | gbrain-evals current results |
| Hindsight | 94.6% | accuracy | Agent Memory Benchmark, run by Vectorize (maker of Hindsight) |
Five highest reported figures, one per system. See linked sources for models and scoring.
LoCoMo
Answer accuracy · higher is better
| System | Value | Metric | Source |
|---|---|---|---|
| ByteRover | 96.1% | overall accuracy | ByteRover README |
| SmartSearch | 93.5% | accuracy | SmartSearch paper, arXiv 2603.15599 |
| EverMemOS | 93.05% | accuracy | EverMemOS paper, arXiv 2601.02163 |
| Mem0 | 92.5% | score | Mem0 README |
| Hindsight | 92% | accuracy | Agent Memory Benchmark, run by Vectorize (maker of Hindsight) |
Five highest reported figures, one per system. See linked sources for models and scoring.
Speed
Seconds per query · lower is better
| System | Value | Metric | Source |
|---|---|---|---|
| OpenAI memory | 0.466 s | total latency p50 | Mem0 paper, arXiv 2504.19413 |
| Mem0 | 0.708 s | total latency p50 | Mem0 paper, arXiv 2504.19413 |
| Mem0 graph | 1.091 s | total latency p50 | Mem0 paper, arXiv 2504.19413 |
| Zep | 1.292 s | total latency p50 | Mem0 paper, arXiv 2504.19413 |
| A-Mem | 1.41 s | total latency p50 | Mem0 paper, arXiv 2504.19413 |
Five fastest reported configurations. Lower is better.
Finding the right memory
Before a system can answer, it has to retrieve the right thing. This is measured separately and never mixed with accuracy.
Finding the evidence
Was the right session in the top five? · higher is better
LongMemEval, recall at five. Projects count a hit differently, so these are not directly comparable — including ours.
Strict recall
Every needed session found · higher is better
A harder count: every session a question needs must be in the top five. Run by gbrain’s authors.
Slow tail
Worst case, seconds · lower is better
| System | Value | Metric | Source |
|---|---|---|---|
| OpenAI memory | 0.889 s | total latency p95 | Mem0 paper, arXiv 2504.19413 |
| Mem0 | 1.44 s | total latency p95 | Mem0 paper, arXiv 2504.19413 |
| Mem0 graph | 2.59 s | total latency p95 | Mem0 paper, arXiv 2504.19413 |
| Zep | 2.926 s | total latency p95 | Mem0 paper, arXiv 2504.19413 |
| A-Mem | 4.374 s | total latency p95 | Mem0 paper, arXiv 2504.19413 |
Five fastest reported configurations. Lower is better.
Very long conversations
BEAM runs the same memory over 100 thousand, 1 million and 10 million tokens of history.
BEAM 100K
Very long conversations · higher is better
| System | Value | Metric | Source |
|---|---|---|---|
| Hindsight | 86.2% | accuracy | Agent Memory Benchmark, run by Vectorize (maker of Hindsight) |
| Mastra | 83.3% | published BEAM score | Mastra BEAM research, October 29, 2026 |
| Cognee | 79% | score (0 to 1 scale) | Cognee README |
| OpenViking | 70.76% | nugget score | OmniMemEval reproduced result snapshot |
| Mem0 | 70.41% | nugget score | OmniMemEval reproduced result snapshot |
Five highest reported figures, one per system. See linked sources for models and scoring.
BEAM 1M
Very long conversations · higher is better
| System | Value | Metric | Source |
|---|---|---|---|
| Mastra | 80.6% | published BEAM score | Mastra BEAM research, October 29, 2026 |
| Hindsight | 79.1% | accuracy | Agent Memory Benchmark, run by Vectorize (maker of Hindsight) |
| ReMe | 65% | agentic score | ReMe README |
| Mem0 | 64.1% | score | Mem0 README |
| Honcho | 63.1% | accuracy | Honcho evaluation site |
Five highest reported figures, one per system. See linked sources for models and scoring.
BEAM 10M
Very long conversations · higher is better
| System | Value | Metric | Source |
|---|---|---|---|
| Cognee | 67% | score (0 to 1 scale) | Cognee README |
| Hindsight | 64.1% | accuracy | Agent Memory Benchmark, run by Vectorize (maker of Hindsight) |
| OpenViking | 58.14% | nugget score | OmniMemEval reproduced result snapshot |
| MemOS | 56.75% | score | MemOS README |
| Zep | 56.11% | nugget score | OmniMemEval reproduced result snapshot |
Five highest reported figures, one per system. See linked sources for models and scoring.
Who ran the test changes the answer
The same software scores differently depending on who measures it and how. This is the case for one neutral operator running everything.
Same system, two scorekeepers
Zep on LoCoMo · higher is better
| Run by Mem0 | 65.99% |
|---|---|
| Run by Zep | 75.14% |
Same software, two scorekeepers, nine points apart. This is why one neutral operator should run everything.
Reader changes the score
LongMemEval · higher is better
| Zep gpt-4o | 71.2% |
|---|---|
| Full context gpt-4o | 60.2% |
| Zep gpt-4o-mini | 63.8% |
| Full context gpt-4o-mini | 55.4% |
The same memory, two different answering models. Compare each bar only with the no-memory baseline beside it.
One harness, run by others
LoCoMo · higher is better
| FullText | 73.83% |
|---|---|
| A-Mem | 64.16% |
| NaiveRAG | 63.64% |
| Mem0 | 61.69% |
| Mem0 graph | 60.32% |
| MemoryOS | 58.25% |
| Mem0 | 36.49% |
Every row here ran on the same harness. Look at Mem0 free versus paid: the same product, 25 points apart.
What projects claim about themselves
LoCoMo · higher is better
| System | Value | Metric | Source |
|---|---|---|---|
| ByteRover | 96.1% | overall accuracy | ByteRover README |
| SmartSearch | 93.5% | accuracy | SmartSearch paper, arXiv 2603.15599 |
| EverMemOS | 93.05% | accuracy | EverMemOS paper, arXiv 2601.02163 |
| Mem0 | 92.5% | score | Mem0 README |
| Hindsight | 92% | accuracy | Agent Memory Benchmark, run by Vectorize (maker of Hindsight) |
Five highest reported figures, one per system. See linked sources for models and scoring.
Same harness, vendor-operated
Higher is better
LoCoMo
LongMemEval S
PersonaMem 32k
LifeBench en
The Agent Memory Benchmark is run by Vectorize, who also make Hindsight. Same test for every row, but the scorekeeper is competing.
What we measured ourselves
Run on this machine against the same pinned datasets. The multi-hop runs are from 8 October and are not signed yet.
Multi-hop retrieval
Recall at five · higher is better
| Mnemosyne HotpotQA | 93.45% (interval 92.35% to 94.5%) |
|---|---|
| HippoRAG 2 HotpotQA | 96.3% |
| Mnemosyne 2Wiki | 91.7% (interval 90.6% to 92.67%) |
| HippoRAG 2 2Wiki | 90.4% |
| Mnemosyne MuSiQue | 66.72% (interval 64.85% to 68.6%) |
| HippoRAG 2 MuSiQue | 74.7% |
1,000 questions each, 8 October. Red is HippoRAG 2’s published result, which used a much larger model: context, not a head-to-head. Retrieval only, and not yet signed.
What fixing graph retrieval did
Recall at five, July versus October · higher is better
| HotpotQA before | 37.4% |
|---|---|
| HotpotQA after | 93.45% |
| 2Wiki before | 23.73% |
| 2Wiki after | 91.7% |
| MuSiQue before | 10.42% |
| MuSiQue after | 66.72% |
Same questions, same corpus, same pinned files. In July the graph channel fired on zero of 1,000 queries; now it works. We publish both so the jump is visible.
Our LongMemEval run
Did the right evidence come back? · higher is better
| nDCG@5 | 29.67% (interval 26.38% to 33.04%) |
|---|---|
| Recall@5 | 28.06% (interval 24.88% to 31.37%) |
500 questions, signed and reproducible. Whiskers show the range. This asks only whether the right evidence came back, not whether the answer was right.
How to read these
Finding evidence and answering well are different things, so they never share a chart.
Our results in full
Every metric we have measured, with its range and the evidence behind it.
| System | Benchmark | Metric | Result | Evidence |
|---|---|---|---|---|
| mnemosyne-localregistered-operator-retrieval | longmemeval-retrievallongmemeval-retrieval-v1 | recall_at_5retrieval | 0.2806 ratioInterval: 0.24880000000000002 to 0.31373333333333336 | View run 2026-10-04-longmemeval-retrieval-b0cdbd89operator-run; not publishableOperator: Mnemosyne projectRun disclosuresSystemmnemosyne-local Trackregistered-operator-retrieval Benchmarklongmemeval-retrieval longmemeval-retrieval-v1 Publicationoperator-run; not publishable OperatorMnemosyne project; disclosed: true Metrics
Immutable artifacts run_commit b0cdbd8986b94c91a3bf20833ed33f405612025abuild_fingerprint sha256:7ba4789e105be00461f40cdf5c3b84e29785b2234839976e3a20ef8d954b079cconfig_digest sha256:3cad9149f630c8c2d610ad87d9203e32713d071df6e96bd918c610291d3492a9bundle_digest sha256:c8449069cc96e7be7700283fec67d5ed53150c37d6ded0af634535c04e99f175trace_index_digest sha256:7146385bdc49de2c0353dd91689c08a4a60b74083de39f2bd5a7e802efabf20a |
| mnemosyne-localregistered-operator-retrieval | longmemeval-retrievallongmemeval-retrieval-v1 | ndcg_at_5retrieval | 0.2967188496001503 ratioInterval: 0.26378166565316336 to 0.33038741454372256 | View run 2026-10-04-longmemeval-retrieval-b0cdbd89operator-run; not publishableOperator: Mnemosyne projectRun disclosuresSystemmnemosyne-local Trackregistered-operator-retrieval Benchmarklongmemeval-retrieval longmemeval-retrieval-v1 Publicationoperator-run; not publishable OperatorMnemosyne project; disclosed: true Metrics
Immutable artifacts run_commit b0cdbd8986b94c91a3bf20833ed33f405612025abuild_fingerprint sha256:7ba4789e105be00461f40cdf5c3b84e29785b2234839976e3a20ef8d954b079cconfig_digest sha256:3cad9149f630c8c2d610ad87d9203e32713d071df6e96bd918c610291d3492a9bundle_digest sha256:c8449069cc96e7be7700283fec67d5ed53150c37d6ded0af634535c04e99f175trace_index_digest sha256:7146385bdc49de2c0353dd91689c08a4a60b74083de39f2bd5a7e802efabf20a |