Benchmark catalog

Beyond recall.
The whole memory.

The benchmarks we track, and how far each one is wired up.

Benchmark families14Upstream and our own
With development components6Adapters or probes exist
Memory capabilities24Mapped in the coverage map
Whole-memory modules20M01 to M20

Implementation stage

Stage labels describe implementation only, never performance.

14families
Development6
Planned8

Three tracks, separate conclusions

Results from different tracks are never mixed.

Official upstreamOriginal protocol, inputs and scoring, unchanged
Enhanced successorSeparately versioned, differences disclosed
DevelopmentLocal fixtures, never substituted for official results

Coverage is not compatibility

Why one score never describes a whole system.

These benchmarks can test Mnemosyne. They just do not cover everything it does, and what is missing is our adapter, not the benchmark.

What these tests see and miss

Find a benchmark

These labels say how far the code is built. An entry here is not a result and not a score.

Download catalog

14 benchmark families

B01Development components

LongMemEval-S

Retrieval and separate QA adapters exist; the current retrieval registration is a single-system characterization, not an official comparison.

Reporting rules

Official protocol and enhanced successor results must remain separate.

October 28–29 results: 97.20% latest self-reported accuracy. Full 500-question runs with benchmark-specific prompt tuning; benchmark-specific tuning disclosed.

B02Planned family

LongMemEval-V2

Planned family; complete official protocol, licensed input pins, adapter and measured evidence remain required.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B03Development components

HippoRAG: MuSiQue / 2Wiki / HotpotQA

Retrieval and reader adapter source exists; positive graph-effect and comparable reader evidence remain open.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B04Development components

MemoryAgentBench

Development adapter source exists; official execution and upstream contribution remain open.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B05Development components

BEAM-1M / BEAM-10M

Development adapter source exists; official scale runs and resource acceptance remain open.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B06Development components

LoCoMo

Ingestion, pinned scoring/tokenizer, native public capture/answer and offline replay components exist. Complete runner integration, dataset license admission, runtime verification and real measured runs remain open.

Reporting rules

Not for leading headline claims under the current publication policy.

B07Planned family

Memora / FAMA

Planned family; complete official protocol, licensed input pins, adapter and measured evidence remain required.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B08Planned family

MemoryArena

Planned family; complete official protocol, licensed input pins, adapter and measured evidence remain required.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B09Planned family

AFTER

Planned family; complete official protocol, licensed input pins, adapter and measured evidence remain required.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B10Planned family

STATE-Bench

Planned family; complete official protocol, licensed input pins, adapter and measured evidence remain required.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B11Planned family

GroupMemBench / GateMem

Planned family; complete official protocol, licensed input pins, adapter and measured evidence remain required.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B12Development components

PM-Bench / TriggerBench

Repository-specific action probes exist; they are not official upstream results.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B13Planned family

EvoMemBench

Planned family; complete official protocol, licensed input pins, adapter and measured evidence remain required.

Reporting rules

Official protocol and enhanced successor results must remain separate.

B14Planned family

EMemBench

Planned family; complete official protocol, licensed input pins, adapter and measured evidence remain required.

Reporting rules

Official protocol and enhanced successor results must remain separate.