Development evidence

Experiments, including the failures.

What ran, what failed, and what is still unknown. These are not admitted leaderboard results.

UPDATED 30 OCTOBER 2026 · MNEMOSYNE RESULTS

LongMemEval-S: latest recorded results

Three complete 500-question runs from the Mnemosyne repository. Totals rechecked against every recorded verdict.

Results & evidence →
Latest overall accuracy97.20%486 / 500 correct
Task-averaged accuracy97.93%Unweighted mean of six categories
Session recall@1099.70%Mean over 500 recorded rows
Abstention accuracy86.67%26 / 30 · four missed abstentions
Captures5Curated development runs
Failed attempts kept1Never removed
Evidence size6.9 MBRaw files, hash-checked
Admitted results0None of these is a score
None of this is a score.
Format validity, stored-state accuracy and firing behavior are different checks. A completed diagnostic is not a completed benchmark. ZIPs include raw records, methodology and file hashes; hashes bind files but do not independently attest execution.

Same workload, different transport

31 files · hash-bound

Completed paired development workload
MCP stdio, correct firings320 / 320
CLI, correct firings200 / 320

Persistent MCP stdio: 320/320 correct firings. CLI: 200/320, with 120 missed short windows. Both recovered every exact-time intention; no duplicate or cancelled firings. Programmed triggers only: this does not measure natural-language memory quality or competitor performance.

Clocked trigger pressure

18 files · hash-bound

Completed development workload
Correct firings207 / 320
Exact-time intentions recovered160 / 160

Five seeds, 320 live intentions: 207 fired correctly and 113 short-window triggers were missed. All 160 exact-time intentions recovered; no duplicate or cancelled firings. This measures the public CLI path, not intrinsic engine capacity.

Dependency formation

21 files · hash-bound

Failed full-corpus attempt
Cases completed1 / 100

1 of 100 cases completed. The next case failed on task-ID reuse; 98 were not attempted. The completed case missed one expected firing. No full-corpus score.

JSON versus schema

78 files · hash-bound

Completed development diagnostic
Plain JSON passed format checks0 / 11
Schema outputs passed format checks11 / 11

22 first-turn responses across 11 inputs. All 11 plain-JSON outputs failed format checks; all 11 schema outputs passed them. Semantic failures remain. No tasks were executed.

Prompt semantics

77 files · hash-bound

Completed development diagnostic
First-turn responses examined22 / 22

22 first-turn responses across 11 inputs. Added instructions corrected one recurrence proposal, but event and condition errors remained and an unrelated task was introduced. No overall improvement established.