LONGMEMEVAL-S · UPDATED 30 OCTOBER 2026

97.20% on
LongMemEval-S.

486 correct out of 500 in Mnemosyne’s latest LongMemEval-S evaluation. Explore all three October 28–29 runs, their category scores, and the limits of what they establish.

Overall accuracy97.20%486 / 500 · 14 incorrect
Task-averaged accuracy97.93%Six equally weighted categories
95% Wilson interval95.36–98.32%95% interval · 500 questions
Recorded inference cost$3.1106Reader + judge rows only

Three complete benchmark runs

All 500 questions are retained in each run, including incorrect answers. Configurations changed together, so this is not an isolated ablation.

  • Dense baseline93.00%93.00%
  • Hybrid + strict abstention93.80%93.80%
  • Hybrid + revised synthesis97.20%97.20%
Three full LongMemEval-S runs
ConfigurationDateCorrectOverallTask averageAbstentionRecorded cost
Dense baseline2026-10-28465 / 50093.00%93.66%28 / 30$2.6035
Hybrid + strict abstention2026-10-29469 / 50093.80%94.85%30 / 30$3.1017
Hybrid + revised synthesis2026-10-29486 / 50097.20%97.93%26 / 30$3.1106

Overall accuracy weights every question equally. Task average weights the six categories equally. The 30 abstention questions are already included in the 500, not an extra category. The v1 run achieved 30/30 abstentions; v2 achieved 26/30.

Latest run, by category

Category counts and correct answers sum to 500 and 486 respectively.

Latest result by LongMemEval question category
Question categoryQuestionsCorrectAccuracy
knowledge update787798.72%
multi session13312694.74%
single session assistant5656100.00%
single session preference3030100.00%
single session user706897.14%
temporal reasoning13312996.99%

Recorded partition breakdown

The saved files label 100 questions as dev and 400 as heldout. An intermediate 31-case diagnostic includes 7 dev and 24 heldout-labelled cases; partition names are reproduced as recorded.

Recorded LongMemEval-S partitions
Partition labelQuestionsCorrectOverallTask average
dev (recorded label)1009898.00%97.87%
heldout (recorded label)40038897.00%97.82%

Retrieval and completion

Retrieval describes evidence selection; it is separate from answer accuracy.

Retrieval and answer completion metrics
MeasureLatest resultDenominator or definition
Mean session recall@1099.70%500 rows with recorded recall, including abstention rows
Mean evidence-turn coverage@1099.90%500 rows with recorded coverage
Full session recall496 / 500Four rows have partial recall: two at 0.75, two at 0.50
Abstention accuracy86.67% (26 / 30)Subset already included in overall accuracy
Truncated / empty outputs0 / 0Recorded finish reasons / nonempty hypotheses

The accompanying source report described recall over 470 answerable questions. The actual summaries average all 500 rows, so this page uses that denominator. On the 470 non-abstention rows, mean session recall is 99.68%.

What ran

Configuration reported in the retained summary and per-question files.

LongMemEval recorded configuration
SettingRecorded configuration
DatasetLongMemEval-S cleaned; full 500-question file; S only
Readeranthropic/claude-haiku-5.5
Reasoning / answer capxhigh; 32,000 tokens (v1 and v2); baseline 4,000 tokens
Judgeopenai/gpt-4o-mini-2024-07-18; 10-token verdict; LongMemEval task-specific prompts
Embeddingcloud:qwen/qwen3-embedding-8b
EvidenceTop 10 sessions; current runner combines dense and lexical retrieval
Cost scopeSum of recorded reader + judge charges; excludes embedding preparation and wider experiment costs

Methodology and provenance

Models, scoring, and retained run records.

  • The current reader prompt includes benchmark-specific answer examples; the retained results do not establish a clean held-out evaluation.
  • The 31-miss diagnostic includes 24 rows labelled heldout and 7 labelled dev. Treat the partition names as labels, not evidence of untouched test data.
  • No immutable run-time code or prompt manifest was supplied with these OpenRouter runs. Current source hashes identify the review snapshot, not proven execution bytes.
  • No independent rerun or matched competitor evaluation was performed.
  • The Wilson interval describes the recorded verdict proportion, not tuning bias or model-judge error.
  • Cost is the sum of retained per-question reader and judge charges; embedding preparation, retries outside the rows, infrastructure and development experiments are not included.

Mnemosyne scored 97.20% (486/500) on LongMemEval-S using the benchmark’s task-specific grading prompts. The retained summaries identify the reader, judge, retrieval mode and token limits. Custom retrieval and reader models are valid ways to evaluate a system; independent reproduction is a separate question. Earlier October 8–9 results remain on the claims page.

Download methodology and provenance.

Inspect the evidence

Original summaries plus sanitized question-level verdicts that reproduce the displayed totals.

Downloads omit benchmark questions, reference answers, generated answers and retrieved conversation text. Original JSONL files are preserved in the Mnemetric evidence repository. Hashes establish file identity; they do not independently prove model execution. This is not a complete replay bundle.