LONGMEMEVAL-S · UPDATED 30 OCTOBER 2026
97.20% on
LongMemEval-S.
486 correct out of 500 in Mnemosyne’s latest LongMemEval-S evaluation. Explore all three October 28–29 runs, their category scores, and the limits of what they establish.
Three complete benchmark runs
All 500 questions are retained in each run, including incorrect answers. Configurations changed together, so this is not an isolated ablation.
| Configuration | Date | Correct | Overall | Task average | Abstention | Recorded cost |
|---|---|---|---|---|---|---|
| Dense baseline | 2026-10-28 | 465 / 500 | 93.00% | 93.66% | 28 / 30 | $2.6035 |
| Hybrid + strict abstention | 2026-10-29 | 469 / 500 | 93.80% | 94.85% | 30 / 30 | $3.1017 |
| Hybrid + revised synthesis | 2026-10-29 | 486 / 500 | 97.20% | 97.93% | 26 / 30 | $3.1106 |
Overall accuracy weights every question equally. Task average weights the six categories equally. The 30 abstention questions are already included in the 500, not an extra category. The v1 run achieved 30/30 abstentions; v2 achieved 26/30.
Latest run, by category
Category counts and correct answers sum to 500 and 486 respectively.
| Question category | Questions | Correct | Accuracy |
|---|---|---|---|
| knowledge update | 78 | 77 | 98.72% |
| multi session | 133 | 126 | 94.74% |
| single session assistant | 56 | 56 | 100.00% |
| single session preference | 30 | 30 | 100.00% |
| single session user | 70 | 68 | 97.14% |
| temporal reasoning | 133 | 129 | 96.99% |
Recorded partition breakdown
The saved files label 100 questions as dev and 400 as heldout. An intermediate 31-case diagnostic includes 7 dev and 24 heldout-labelled cases; partition names are reproduced as recorded.
| Partition label | Questions | Correct | Overall | Task average |
|---|---|---|---|---|
| dev (recorded label) | 100 | 98 | 98.00% | 97.87% |
| heldout (recorded label) | 400 | 388 | 97.00% | 97.82% |
Retrieval and completion
Retrieval describes evidence selection; it is separate from answer accuracy.
| Measure | Latest result | Denominator or definition |
|---|---|---|
| Mean session recall@10 | 99.70% | 500 rows with recorded recall, including abstention rows |
| Mean evidence-turn coverage@10 | 99.90% | 500 rows with recorded coverage |
| Full session recall | 496 / 500 | Four rows have partial recall: two at 0.75, two at 0.50 |
| Abstention accuracy | 86.67% (26 / 30) | Subset already included in overall accuracy |
| Truncated / empty outputs | 0 / 0 | Recorded finish reasons / nonempty hypotheses |
The accompanying source report described recall over 470 answerable questions. The actual summaries average all 500 rows, so this page uses that denominator. On the 470 non-abstention rows, mean session recall is 99.68%.
What ran
Configuration reported in the retained summary and per-question files.
| Setting | Recorded configuration |
|---|---|
| Dataset | LongMemEval-S cleaned; full 500-question file; S only |
| Reader | anthropic/claude-haiku-5.5 |
| Reasoning / answer cap | xhigh; 32,000 tokens (v1 and v2); baseline 4,000 tokens |
| Judge | openai/gpt-4o-mini-2024-07-18; 10-token verdict; LongMemEval task-specific prompts |
| Embedding | cloud:qwen/qwen3-embedding-8b |
| Evidence | Top 10 sessions; current runner combines dense and lexical retrieval |
| Cost scope | Sum of recorded reader + judge charges; excludes embedding preparation and wider experiment costs |
Methodology and provenance
Models, scoring, and retained run records.
- The current reader prompt includes benchmark-specific answer examples; the retained results do not establish a clean held-out evaluation.
- The 31-miss diagnostic includes 24 rows labelled heldout and 7 labelled dev. Treat the partition names as labels, not evidence of untouched test data.
- No immutable run-time code or prompt manifest was supplied with these OpenRouter runs. Current source hashes identify the review snapshot, not proven execution bytes.
- No independent rerun or matched competitor evaluation was performed.
- The Wilson interval describes the recorded verdict proportion, not tuning bias or model-judge error.
- Cost is the sum of retained per-question reader and judge charges; embedding preparation, retries outside the rows, infrastructure and development experiments are not included.
Mnemosyne scored 97.20% (486/500) on LongMemEval-S using the benchmark’s task-specific grading prompts. The retained summaries identify the reader, judge, retrieval mode and token limits. Custom retrieval and reader models are valid ways to evaluate a system; independent reproduction is a separate question. Earlier October 8–9 results remain on the claims page.
Inspect the evidence
Original summaries plus sanitized question-level verdicts that reproduce the displayed totals.
- All recomputed metrics and partition breakdowns
- 1,500 question-level verdicts (JSON)
- Original file hashes and import provenance
- Dense baseline — original summary
- Hybrid + strict abstention — original summary
- Hybrid + revised synthesis — original summary
Downloads omit benchmark questions, reference answers, generated answers and retrieved conversation text. Original JSONL files are preserved in the Mnemetric evidence repository. Hashes establish file identity; they do not independently prove model execution. This is not a complete replay bundle.