LoCoMo Benchmark

    本地归档的 L3 评测分数（offline / api / live），来自 ci/eval/locomo。本页只展示指标——对话子集仍在 ci/eval/locomo/data/（CC BY-NC 4.0）。

    完整归因与五项指标结论见仓库 评测报告。

    Latest release 0.2.2 (v0.2.2) · 2026-10-01T00:00:00Z

    Offline passAPI failLive pass

    Fixture: ci/eval/locomo/data/locomo_subset.json — CC BY-NC 4.0 subset

    Summary

    VersionOfflineAPI hybrid R@5vs BM25Live precisionConflict
    0.2.2pass0.383 fail-12.0%0.880 pass0.950

    Latest breakdown (0.2.2)

    offline · pass

    MetricValue
    Hybrid Recall@50.335
    Hybrid MRR@50.199
    BM25 Recall@50.435
    BM25 MRR@50.310
    vs BM25-23.0%
    TARGET_15PCTn_a
    Near-dup supersede0.950 (19/20)
    Cross-user leaks0
    Persistence reopen1
    Extraction precisionskipped

    Gates: isolation=pass · persistence=pass · conflict=pass · hybrid_ge_bm25=n_a · target_15pct=n_a · extraction_precision=skipped · overall=pass

    api · fail model=qwen3.7-text-embedding-flash dim=1024

    MetricValue
    Hybrid Recall@50.383
    Hybrid MRR@50.225
    BM25 Recall@50.435
    BM25 MRR@50.310
    vs BM25-12.0%
    TARGET_15PCTmissed
    Near-dup supersede0.950 (19/20)
    Cross-user leaks0
    Persistence reopen1
    Extraction precisionskipped

    Gates: isolation=pass · persistence=pass · conflict=pass · hybrid_ge_bm25=fail · target_15pct=fail · extraction_precision=skipped · overall=fail

    live · pass extractor=deepseek-flash

    MetricValue
    Hybrid Recall@50.335
    Hybrid MRR@50.199
    BM25 Recall@50.435
    BM25 MRR@50.310
    vs BM25-23.0%
    TARGET_15PCTn_a
    Near-dup supersede0.950 (19/20)
    Cross-user leaks0
    Persistence reopen1
    Extraction precision0.880 (66/75)

    Gates: isolation=pass · persistence=pass · conflict=pass · hybrid_ge_bm25=n_a · target_15pct=n_a · extraction_precision=pass · overall=pass

    Diff vs previous

    Only one archived version — no previous release to compare.

    分数如何产生

    1. Push CI 保留离线硬门禁（moon run ci/eval/locomo）。
    2. 发版流水线在 Publish mooncakes / tag / GitHub Release 后结束（不再跑 LoCoMo archive）。
    3. 发版后在本地跑三档并提交 benchmarks/locomo/results/<version>.json（见下方）。
    4. 本页随 main 的 Pages 部署重建。

    本地命令

    # 离线档（进 push CI）
moon run ci/eval/locomo --target native

# api / live 单档
moon run ci/eval/locomo --target native -- --embedder api   # MOOMEM_EMBED_* / QWEN_*
moon run ci/eval/locomo --target native -- --live           # DEEPSEEK_*，全量 100 条

# 发版后本地归档三档并 commit + push（NEW_VERSION = 已发布版本）
set -a && source .env && set +a
NEW_VERSION=0.4.0 bash scripts/ci/run-locomo-benchmark-archive.sh

    api 档环境映射：QWEN_* → MOOMEM_EMBED_*。live 档只读 DEEPSEEK_*。结果 JSON 永不写入 API Key。