{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreicnhweon7fbgqbuiwggwarmj725em7dxydwzpfsvqs3uvqsjxdw2m",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mm7ilb6wf622"
  },
  "path": "/t/preprint-on-llm-context-compaction/176090#post_1",
  "publishedAt": "2026-05-19T08:18:31.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "lost in compaction",
    "GitHub - profff/lost-in-compaction · GitHub"
  ],
  "textContent": "Hi HuggingFace community!\n\nSharing a preprint that some of you might find interesting,\non what LLMs forget when they compact their conversation history.\n\nPaper: “Lost in Compaction: Measuring Information Loss in LLM Context Summaries”\nDOI: lost in compaction\nCode, data, human-judge calibration: GitHub - profff/lost-in-compaction · GitHub\n\nThree findings that surprised me:\n\n  1. In the compacted zone of a context, LLM recall drops to 0-7% even though\nkeyword search still finds 82-93% of the facts. Information is present\nin the context but ignored by attention.\n\n  2. Compaction damages even untouched parts of the context: remaining-zone\nrecall drops by ~20pp as compaction increases. Adding more “preserved”\nsummaries dilutes attention rather than helping.\n\n  3. The compaction phase itself is non-deterministic at temperature zero:\nrecall measurements on identical conversations span up to a factor 14×\nacross replicates. Single-shot benchmarks of compaction strategies are\nunreliable, replicates are mandatory.\n\n\n\n\nMethodology in short:\n\n  * 234 LongMemEval facts naturally embedded in 190K-token contexts\n  * Single-pass compaction sweep (5-98%) on Claude Haiku 4.5 and Sonnet 4.6\n  * Multi-pass strategy comparison (Brutal, Incremental, Frozen, FrozenRanked)\non a 5M-token conversation with 4-6 replicates per cell\n\n\n\nIndependent, self-funded research (out of pocket, no institutional\naffiliation, no doctorate). Happy to answer technical questions about the\nmethodology, the strategies, or the follow-up directions I’m considering\n(verbatim store + on-demand expansion, structured frozen graphs).\n\n* * *\n\nP.S. — While I’m here: if anyone has 3+ recent cs.CL papers on arXiv and\nwould consider endorsing my submission, I’d be very grateful. HAL France\nrejected the deposit on credential grounds, so arXiv via personal\nendorsement is the route I’m exploring. I’d send the endorsement URL by DM\nafter we connect, per arXiv’s one-to-one sharing policy.\n\nThanks for reading,\nOlivier",
  "title": "Preprint on LLM context compaction"
}