{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiel65owx7j5mux4kp4anoumy3ilqsg67sqc463fjwzdrvcwkkh6ke",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mpojqtslafq2"
  },
  "path": "/t/evidence-saturation-k-retrieval-depth-should-be-calibrated-not-guessed/177363#post_1",
  "publishedAt": "2026-07-02T16:10:49.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "#\n\nI ran a small pilot benchmark on evidence depth for LLM calls.\n\nThe main result is methodological:\n\n> **Correctness-only calibration can be blind.**\n\nIn a dual-instrumented sweep, the same model response was scored on two axes at the same evidence depth `k`:\n\n  1. factual correctness\n  2. epistemic contamination / framing leakage\n\n\n\nFor six of seven valid model runs, correctness stayed flat at `1.000` for every `k ≥ 1`.\n\nA correctness-only benchmark would therefore say:\n\n> no saturation problem.\n\nBut a separate contamination axis still moved underneath it. The contamination signal was small, but non-zero, with maximum severity values around `0.05–0.08` for several models.\n\nSo the claim is **not** :\n\n> more context simply makes models wrong.\n\nThe sharper claim is:\n\n> more context can look harmless on correctness while already changing another reliability axis.\n\n* * *\n\n## Definition\n\nI use `k*` as the evidence-saturation point:\n\n`k* = argmax_k Reliability(model, task, evidence_k)`\n\nBut `Reliability` has to specify the axis being optimized.\n\nA `k*` calibrated only on factual correctness may not be the right `k*` for contamination resistance, state hygiene, routing, or governance.\n\nAlso, this is not the same as retriever `top-k`.\n\n  * **Retriever top-k** : how many chunks a retriever returns by similarity.\n  * **Evidence-saturation k*** : how many decision-relevant fragments should actually enter the model call under a measured reliability objective.\n  * **State-density k** : a separate compression/state-density diagnostic; not the same object.\n\n\n\n* * *\n\n## Task variance\n\nA first task-battery probe also suggests that `k*` should not be treated as a single fixed number per model.\n\nAcross five task types:\n\n  * factual recall\n  * multi-hop\n  * state tracking\n  * conflict resolution\n  * constraint following\n\n\n\nthe reliability-optimal `k*` moved across the ladder from small `k` to full context.\n\nThis is still a pilot result, not a universal law. But it argues against fixed defaults such as:\n\n  * always top-5\n  * always top-10\n  * always fill the context window\n\n\n\nA safer assumption is:\n\n> measure `k*` per model, task type, context format, and reliability axis.\n\n* * *\n\n## Who might care?\n\nThis is mainly relevant for people building deployed RAG systems, long-memory agents, model routers, or inference-time governance layers.\n\n### RAG systems\n\nMany RAG pipelines use fixed retrieval defaults such as `top-3`, `top-5`, or `top-10`.\n\nThe retriever can still rank candidate chunks, but the number of fragments actually injected into the model should be calibrated empirically.\n\n`k*` turns retrieval depth from a default into a measured deployment parameter.\n\n### Long-memory agents\n\nLong-running assistants often accumulate context:\n\n> previous decisions, prior messages, user memories, state summaries, tool outputs\n\nThe risk is that persistence becomes context accumulation.\n\nA calibrated `k*` can help decide how many prior state slices or memories should be reactivated for a turn.\n\n### Model routing\n\nA small model may be reliable at low `k`, but fragile when overloaded with many fragments.\n\nA larger model may justify its cost only when it remains stable at higher `k`.\n\nSo `k*` can become a routing feature:\n\n`task → model → calibrated evidence depth → verifier or no verifier`\n\n### Cost and latency control\n\nIf additional context does not improve the reliability axis being optimized, it is just cost, latency, and attack surface.\n\nA measured `k*` helps avoid paying for context that does not improve the system.\n\n### Governance and audit\n\nFor deployed systems, especially in enterprise settings, context injection should not be an informal prompt-engineering choice.\n\nInstead of saying:\n\n> we gave the model a lot of relevant context\n\na system can say:\n\n> for this model, task type, context format, and reliability axis, we measured this evidence depth as optimal.\n\nThat is easier to audit, reproduce, and regression-test.\n\n### High-stakes RAG\n\nFor domains such as legal, medical, or finance, the immediate claim is not that these pilot `k*` values transfer.\n\nThey do not.\n\nThe point is methodological:\n\n> correctness-only calibration may miss drift, framing contamination, or ontology leakage exactly where auditability matters most.\n\n* * *\n\n## Caveats\n\nThis is a pilot benchmark.\n\nCurrent limitations:\n\n  * small synthetic datasets\n  * heuristic contamination metrics\n  * hosted-provider routing / backend identity remains a reproducibility issue\n  * task-battery results are preliminary\n  * reported `k*` values are not universal constants\n  * the method calibrates a deployment profile; it does not prove a universal law about all models or all RAG systems\n\n\n\n* * *\n\n## Practical takeaway\n\nDo not assume:\n\n> more context is better\n\nDo not calibrate retrieval depth on correctness alone.\n\nMeasure:\n\n  * `k*` per model\n  * `k*` per task type\n  * `k*` per context format\n  * `k*` per reliability axis\n\n\n\nFor RAG, agents, memory systems, and routers, evidence depth should be treated as a calibrated inference-time control parameter.\n\nI am preparing the full write-up / preprint. I would be interested whether others have observed similar evidence-saturation or axis-specific failure patterns in RAG, agent memory, or model routing systems.",
  "title": "Evidence saturation k*: retrieval depth should be calibrated, not guessed"
}