{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreihfizgr3muqb32crfqoip3p6qwvifa5qxcxra2etur7hlonwnyog4",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mprv6grww7n2"
  },
  "path": "/t/evidence-saturation-k-retrieval-depth-should-be-calibrated-not-guessed/177363#post_2",
  "publishedAt": "2026-07-03T23:26:08.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "node postprocessors",
    "discussion",
    "env docs",
    "(click for more details)",
    "Sufficient Context",
    "Sufficient Context: A New Lens on Retrieval Augmented Generation Systems",
    "Lost in the Middle",
    "LongContextReorder",
    "RAGAS metrics",
    "Noise Sensitivity"
  ],
  "textContent": "I thought about how to make this easier to test:\n\n* * *\n\nShort version: I think the most important part is not “what is the best `k`?”, but “which `k` are we talking about, at which stage of the RAG pipeline, and for which reliability axis?”\n\nI have seen nearby patterns in adaptive retrieval depth, sufficient-context evaluation, irrelevant-context robustness, RAG diagnostic metrics, and long-context position effects. I would not treat those as the same claim, though. The interesting part of your framing, to me, is that `k*` is not just a retrieval hyperparameter. It is a calibration target for final evidence depth under a chosen reliability axis.\n\nThe smallest useful next step may be:\n\n  1. split `top_k` into several pipeline-level variables;\n  2. measure when the context first becomes sufficient;\n  3. only then measure correctness, groundedness, and contamination/framing leakage;\n  4. validate the actual injected fragments before interpreting the curve;\n  5. log where the decisive evidence lands in the prompt.\n\n\n\nThe rest of this comment is just a proposed testing map.\n\n* * *\n\n## 1. I would first disambiguate `k`\n\nI would avoid using one overloaded `top_k` term.\n\nName | Meaning\n---|---\n`candidate_k` | number of candidates returned by the first retriever\n`rerank_k` / `top_n` | number of candidates kept after reranking or filtering\n`injected_k` | number of evidence fragments actually inserted into the LLM prompt\n`effective_k` | number of distinct/useful evidence units after deduplication, compression, summarization, or prompt packing\n\nI would interpret your `k*` mostly as an `injected_k` / `effective_k` question, not as the retriever’s initial `top_k`.\n\nThis matters because real RAG stacks often separate retrieval, reranking, filtering, compression, and final prompt construction. For example, LlamaIndex has retrieval-time node postprocessors and rerankers where `top_n` can mean the number of nodes returned after reranking, not the number of original candidates. Open WebUI also has practical discussion around `RAG_TOP_K` and `RAG_TOP_K_RERANKER`, which is a useful reminder that “top k” can mean different things in different layers: discussion, env docs.\n\nSo before interpreting any `k*` curve, I would log the actual fragment IDs and fragment types that reach the model.\n\nMore detail on the k split (click for more details)\n\n* * *\n\n## 2. I would separate the sufficiency threshold from the contamination threshold\n\nI would not look for one universal `k*` at first.\n\nI would separate at least three regions:\n\nRegion | Question\n---|---\nevidence-insufficient | are required facts still missing?\nevidence-sufficient | do the retrieved snippets contain enough information to answer?\nover-context / contamination region | does additional context start shifting wording, assumptions, framing, source emphasis, or state?\n\nThat distinction seems important because a model can be correct over a range of `k` values while still changing some other reliability axis.\n\nThis connects well with the “sufficient context” line of work. The Sufficient Context project tests whether retrieved snippets alone could plausibly answer the question, and the paper Sufficient Context: A New Lens on Retrieval Augmented Generation Systems uses that idea to separate “retrieval did not provide enough information” from “the model had enough information but failed to use it.”\n\nFor this thread, I would phrase it as:\n\n> first measure the smallest `injected_k` where the context becomes sufficient; then measure whether correctness, groundedness, or contamination/framing leakage changes after that point.\n\nA minimal sufficiency-first test (click for more details)\n\n* * *\n\n## 3. I would validate the actual injected condition before reading the curve\n\nOne implementation trap: the condition label is not enough.\n\nA row labeled `gold + distractor` should be checked against the actual injected fragment IDs and fragment kinds. Otherwise the contamination curve can accidentally measure data assembly mistakes.\n\nI would add a validation step like this before any judge or generator result is interpreted:\n\nIntended condition | Required validation\n---|---\n`gold_only` | only required evidence is injected\n`gold_plus_random` | gold evidence plus at least one random irrelevant fragment\n`gold_plus_semantic` | gold evidence plus a semantically related non-answer fragment\n`gold_plus_duplicate` | gold evidence plus a duplicate or near-duplicate\n`gold_plus_stale` | gold evidence plus temporally stale evidence\n`gold_plus_conflict` | gold evidence plus a genuinely conflicting fragment\n`gold_plus_frame_shift` | gold evidence plus a fragment that changes framing but not necessarily the answer\n`gold_plus_adversarial` | gold evidence plus instruction-like or adversarial text\n\nIf this validation fails, I would not interpret the contamination score yet.\n\nWhy I think condition validation matters (click for more details)\n\n* * *\n\n## 4. I would log prompt position, not only evidence inclusion\n\nWhen `k` changes, the decisive evidence may move inside the prompt.\n\nSo a `k` effect can be mixed with a prompt-position effect.\n\nFor every run, I would log:\n\n\n    decisive_evidence_rank_in_retrieval\n    decisive_evidence_rank_after_rerank\n    decisive_evidence_position_in_prompt\n    decisive_evidence_token_start\n    decisive_evidence_token_end\n\n\nThis is not only theoretical. Lost in the Middle showed that models can be sensitive to where relevant information appears in long context. LlamaIndex also has a LongContextReorder postprocessor, which is a practical sign that node order can matter when a large top-k is placed into context.\n\nSo if contamination changes at larger `k`, I would check whether the decisive evidence was also pushed into a worse prompt position.\n\nHow this changes the interpretation of k (click for more details)\n\n* * *\n\n## 5. I would keep contamination/framing leakage separate from existing RAG metrics\n\nI think your contamination/framing axis is adjacent to existing RAG metrics, but probably not identical to any one of them.\n\nExisting tools already give useful vocabulary. For example, RAGAS metrics include faithfulness, answer relevancy, context recall, context precision, and context utilization. RAGAS also has Noise Sensitivity, which measures how often a system makes errors when using relevant or irrelevant retrieved documents.\n\nThat is related, but I would not collapse framing leakage into noise sensitivity.\n\nA useful separation might be:\n\nAxis | What it measures\n---|---\ncorrectness | final answer is factually right\nsufficiency | context contains enough evidence to answer\ncontext recall | necessary evidence was retrieved\ncontext precision | irrelevant evidence is not dominating the context\ncontext utilization | answer-relevant context is ranked/used well\nfaithfulness / groundedness | answer is supported by supplied context\nnoise sensitivity | retrieved docs cause incorrect responses\ncontamination / framing leakage | extra context shifts framing, assumptions, vocabulary, source emphasis, or state even if correctness is flat\n\nThe last row is the hard part.\n\nI would treat it as an additional rubric, not as a replacement for correctness or groundedness.\n\nAdjacent prior work I would use as controls, not as exact equivalents (click for more details)\n\n* * *\n\n## 6. I would unit-test the contamination judge/rubric\n\nIf the contamination score is judged by an LLM or by a heuristic rubric, I would test that evaluator before trusting the `k` curve.\n\nOtherwise, the curve may partly measure:\n\n  * judge sensitivity to longer prompts;\n  * judge sensitivity to distractor wording;\n  * formatting artifacts;\n  * condition-construction mistakes;\n  * prompt-position effects;\n  * actual model behavior.\n\n\n\nA small rubric test suite could include:\n\nCase | Expected behavior\n---|---\ngold only | no contamination\ngold + neutral filler | no contamination\ngold + random irrelevant | usually no framing leakage unless used\ngold + semantic distractor | possible distractor sensitivity\ngold + duplicate | should not be called framing leakage by default\ngold + stale evidence | stale-memory / temporal-validity issue\ngold + conflict | conflict-handling issue\ngold + frame-shifting evidence | likely framing leakage candidate\ngold + adversarial instruction-like text | adversarial stress test, separate from ordinary leakage\n\nSuggested judge/rubric output schema (click for more details)\n\n* * *\n\n## 7. Minimal protocol I would suggest\n\nOutside the details, my proposed minimal protocol would be:\n\n  1. Fix corpus, chunking, retriever, reranker, prompt format, model, and judge.\n  2. Sweep only final `injected_k`.\n  3. Log actual injected fragment IDs/kinds and prompt positions.\n  4. First measure sufficiency.\n  5. Then measure correctness, groundedness, and contamination/framing leakage.\n  6. Add unit tests for the contamination rubric.\n  7. Compare fixed-k, adaptive-k, reranked-k, compressed-k, and oracle-evidence conditions.\n\nDetailed version of the protocol (click for more details)\n\n* * *\n\n## 8. Practical next actions\n\nIf you wanted to make this easier for others to reproduce, I would add these in roughly this order:\n\nEffort | Addition\n---|---\nlow | define whether `k` means `candidate_k`, `rerank_k`, `injected_k`, or `effective_k`\nlow | log actual injected fragment IDs and fragment kinds\nlow | log decisive evidence position in the prompt\nmedium | add retrieval-only sufficiency sweep\nmedium | add condition validation before generation/judging\nmedium | add unit tests for the contamination/framing rubric\nmedium | compare fixed-k with one adaptive-k baseline\nhigher | add HotpotQA-style public multi-hop preview\nhigher | compare raw chunks vs reranked vs compressed/filtered evidence\n\nMy main takeaway is:\n\n> The valuable target is probably not a universal `k*`, but a reproducible calibration procedure: for a given corpus, chunking scheme, retriever, reranker, prompt builder, model, and reliability axis, what final evidence depth is sufficient, and when do extra fragments start changing another axis?\n\nThat would make the idea easier to test, compare, and falsify without forcing every reader to guess which part of the RAG pipeline the reported `k*` belongs to.",
  "title": "Evidence saturation k*: retrieval depth should be calibrated, not guessed"
}