I ran a small pilot benchmark on evidence depth for LLM calls.

The main result is methodological:

Correctness-only calibration can be blind.

In a dual-instrumented sweep, the same model response was scored on two axes at the same evidence depth k:

1. factual correctness
2. epistemic contamination / framing leakage

For six of seven valid model runs, correctness stayed flat at 1.000 for every k ≥ 1.

A correctness-only benchmark would therefore say:

no saturation problem.

But a separate contamination axis still moved underneath it. The contamination signal was small, but non-zero, with maximum severity values around 0.05–0.08 for several models.

So the claim is not :

more context simply makes models wrong.

The sharper claim is:

more context can look harmless on correctness while already changing another reliability axis.

---

Definition

I use k* as the evidence-saturation point:

k* = argmax_k Reliability(model, task, evidence_k)

But Reliability has to specify the axis being optimized.

A k* calibrated only on factual correctness may not be the right k* for contamination resistance, state hygiene, routing, or governance.

Also, this is not the same as retriever top-k.

• Retriever top-k : how many chunks a retriever returns by similarity.
• Evidence-saturation k* : how many decision-relevant fragments should actually enter the model call under a measured reliability objective.
• State-density k : a separate compression/state-density diagnostic; not the same object.

---

Task variance

A first task-battery probe also suggests that k* should not be treated as a single fixed number per model.

Across five task types:

• factual recall
• multi-hop
• state tracking
• conflict resolution
• constraint following

the reliability-optimal k* moved across the ladder from small k to full context.

This is still a pilot result, not a universal law. But it argues against fixed defaults such as:

• always top-5
• always top-10
• always fill the context window

A safer assumption is:

measure k* per model, task type, context format, and reliability axis.

---

Who might care?

This is mainly relevant for people building deployed RAG systems, long-memory agents, model routers, or inference-time governance layers.

RAG systems

Many RAG pipelines use fixed retrieval defaults such as top-3, top-5, or top-10.

The retriever can still rank candidate chunks, but the number of fragments actually injected into the model should be calibrated empirically.

k* turns retrieval depth from a default into a measured deployment parameter.

Long-memory agents

Long-running assistants often accumulate context:

previous decisions, prior messages, user memories, state summaries, tool outputs

The risk is that persistence becomes context accumulation.

A calibrated k* can help decide how many prior state slices or memories should be reactivated for a turn.

Model routing

A small model may be reliable at low k, but fragile when overloaded with many fragments.

A larger model may justify its cost only when it remains stable at higher k.

So k* can become a routing feature:

task → model → calibrated evidence depth → verifier or no verifier

Cost and latency control

If additional context does not improve the reliability axis being optimized, it is just cost, latency, and attack surface.

A measured k* helps avoid paying for context that does not improve the system.

Governance and audit

For deployed systems, especially in enterprise settings, context injection should not be an informal prompt-engineering choice.

Instead of saying:

we gave the model a lot of relevant context

a system can say:

for this model, task type, context format, and reliability axis, we measured this evidence depth as optimal.

That is easier to audit, reproduce, and regression-test.

High-stakes RAG

For domains such as legal, medical, or finance, the immediate claim is not that these pilot k* values transfer.

They do not.

The point is methodological:

correctness-only calibration may miss drift, framing contamination, or ontology leakage exactly where auditability matters most.

---

Caveats

This is a pilot benchmark.

Current limitations:

• small synthetic datasets
• heuristic contamination metrics
• hosted-provider routing / backend identity remains a reproducibility issue
• task-battery results are preliminary
• reported k* values are not universal constants
• the method calibrates a deployment profile; it does not prove a universal law about all models or all RAG systems

---

Practical takeaway

Do not assume:

more context is better

Do not calibrate retrieval depth on correctness alone.

Measure:

• k* per model
• k* per task type
• k* per context format
• k* per reliability axis

For RAG, agents, memory systems, and routers, evidence depth should be treated as a calibrated inference-time control parameter.

I am preparing the full write-up / preprint. I would be interested whether others have observed similar evidence-saturation or axis-specific failure patterns in RAG, agent memory, or model routing systems.