{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreig2gtijhimd66ivvdgcghryqlfjq57ie7unquk22ntgtxqlu26tpq",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mhv7uo5h26d2"
  },
  "path": "/t/faiss-lmdb-rag-on-a-50-year-corpus-works-great-until-you-ask-what-happened-in-2020-time-aware-retrieval-problem/174583#post_3",
  "publishedAt": "2026-03-25T07:27:06.000Z",
  "site": "https://discuss.huggingface.co",
  "textContent": "John6666:\n\n> FAISS\n\nThank you so much for this detailed answer — it genuinely helped unblock my thinking.\n\nYour explanation of filtered ANN retrieval with FAISS selectors, along with your perspective on the issue, gave me a much clearer systems-level understanding of handling temporal queries in production—without immediately resorting to year-wise sharding. I also appreciate you pointing me toward papers about diachronic RAG, temporal IR / QA research —I’m not very familiar with these, and it opened up several directions I hadn’t considered.\n\nBased on your guidance, this is the flow I’m planning to implement:\n\n  1. Keep a single main IVFPQ index (no sharding for now).\n\n  2. Build a fast time-window → vector-ID mapping from chunk timestamps.\n\n  3. For explicit temporal queries, run FAISS subset search using ID selectors (time-filtered branch).\n\n  4. In parallel, run a smaller global unfiltered branch as recall protection.\n\n  5. Merge/union both candidate sets and deduplicate.\n\n  6. Continue with exact memmap rerank + cross-encoder rerank on the merged set.\n\n  7. Add focused evaluation for temporal quality (not just overall recall), including in-window recall/nDCG and query-class latency.\n\n\n\n\nReally grateful again  — your response was super actionable and timely.",
  "title": "FAISS + LMDB RAG on a 50-year corpus works great — until you ask ‘what happened in 2020?’ (time-aware retrieval problem)"
}