FAISS + LMDB RAG on a 50-year corpus works great — until you ask ‘what happened in 2020?’ (time-aware retrieval problem)

John6666: > FAISS Thank you so much for this detailed answer — it genuinely helped unblock my thinking. Your explanation of filtered ANN retrieval with FAISS selectors, along with your perspective on the issue, gave me a much clearer systems-level understanding of handling temporal queries in production—without immediately resorting to year-wise sharding. I also appreciate you pointing me toward papers about diachronic RAG, temporal IR / QA research —I’m not very familiar with these, and it opened up several directions I hadn’t considered. Based on your guidance, this is the flow I’m planning to implement: 1. Keep a single main IVFPQ index (no sharding for now). 2. Build a fast time-window → vector-ID mapping from chunk timestamps. 3. For explicit temporal queries, run FAISS subset search using ID selectors (time-filtered branch). 4. In parallel, run a smaller global unfiltered branch as recall protection. 5. Merge/union both candidate sets and deduplicate. 6. Continue with exact memmap rerank + cross-encoder rerank on the merged set. 7. Add focused evaluation for temporal quality (not just overall recall), including in-window recall/nDCG and query-class latency. Really grateful again — your response was super actionable and timely.