{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiexiyj2tinv46a7yizpvbpfkrm5vxy2obgv6y6gcorrhmggqwyeue",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpvd2nhavi62"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreigzm2dk5ayfzo2muwlneihfcoufmj3rmugeum3dsewsxkccd33xoi"
    },
    "mimeType": "image/webp",
    "size": 66200
  },
  "path": "/__c1b9e06dc90a7e0a676b/building-a-document-qa-bot-why-embeddings-are-trickier-than-they-look-3jl8",
  "publishedAt": "2026-07-05T10:00:32.000Z",
  "site": "https://dev.to",
  "tags": [
    "ai",
    "python",
    "tutorial",
    "webdev",
    "InterWest Info AI"
  ],
  "textContent": "I spent a weekend building a Q&A bot for my team's internal docs. It sounded easy: dump PDFs into a vector database, query with embeddings, get answers. Three days later, I had a working prototype — and a healthy respect for all the hidden traps.\n\n##  The Problem\n\nOur team had 200+ pages of configuration guides scattered across Confluence, Google Docs, and a few dusty PDFs. Every week someone asked \"How do we set up the OAuth flow again?\" or \"What's the default timeout?\" I figured a semantic search bot could answer these instantly.\n\nI started simple. Use OpenAI embeddings, store them in Pinecone, then use GPT-4 to generate answers from retrieved chunks. Classic RAG (Retrieval-Augmented Generation).\n\n##  What I Tried That Didn't Work\n\n**First attempt: naive chunking.** I split every document into 500-character chunks with 50-character overlap. Straight into Pinecone. The first query returned garbage — chunks that mentioned \"OAuth\" but were actually about something else, or chunks too short to contain the answer.\n\n**Second attempt: bigger chunks with no overlap.** 2000 characters, no overlap. Queries matched better, but answers from GPT were often incomplete because the relevant sentence was split across two chunks.\n\n**Third attempt: using only the first 3 chunks.** I tried retrieving the top 3 chunks and concatenating them. Sometimes that worked, but often the best chunk was rank 4 or 5. And concatenating introduced noise that confused the model.\n\n##  What Eventually Worked\n\nI landed on a hybrid approach that balances precision and context length:\n\n  1. **Chunk by paragraphs** instead of fixed character counts. Preserves natural boundaries.\n  2. **Embed with a dense retriever** (text-embedding-ada-002) but also add a simple keyword index for exact matches.\n  3. **Retrieve 10 chunks** , then rerank using a lightweight cross-encoder to pick the 3 most relevant.\n  4. **Feed those 3 chunks as context** to the generative model with a strict instruction: _answer only from the context, say \"I don't know\" if irrelevant._\n\n\n\nHere's the core pipeline in Python — I'm using `sentence-transformers` for the cross-encoder and OpenAI for embeddings + generation, but the technique is service-agnostic:\n\n\n\n    import openai\n    from sentence_transformers import CrossEncoder\n    import numpy as np\n\n    # Step 1: chunk your documents into paragraphs\n    # (Assume we have a list of strings called paragraphs)\n\n    # Step 2: embed all paragraphs using OpenAI\n    response = openai.Embedding.create(\n        input=paragraphs,\n        model=\"text-embedding-ada-002\"\n    )\n    embeddings = np.array([d[\"embedding\"] for d in response[\"data\"]])\n\n    # Step 3: search function with reranking\n    reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')\n\n    def answer_query(query, top_k=10, rerank_top=3):\n        # Embed the query\n        q_emb = openai.Embedding.create(input=[query], model=\"text-embedding-ada-002\")[\"data\"][0][\"embedding\"]\n        # Cosine similarity (simplified)\n        scores = np.dot(embeddings, q_emb) / (np.linalg.norm(embeddings, axis=1) * np.linalg.norm(q_emb))\n        top_indices = np.argsort(scores)[-top_k:][::-1]\n        candidates = [paragraphs[i] for i in top_indices]\n        # Rerank\n        rerank_scores = reranker.predict([(query, c) for c in candidates])\n        best_idx = np.argsort(rerank_scores)[-rerank_top:][::-1]\n        context = \"\\n\\n---\\n\\n\".join([candidates[i] for i in best_idx])\n        # Generate answer\n        response = openai.ChatCompletion.create(\n            model=\"gpt-4\",\n            messages=[\n                {\"role\": \"system\", \"content\": \"Answer based on context. Say 'I don\\'t know' if not found.\"},\n                {\"role\": \"user\", \"content\": f\"Context:\\n{context}\\n\\nQuestion: {query}\"}\n            ]\n        )\n        return response[\"choices\"][0][\"message\"][\"content\"]\n\n\n##  Lessons Learned & Trade-offs\n\n  * **Chunking strategy matters more than I expected.** Paragraph-level chunks work best for narrative docs, but code snippets or tables need different handling. I ended up splitting tables into individual rows.\n  * **Reranking adds ~100ms latency** but cuts hallucination rate by half. Worth it.\n  * **Cost adds up.** Embedding 200 pages cost ~$0.20, but every query uses both embedding + generation. For high traffic, either cache common queries or use a cheaper embedding model.\n  * **The cross-encoder model is small** — I run it locally, no API calls needed. That saved money but increased memory usage (~300MB).\n  * **Exact keyword matching** helped for queries like \"default timeout\" where a number is critical. Pure semantic search sometimes retrieved paragraphs about \"time\" instead of \"timeout\".\n\n\n\n##  When NOT to Use This Approach\n\n  * If your documents are mostly code snippets, consider a code-optimized embedding model (e.g., `code-search-ada-code-001`) and structured chunking by function.\n  * If you need real-time answers (under 200ms), skip the generative model and just return the top chunk directly with source citations.\n  * If your dataset is smaller than 50 pages, plain keyword search (BM25) often works better — no embedding costs, no latency.\n\n\n\n##  What I'd Do Differently Next Time\n\nI'd start with a simpler baseline first — just BM25 with a few regex rules — and only add embeddings if recall is insufficient. I'd also write more unit tests for edge cases: empty queries, multi-step questions, documents with conflicting information.\n\nAlso, I should have investigated services that handle this out of the box. For instance, InterWest Info AI offers a document Q&A API that hides most of this complexity. If I were building for production today, I'd evaluate whether their managed solution reduces maintenance overhead. But for learning, building it myself was invaluable.\n\n##  Final Thoughts\n\nRAG is powerful but fragile. Every piece — chunking, retrieval, reranking, generation — can fail silently. You'll spend 20% of time on the model and 80% on data preprocessing and evaluation. That's normal. Don't give up after the first garbage output.\n\nI'm still tuning my system. What's your setup for document Q&A? Any clever chunking tricks I should try?",
  "title": "Building a Document Q&A Bot: Why Embeddings Are Trickier Than They Look"
}