{
"$type": "site.standard.document",
"bskyPostRef": {
"cid": "bafyreigvgpt5b5el3xtpp2fowtnv7ozc7ije2dl5k2rblj4pqyvypmkmfi",
"uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpszb3exdm42"
},
"coverImage": {
"$type": "blob",
"ref": {
"$link": "bafkreigftmrvasn3vu5rr7ydddx3qdlivmnwoxsf27kyjjhhioorsvmwyq"
},
"mimeType": "image/webp",
"size": 64786
},
"path": "/hiroki-kameyama/evals-automatically-measuring-rag-answer-quality-13l2",
"publishedAt": "2026-07-04T11:45:57.000Z",
"site": "https://dev.to",
"tags": [
"ai",
"mlops",
"llm",
"python"
],
"textContent": "## Introduction\n\nIn the previous RAG implementation, we built a working system — but we could only verify \"is this actually correct?\" by reading answers manually.\n\n\n\n [Before] Manual verification\n Ask \"How do you calculate F1 score?\" → check the answer by eye\n\n [Now — Evals]\n Prepare test cases and automatically score quality\n\n\nEvals means preparing an \"evaluation dataset\" (questions and expected answers) and automatically grading the system's responses.\n\n## Three Evaluation Dimensions\n\nRAG system evaluation breaks down into three dimensions:\n\nDimension | Meaning | What It Measures\n---|---|---\n**Faithfulness** | Grounding | Does the answer rely on retrieved documents? (No hallucinations?)\n**Answer Relevancy** | Relevance | Is the answer appropriate for the question?\n**Context Recall** | Retrieval recall | Did the system correctly retrieve documents containing the answer?\n\n## Directory Structure\n\n\n pgvector-tutorial/\n ├── existing files (01–13)\n │\n ├── evals/\n │ ├── dataset.py # ★ Evaluation dataset definition\n │ ├── eval_rag.py # ★ RAG evaluation\n │ ├── eval_agent.py # ★ Agent evaluation\n │ └── report.py # ★ Evaluation report generation\n\n\n## 1. Install Libraries\n\n\n pip install pandas tabulate\n pip freeze > requirements.txt\n\n\n## 2. Evaluation Dataset — `evals/dataset.py`\n\nThe evaluation dataset consists of sets of \"question, expected answer elements, and expected reference documents.\"\n\n\n\n # evals/dataset.py\n\n EVAL_DATASET = [\n {\n \"id\": \"eval_001\",\n \"question\": \"How do you calculate the F1 score?\",\n \"expected_answer_keywords\": [\"Precision\", \"Recall\", \"harmonic mean\", \"2\"],\n \"expected_docs\": [\"Evaluation metrics for machine learning models\"],\n \"category\": \"ML\",\n },\n {\n \"id\": \"eval_002\",\n \"question\": \"How do you evaluate a model with scikit-learn?\",\n \"expected_answer_keywords\": [\"cross_val_score\", \"classification_report\", \"scikit-learn\"],\n \"expected_docs\": [\"Model evaluation with scikit-learn\"],\n \"category\": \"ML\",\n },\n {\n \"id\": \"eval_003\",\n \"question\": \"How can I reduce AWS costs?\",\n \"expected_answer_keywords\": [\"EC2\", \"spot instances\", \"cost\"],\n \"expected_docs\": [\"AWS cost optimization in practice\"],\n \"category\": \"Cloud\",\n },\n {\n \"id\": \"eval_004\",\n \"question\": \"How do you handle missing values in Pandas?\",\n \"expected_answer_keywords\": [\"missing values\", \"DataFrame\", \"Pandas\"],\n \"expected_docs\": [\"Data preprocessing with Pandas\"],\n \"category\": \"Python\",\n },\n {\n \"id\": \"eval_005\",\n \"question\": \"How do you write a Kubernetes manifest file?\",\n \"expected_answer_keywords\": [\"YAML\", \"Pod\", \"Kubernetes\"],\n \"expected_docs\": [\"Kubernetes Pod basics\"],\n \"category\": \"Cloud\",\n },\n ]\n\n\n## 3. RAG Evaluation — `evals/eval_rag.py`\n\n\n # evals/eval_rag.py\n import sys\n import os\n sys.path.append(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))\n\n import psycopg2\n from google import genai\n from google.genai import types\n from dotenv import load_dotenv\n import time\n from evals.dataset import EVAL_DATASET\n\n load_dotenv()\n\n client = genai.Client(api_key=os.getenv(\"GEMINI_API_KEY\"))\n\n conn = psycopg2.connect(\n host=os.getenv(\"DB_HOST\"),\n port=os.getenv(\"DB_PORT\"),\n dbname=os.getenv(\"DB_NAME\"),\n user=os.getenv(\"DB_USER\"),\n password=os.getenv(\"DB_PASSWORD\"),\n )\n cur = conn.cursor()\n\n\n def get_query_embedding(text: str) -> list[float]:\n result = client.models.embed_content(\n model=\"gemini-embedding-001\",\n contents=text,\n config=types.EmbedContentConfig(\n task_type=\"RETRIEVAL_QUERY\",\n output_dimensionality=768,\n ),\n )\n return result.embeddings[0].values\n\n\n def search(query: str, top_k: int = 3) -> list[dict]:\n query_embedding = get_query_embedding(query)\n cur.execute(\"\"\"\n SELECT title, body,\n 1 - (embedding <=> %s::vector) AS similarity\n FROM documents\n ORDER BY embedding <=> %s::vector\n LIMIT %s;\n \"\"\", (query_embedding, query_embedding, top_k))\n rows = cur.fetchall()\n return [\n {\"title\": r[0], \"body\": r[1], \"similarity\": round(r[2], 4)}\n for r in rows\n ]\n\n\n def rag_answer(question: str) -> tuple[str, list[dict]]:\n \"\"\"Generate a RAG answer and return the documents used.\"\"\"\n docs = search(question, top_k=3)\n context = \"\\n\\n\".join([f\"[{d['title']}]\\n{d['body']}\" for d in docs])\n prompt = f\"\"\"Answer the question based on the following documents.\n\n # Reference Documents\n {context}\n\n # Question\n {question}\n\n # Answer (concisely, based on the reference documents)\"\"\"\n\n for attempt in range(3):\n try:\n response = client.models.generate_content(\n model=\"gemini-2.5-flash\",\n contents=prompt,\n )\n return response.text, docs\n except Exception as e:\n if (\"503\" in str(e) or \"429\" in str(e)) and attempt < 2:\n time.sleep((attempt + 1) * 10)\n else:\n raise\n\n\n # ══════════════════════════════════════════\n # Evaluation functions\n # ══════════════════════════════════════════\n\n def eval_context_recall(retrieved_docs: list[dict], expected_docs: list[str]) -> float:\n \"\"\"\n Context Recall: Were expected documents included in the search results?\n Score = fraction of expected docs actually retrieved\n \"\"\"\n retrieved_titles = [d[\"title\"] for d in retrieved_docs]\n hit = sum(1 for expected in expected_docs if expected in retrieved_titles)\n return hit / len(expected_docs) if expected_docs else 0.0\n\n\n def eval_answer_relevancy(answer: str, keywords: list[str]) -> float:\n \"\"\"\n Answer Relevancy: Did the answer contain the expected keywords?\n Score = fraction of expected keywords found in the answer\n \"\"\"\n hit = sum(1 for kw in keywords if kw.lower() in answer.lower())\n return hit / len(keywords) if keywords else 0.0\n\n\n def eval_faithfulness(answer: str, retrieved_docs: list[dict]) -> float:\n \"\"\"\n Faithfulness: Is the answer grounded in the retrieved documents?\n Uses LLM-as-a-Judge pattern.\n Score = 0.0–1.0 (LLM-scored)\n \"\"\"\n context = \"\\n\\n\".join([f\"[{d['title']}]\\n{d['body']}\" for d in retrieved_docs])\n prompt = f\"\"\"Evaluate the following context and answer.\n\n # Context (retrieved documents)\n {context}\n\n # Answer\n {answer}\n\n Evaluation criteria:\n - Is the answer based on the content of the context?\n - Does it add information not present in the context? (hallucination)\n\n Return only a score from 0.0 to 1.0. No explanation. Numbers only.\"\"\"\n\n for attempt in range(3):\n try:\n response = client.models.generate_content(\n model=\"gemini-2.5-flash\",\n contents=prompt,\n )\n score_text = response.text.strip()\n return float(score_text)\n except (ValueError, Exception) as e:\n if (\"503\" in str(e) or \"429\" in str(e)) and attempt < 2:\n time.sleep((attempt + 1) * 10)\n else:\n return 0.5 # Default value on eval failure\n\n\n def run_eval():\n \"\"\"Evaluate RAG against the full evaluation dataset.\"\"\"\n results = []\n\n print(\"Starting RAG evaluation...\")\n print(\"=\" * 60)\n\n for item in EVAL_DATASET:\n print(f\"\\n[{item['id']}] {item['question']}\")\n\n answer, retrieved_docs = rag_answer(item[\"question\"])\n time.sleep(2) # Rate limit safety\n\n context_recall = eval_context_recall(retrieved_docs, item[\"expected_docs\"])\n answer_relevancy = eval_answer_relevancy(answer, item[\"expected_answer_keywords\"])\n faithfulness = eval_faithfulness(answer, retrieved_docs)\n time.sleep(2)\n\n overall = (context_recall + answer_relevancy + faithfulness) / 3\n\n result = {\n \"id\": item[\"id\"],\n \"question\": item[\"question\"][:30] + \"...\",\n \"context_recall\": round(context_recall, 2),\n \"answer_relevancy\": round(answer_relevancy, 2),\n \"faithfulness\": round(faithfulness, 2),\n \"overall\": round(overall, 2),\n }\n results.append(result)\n\n print(f\" Context Recall: {context_recall:.2f}\")\n print(f\" Answer Relevancy: {answer_relevancy:.2f}\")\n print(f\" Faithfulness: {faithfulness:.2f}\")\n print(f\" Overall: {overall:.2f}\")\n\n return results\n\n\n if __name__ == \"__main__\":\n results = run_eval()\n\n print(\"\\n\" + \"=\" * 60)\n print(\"Evaluation Summary\")\n print(\"=\" * 60)\n\n avg_recall = sum(r[\"context_recall\"] for r in results) / len(results)\n avg_relevancy = sum(r[\"answer_relevancy\"] for r in results) / len(results)\n avg_faith = sum(r[\"faithfulness\"] for r in results) / len(results)\n avg_overall = sum(r[\"overall\"] for r in results) / len(results)\n\n print(f\"Context Recall: {avg_recall:.2f}\")\n print(f\"Answer Relevancy: {avg_relevancy:.2f}\")\n print(f\"Faithfulness: {avg_faith:.2f}\")\n print(f\"Overall: {avg_overall:.2f}\")\n\n\n\n python evals/eval_rag.py\n\n\nSample output:\n\n\n\n Starting RAG evaluation...\n ============================================================\n\n [eval_001] How do you calculate the F1 score?\n Context Recall: 1.00\n Answer Relevancy: 1.00\n Faithfulness: 0.92\n Overall: 0.97\n\n [eval_002] How do you evaluate a model with scikit-learn?\n Context Recall: 1.00\n Answer Relevancy: 0.75\n Faithfulness: 0.88\n Overall: 0.88\n\n ============================================================\n Evaluation Summary\n ============================================================\n Context Recall: 0.95\n Answer Relevancy: 0.85\n Faithfulness: 0.90\n Overall: 0.90\n\n\n## 4. Reading the Results\n\nScore | Meaning\n---|---\n0.9+ | Excellent. Production-ready.\n0.7–0.9 | Good. Room for improvement.\n0.5–0.7 | Needs improvement. Review documents and search config.\nUnder 0.5 | Problem. Reconsider the design.\n\n### What to do when each metric is low\n\n**When Context Recall is low**\n→ Retrieval isn't finding the expected documents\n→ Increase `top_k`, revisit document chunking, add metadata filters\n\n**When Answer Relevancy is low**\n→ The answer is drifting from the question\n→ Improve the prompt, add a system prompt\n\n**When Faithfulness is low**\n→ The answer includes information not in the retrieved documents (hallucination)\n→ Explicitly state in the prompt: \"Do not answer questions not covered in the documents\"\n\n## 5. The LLM-as-a-Judge Pattern\n\nUsing an LLM to score itself, as in `eval_faithfulness()`, is called **LLM-as-a-Judge**.\n\n\n\n Traditional evaluation:\n Human defines correct answers → rule-based scoring\n → Fast, stable → struggles with nuanced judgment\n\n LLM-as-a-Judge:\n LLM understands evaluation criteria and scores\n → Can handle complex judgments → costs more, scores can vary\n\n\nThis implementation combines both:\n\nMetric | Approach | Reason\n---|---|---\nContext Recall | Rule-based (title match) | Clear ground truth\nAnswer Relevancy | Rule-based (keyword match) | Clear ground truth\nFaithfulness | LLM-as-a-Judge | Hallucination detection requires nuanced judgment\n\n## 6. Common Errors\n\nError | Cause | Fix\n---|---|---\n`ValueError: could not convert string to float` | LLM returned non-numeric output | Strengthen prompt, handle with default value\n`429 RESOURCE_EXHAUSTED` | Rate limit hit | Increase `time.sleep()` wait time\nScore always 0 | Keyword variation/mismatch | Revise `expected_answer_keywords`\n\n## Next Steps\n\n * **[Chapter 3: Observability]** — Trace each RAG step with Langfuse and visualize behavior\n * **Integrate RAGAS** — `pip install ragas` for a more advanced evaluation framework\n * **Continuous evaluation (CI/CD)** — Combine with GitHub Actions in Chapter 5\n\n",
"title": "Evals — Automatically Measuring RAG Answer Quality"
}