{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreigiqhhgmkdrl2hqkye35y3qt7eaab5dcjbge75rfjptuh6qxbaxua",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpo6xmwratd2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreidubq7cb5eadmzljaypx5hhfb4lin6tal67pdn4jb3uzi62ywxu2q"
    },
    "mimeType": "image/webp",
    "size": 93270
  },
  "path": "/rogervinas/strands-agents-langfuse-evaluations-25hd",
  "publishedAt": "2026-07-02T13:18:29.000Z",
  "site": "https://dev.to",
  "tags": [
    "langfuse",
    "strandsagents",
    "evaluations",
    "ai",
    "AI was supposed to take my job — instead it gave me a new one: Evaluations",
    "See source code at Github",
    "Strands Agents",
    "Arize Phoenix",
    "MLflow",
    "LangSmith",
    "W&B Weave",
    "Datadog",
    "AWS AgentCore",
    "Azure AI Foundry",
    "Google Vertex AI",
    ".python-version",
    "uv",
    "self-hosted with Docker Compose",
    "docker-compose-langfuse.yml",
    "official docker-compose",
    "http://localhost:3000",
    "http://localhost:8000",
    "src/banking_sentinel",
    "Strands Agents tutorial",
    "Strands session management docs",
    "api.py",
    "Strands Evals SDK",
    "evals/strands/run_evaluations.py",
    "DeepEval",
    "Ragas",
    "Braintrust autoevals",
    "pytest",
    "evals/langfuse/create_dataset.py",
    "evals/langfuse/run_experiment.py",
    "Langfuse API",
    "evals/langfuse/create_annotation_queue.py",
    "evals/langfuse/create_prompt.py",
    "agent.py",
    "AI Engineering Loop",
    "Strands Agents docs",
    "Langfuse docs",
    "Langfuse × Strands Agents integration",
    "Strands Agents quick start",
    "Langfuse tracing",
    "Langfuse datasets",
    "Langfuse model-based evaluations",
    "Langfuse user feedback",
    "Langfuse annotation queues",
    "Langfuse prompt management",
    "@app.post"
  ],
  "textContent": "> Before diving in, check out **AI was supposed to take my job — instead it gave me a new one: Evaluations** , a presentation that walks through this PoC.\n\nSee source code at Github\n\nIn this project we will build a Python banking assistant agent using Strands Agents and make it observable and continuously evaluated using Langfuse — step by step.\n\nStrands Agents is a lightweight Python SDK for building LLM-powered agents with tool use and session memory, open-sourced by AWS in May 2025. It is Python-native — which pairs well with the Langfuse Python SDK — and new enough to be worth exploring. Any other Python agent framework would work just as well for this PoC.\n\nWith **classic applications** , **quality is enforced through unit tests, integration tests, and static analysis** — every function has a defined contract and a deterministic output you can assert on. In production, metrics (error rates, latency, memory) surface failures reliably.\n\n**AI applications break both of these**. The same input may yield different outputs on each run — wording changes, tools get called in a different order, edge cases surface unpredictably. And in production, a request can return 200 OK in 300ms with a confident, completely wrong answer — classic metrics won't catch it. **You need something more!**\n\nThat's where **traces** and **evaluations** come in — supported by a growing number of platforms: Langfuse, Arize Phoenix, MLflow, LangSmith, W&B Weave, Datadog, AWS AgentCore, Azure AI Foundry, Google Vertex AI\n\nThis PoC uses **Langfuse** because it is open-source and is self-hostable with a single `docker compose up`, providing these features:\n\n  * **Tracing** — recording a structured tree of every LLM call, tool call, and sub-agent step: inputs, outputs, latency, cost.\n  * **Evaluations** — running scored assessments of agent outputs:\n    * **Offline** — run against a fixed, curated dataset before or after a change; re-runnable as a CI quality gate.\n    * **Online** — async, triggered by live traces; catch issues that didn't appear in your fixed dataset.\n  * **External Evaluations** — attaching scores to live traces programmatically from your own code.\n  * **Annotation Queues** — routing traces to human reviewers via explicit programmatic calls.\n  * **Prompt Management** — versioning prompt templates and pulling them at runtime via SDK.\n\n\n\nThe app under evaluation is the **banking sentinel** — a customer support agent for ROGERVINAS bank built with Strands Agents: 3 mock accounts with 5 transactions each, and tools to freeze/unfreeze cards, look up transactions, and open or track disputes.\n\n**Ready?** Get the agent and Langfuse running (Configuration → Run), open the chat UI and play with it, then go through the Implementation steps to see how each piece works.\n\n  * Configuration\n  * Run\n  * Implementation\n    * Step 1: The Banking Agent\n    * Step 2: Tracing\n    * Step 3: Strands Native Evaluations\n    * Step 4: Experiments\n    * Step 5: Online Evaluations (LLM-as-judge)\n    * Step 6: External Evaluations\n    * Step 7: Annotation Queues\n    * Step 8: Prompt Management\n  * CI/CD\n  * Documentation\n\n\n\n##  Configuration\n\n###  Prerequisites\n\n  * Python (version pinned in .python-version)\n  * uv\n  * Docker + Docker Compose\n  * A model provider (see below)\n\n\n\n###  Install dependencies\n\n\n    uv sync --extra dev --extra evals\n\n\n  * `dev` — dependencies for unit tests\n  * `evals` — dependencies for evaluations\n\n\n\n###  Environment\n\n\n    cp .env.example .env\n\n\nEdit `.env` to set your model provider and Langfuse credentials.\n\n###  Model providers\n\nSet `MODEL_PROVIDER` in `.env`:\n\nProvider | `MODEL_PROVIDER` | Requirements\n---|---|---\nOllama (default) | `ollama` |  `ollama serve` + `ollama pull llama3.1:8b`\nAWS Bedrock | `bedrock` | AWS credentials configured\nGoogle Gemini | `gemini` |  `GOOGLE_API_KEY` in `.env`\n\n###  Langfuse\n\nWe use the self-hosted with Docker Compose deployment. This PoC's docker-compose-langfuse.yml is a copy of the official docker-compose.\n\nTo start:\n\n\n\n    docker compose -f docker-compose-langfuse.yml up -d\n\n\nOpen the Langfuse UI at http://localhost:3000 → project `banking-sentinel`, with these pre-provisioned credentials:\n\n\n\n    Email:      admin@local.dev\n    Password:   password\n\n\nTo stop:\n\n\n\n    docker compose -f docker-compose-langfuse.yml down\n\n\nTo stop and delete all data:\n\n\n\n    docker compose -f docker-compose-langfuse.yml down -v\n\n\n##  Run\n\n\n    uv run uvicorn banking_sentinel.api:app --reload\n\n\nOpen http://localhost:8000 and try it out:\n\n  * Use the textboxes to simulate a different **User ID** , **Account ID** , **Tier** , and **Session ID**.\n  * Chat with the agent — for example, say _\"Help, I do not have Netflix!\"_.\n  * Each answer may include **suggested actions** you can click directly, plus 👍 / 👎 feedback buttons.\n\n\n\nAgent traces are sent to Langfuse automatically via OpenTelemetry, as the `LANGFUSE_*` and `OTEL_*` env vars are set in `.env`.\n\n##  Implementation\n\nWith the agent and Langfuse running, let's walk through how each piece works, step by step.\n\n###  Step 1: The Banking Agent\n\nThe domain: a banking customer support agent — the **Sentinel** — for ROGERVINAS bank. Three mock accounts (`ACC-1001`, `ACC-1002`, `ACC-1003`), five transactions each, and seven tools:\n\n  * `freeze_card` — Freeze a card\n  * `unfreeze_card` — Unfreeze a card\n  * `is_card_frozen` — Check freeze status\n  * `get_transactions` — List transactions between two dates\n  * `open_dispute` — Open a dispute on a transaction\n  * `get_dispute_status` — Check dispute status\n  * `list_disputes` — List all disputes for an account\n\n\n\nThe implementation is intentionally minimal — a single process, file-based session storage, no RAG (the knowledge base is short enough to inline directly in the system prompt), no MCP, no external service calls. The goal here is to keep the agent simple so the focus stays on observability and evaluations. The full implementation lives under src/banking_sentinel; if you are new to Strands, follow the Strands Agents tutorial first.\n\nIn production, the agent would run behind a load balancer with multiple replicas, connect to real external services via MCP or direct API calls, and swap the local file-based session storage for a persistent store — see the Strands session management docs.\n\n###  Step 2: Tracing\n\n**Traces are not standardised** — what you get depends entirely on the framework and instrumentation. The GenAI OTel semantic conventions are still experimental, so SDKs and platforms implement them inconsistently — always verify your traces in the UI before relying on them.\n\nIn this PoC, for example, the Strands `[otel]` extra emits spans but does not set trace-level `input`/`output` on the root span — and we need those in later steps. So api.py wraps the entire `/chat` request in `langfuse.start_as_current_observation()`, which sets `input`/`output` on the root span while capturing the Strands OTel spans as children:\n\n\n\n    with langfuse.start_as_current_observation(name=\"banking-sentinel-chat\", as_type=\"generation\") as span:\n        # ... run the agent ...\n        span.update(input=request.message, output=response.answer, prompt=prompt_obj)\n\n\nThis produces the following span hierarchy in Langfuse:\n\n\n\n    banking-sentinel-chat  ← Langfuse-native root span (input/output/user_id)\n      └── Strands OTel spans  ← captured automatically\n\n\nOpen http://localhost:3000 → **Traces** to see it:\n\n###  Step 3: Strands Native Evaluations\n\nThe Strands Evals SDK provides `Case`, `Experiment`, and `OutputEvaluator`. No Langfuse required — runs fully offline and produces a local report. The full code for this step lives in evals/strands/run_evaluations.py.\n\nEach `Case` bundles an input and expected output:\n\n\n\n    CASES = [\n        Case(\n            name=\"unauthorized-netflix-charge\",\n            input={\n                \"userId\": \"user-1001\", \"accountId\": \"ACC-1001\", \"accountTier\": \"Standard\",\n                \"message\": \"I don't have Netflix but I see a charge on my account\",\n            },\n            expected_output={\n                \"suggestedActions\": [\"FREEZE_CARD\"],\n                \"claim\": \"The AI agent found a Netflix charge of 9.99 and offered the user to open a dispute\",\n            },\n        ),\n        # ...\n    ]\n\n\nTwo evaluators score each result — one deterministic, one LLM-as-judge:\n\n\n\n    class CorrectnessEvaluator(OutputEvaluator):\n        pass\n\n    class ClaimEvaluator(OutputEvaluator):\n        pass\n\n    correctness_evaluator = CorrectnessEvaluator(\n        model=_model,\n        rubric=\"Score 1.0 if the actual output's suggested_actions contains all actions listed in expected_output's suggestedActions. Score 0.0 if any expected action is missing.\",\n    )\n\n    claim_evaluator = ClaimEvaluator(\n        model=_model,\n        rubric=\"Score 1.0 if the actual output's answer matches the claim in expected_output. Score 0.0 if the answer does not match the claim.\",\n    )\n\n\n> **Note:** Strands Evals can evaluate any Python callable — there is no coupling to the Strands agent itself. If you prefer a different evaluation framework, alternatives include DeepEval, Ragas (RAG-focused), Braintrust autoevals, or plain pytest with custom assertions.\n\nThere are two ways to run the task:\n\n  * **Embedded** — the agent is instantiated in-process; external services are mocked, and you can inspect internal state (white-box). Fast, no server needed, ideal for CI. Use this when you want fast, isolated, reproducible runs.\n  * **API** — the task hits a running server with real external services (black-box). Use this to validate against a live deployment or when mocking is not practical.\n\n\n\nRun embedded — no server needed:\n\n\n\n    uv run python -m evals.strands.run_evaluations embedded\n\n\nRun against a running server:\n\n\n\n    uv run python -m evals.strands.run_evaluations api --url http://localhost:8000\n\n\nReport should look like:\n\n\n\n    ============================================================\n    Evaluator: CorrectnessEvaluator\n    Overall score: 1.00\n      ✅ unauthorized-netflix-charge: score=1.00 — suggested_actions contains the expected 'FREEZE_CARD'.\n      ✅ expired-dispute-window: score=1.00 — suggested_actions contains the expected 'FREEZE_CARD'.\n\n    ============================================================\n    Evaluator: ClaimEvaluator\n    Overall score: 1.00\n      ✅ unauthorized-netflix-charge: score=1.00 — identifies the Netflix charge and offers to open a dispute.\n      ✅ expired-dispute-window: score=1.00 — explains the 14-day dispute window has expired.\n\n    ✅ Evaluation PASSED\n\n\n###  Step 4: Experiments\n\nA Langfuse **experiment** is an offline evaluation run: your agent is executed against a curated dataset, each output is scored automatically, and the results are stored so you can compare across code versions, prompt changes, and model upgrades over time.\n\nA **dataset** is a versioned collection of test cases — each item has an `input`, an `expected_output`, and optional `metadata`. Each experiment execution is named and stored against that dataset, with evaluator scores recorded per item.\n\nWhere Step 3: Strands Native Evaluations produces a local pass/fail report with no history, Langfuse Experiments add:\n\n  * **Comparison across runs** — see how scores change between code versions, prompt changes, or model upgrades side by side in the dashboard\n  * **Persistent results** — every run is stored; you can go back and audit any historical experiment\n\n\n\nBoth Step 3: Strands Native Evaluations and Step 4: Experiments act as a CI quality gate — the script exits non-zero if scores drop below threshold.\n\nThis step has three parts:\n\n  1. Create the dataset\n  2. Implement the experiment\n  3. Run the experiment\n\n\n\n####  Create the dataset\n\nEach item has an `input`, an `expected_output`, and optional `metadata`. Datasets can be created in two ways:\n\n  * **Via UI** : go to http://localhost:3000 → **Datasets** → **New dataset** , then add items manually\n  * **Programmatically** (idempotent — safe to run repeatedly), see evals/langfuse/create_dataset.py:\n\n\n\n\n    uv run python -m evals.langfuse.create_dataset\n\n\n####  Implement the experiment\n\n**Evaluators** are plain Python callables — there are no built-in evaluators in the SDK and no base class to inherit from. Any function that matches this signature works:\n\n\n\n    from langfuse.experiment import Evaluation\n\n    def my_evaluator(\n        *,\n        input: Any,           # the dataset item's input\n        output: Any,          # what the task returned\n        expected_output: Any, # the dataset item's expected_output\n        metadata: Optional[Dict[str, Any]],\n        **kwargs: Any,\n    ) -> Evaluation:          # or List[Evaluation] for multiple metrics at once\n        ...\n        return Evaluation(\n            name=\"my-metric\",   # metric name shown in the dashboard\n            value=1.0,          # int | float | str | bool\n            comment=\"optional\", # shown alongside the score\n        )\n\n\nWe implement the same two evaluators as in Step 3: Strands Native Evaluations — one deterministic (**correctness**), one LLM-as-judge (**claim**). The LLM-as-judge runs **locally** using whatever `MODEL_PROVIDER` you have configured (Ollama, Bedrock, Gemini) — it is not Langfuse's evaluation infrastructure, just your own model called from Python (see evals/langfuse/run_experiment.py):\n\n\n\n    def correctness_evaluator(*, output, expected_output, **kwargs) -> Evaluation:\n        \"\"\"Deterministic: checks if all expected suggested actions are present.\"\"\"\n        expected = set(expected_output.get(\"suggestedActions\", []))\n        actual = set(output.get(\"suggested_actions\", []))\n        score = len(expected & actual) / len(expected) if expected else 1.0\n        return Evaluation(name=\"correctness\", value=score, comment=f\"Expected {expected}, got {actual}\")\n\n    def claim_evaluator(*, output, expected_output, **kwargs) -> Evaluation:\n        \"\"\"LLM-as-judge: runs locally with your configured MODEL_PROVIDER.\"\"\"\n        judge = Agent(model=_model, callback_handler=lambda **_: None)\n        result = judge(\n            f\"Does the following answer match the claim? Reply with YES or NO only.\\n\\n\"\n            f\"Answer: {output['answer']}\\n\\nClaim: {expected_output['claim']}\"\n        )\n        return Evaluation(name=\"claim_match\", value=1.0 if \"YES\" in str(result).upper() else 0.0)\n\n    result = langfuse.run_experiment(\n        name=\"banking-sentinel\",\n        data=dataset.items,\n        task=embedded_task,\n        evaluators=[correctness_evaluator, claim_evaluator],\n        max_concurrency=1,\n    )\n\n\n####  Run the experiment\n\nRun embedded — no server needed:\n\n\n\n    uv run python -m evals.langfuse.run_experiment embedded\n\n\nRun against a running server:\n\n\n\n    uv run python -m evals.langfuse.run_experiment api --url http://localhost:8000\n\n\nOpen http://localhost:3000 → **Datasets** to see results:\n\n###  Step 5: Online Evaluations (LLM-as-judge)\n\nLangfuse can automatically score live traces as they arrive — no code changes needed. In this PoC all chat traces are tagged `banking-sentinel`, and the root span of each is named `banking-sentinel-chat`, making them easy to target.\n\n**Setup:**\n\n**1 — Add LLM Connection:**\nGo to http://localhost:3000 → **Settings → LLM Connections** → add your model provider API key.\n\n**2 — Set default evaluation model:**\nGo to http://localhost:3000 → **LLM-as-a-Judge** → set the **Default Evaluation Model** to the connection you just added.\n\n**3 — Create the evaluator:**\nGo to http://localhost:3000 → **LLM-as-a-Judge** → **Create Evaluator**. Two options:\n\n  * **Built-in evaluators** (e.g. **Helpfulness** , **Hallucination**) — often too generic for a specific domain.\n  * **Custom Evaluator** — your own domain-aware prompt. This is what we use for the PoC.\n\n\n\nFill in the custom evaluator:\n\n  * **Name** — e.g. `banking-sentinel-helpfulness`\n  * **Model** — leave **Use default evaluation model** checked (the model you set in step 2)\n  * **Evaluation prompt** — reference the trace content with `{{input}}` and `{{output}}` (see below)\n  * **Score reasoning prompt** and **Score range prompt** — leave at their defaults or customize at will\n\n\n\nEvaluation prompt:\n\n\n\n    You are evaluating a banking customer-support assistant for ROGERVINAS bank.\n    Customers often report problems informally, without using precise banking\n    terms. Evaluate how helpful the assistant's reply is: does it identify the\n    relevant transaction, explain the applicable policy (e.g. the dispute window),\n    and offer concrete next steps (freeze card, open dispute)?\n\n    Customer message:\n    {{input}}\n\n    Assistant reply:\n    {{output}}\n\n\n**4 — Configure the rule:**\n\n  1. Set target to `Observations`, filter by `Type = GENERATION`\n  2. Ensure **Run on live incoming observations** is checked (it is the default)\n  3. Add filter: `Tags` → `any of` → `banking-sentinel`\n  4. Add filter: `Name` → `any of` → `banking-sentinel-chat` — targets only the root span; avoids double-scoring the inner Strands generation (both carry the tag but have different names)\n  5. Set **Sampling** (100% is fine for PoC — reduce in production to control costs)\n  6. Map the evaluator's prompt variables to the trace fields accordingly\n  7. Click `Execute` — scores existing matching observations immediately and new ones going forward\n\n\n\nOpen http://localhost:3000 → **Traces** and click a `banking-sentinel-chat` trace. The `banking-sentinel-helpfulness` score shows as a small badge on the root span, with the full detail in the **Scores** tab:\n\n> **Note:** The online evaluations themselves produce traces — each LLM-as-judge call is a generation Langfuse records.\n>\n> **Note:** The Langfuse API to create evaluators and rules programmatically is unstable and **only available on Langfuse Cloud** — not in self-hosted deployments.\n\n###  Step 6: External Evaluations\n\nLangfuse lets you attach scores to any trace programmatically from your own code using `langfuse.create_score()`. Common use cases include:\n\n  * **User feedback** — 👍/👎 ratings from end users\n  * **Guardrail results** — PII checks, content policy, format validation\n  * **Agent self-scoring** — quality signal computed inline and submitted back\n  * **Custom pipelines** — any score your application logic can produce\n\n\n\nIn this PoC we implement **user feedback** as our example. Automated evaluators catch a lot, but human judgment is still essential — collecting 👍/👎 feedback from real users gives you a ground-truth signal that no automated evaluator can fully replace.\n\n**How it works:**\n\n  1. The `/chat` endpoint returns a `trace_id` in every response (read from `span.trace_id`)\n  2. The chat UI attaches 👍/👎 buttons to each assistant message, keyed to that `trace_id`\n  3. On click, the UI posts to `/feedback` with `value: 1.0` (👍) or `value: 0.0` (👎)\n  4. The backend (api.py) calls `langfuse.create_score()` — the score appears on the trace immediately\n\n\n\n\n    @app.post(\"/feedback\")\n    def feedback_endpoint(request: FeedbackRequest):\n        langfuse.create_score(\n            trace_id=request.trace_id,\n            name=\"user-feedback\",\n            value=request.value,   # 1.0 = 👍, 0.0 = 👎\n            comment=request.comment,\n        )\n\n\nIn the chat UI, send a message and click 👍 or 👎 on the reply. Then open http://localhost:3000 → **Traces** and click that trace. The `user-feedback` score shows as a small badge on the root span, with the full detail in the **Scores** tab:\n\n###  Step 7: Annotation Queues\n\nAnnotation queues are a human review workflow — domain experts manually score traces to build ground truth, validate LLM-as-judge results, or investigate failures.\n\n**Key concept:** Langfuse provides the queue infrastructure and a programmatic API to add items — but there are no automatic routing rules or triggers built in. Items only enter a queue through an explicit call, either from the UI (ad-hoc) or from your code. **Your code owns the routing logic.**\n\nCommon triggers you can implement:\n\n  * User gives 👎 — route negative feedback traces for human investigation\n  * Experiment score below threshold — enqueue failing traces to build better ground truth\n  * Online evaluator scores low — poll scores and enqueue traces below a quality bar\n  * Specific intent detected — route traces matching certain patterns (complaints, edge cases)\n  * Random sampling — periodically enqueue a % of production traces for ongoing quality checks\n\n\n\n**1 — Create the queue (once, idempotent):**\n\n\n\n    uv run python -m evals.langfuse.create_annotation_queue\n\n\nThis creates a `quality` score config (numeric, 0–1) and the `banking-sentinel-review` queue that uses it, so reviewers score each trace on `quality` — you could attach more score configs for other dimensions (see evals/langfuse/create_annotation_queue.py).\n\nSet `ANNOTATION_QUEUE_ID` in `.env` to the returned queue ID.\n\n**2 — Enqueue traces (example: on 👎):**\nIn this PoC, the `/feedback` endpoint in api.py adds the trace to the queue whenever the user gives a thumbs down:\n\n\n\n    if request.value == 0.0 and _annotation_queue_id:\n        langfuse.api.annotation_queues.create_queue_item(\n            _annotation_queue_id,\n            object_id=request.trace_id,\n            object_type=AnnotationQueueObjectType.TRACE,\n        )\n\n\n**3 — Review the queue:**\n\n  1. Go to http://localhost:3000 → **Annotation Queues**\n  2. Open `banking-sentinel-review`\n  3. For each trace: review the conversation, assign a `quality` score (and optionally add a comment), click **Mark Completed**\n  4. Scores appear on the trace and contribute to your evaluation dashboard\n\n\n\nAnnotating an item — assign the `quality` score and an optional comment:\n\nThe `quality` score then shows as a small badge on the root `banking-sentinel-chat` span:\n\n###  Step 8: Prompt Management\n\nLangfuse can store and version system prompts independently of your code — iterate on the prompt without redeploying the app.\n\n**1 — Create the prompt programmatically:**\n\n\n\n    uv run python -m evals.langfuse.create_prompt\n\n\nEach run creates a new version (see evals/langfuse/create_prompt.py). The `production` label is set automatically, so it is served at runtime.\n\n**2 — Or create the prompt via the UI:**\n\nGo to http://localhost:3000 → **Prompts** → **New prompt** , then:\n\n  * Name it `banking-sentinel-system`, type `Text`\n  * Paste the template using `{{variable}}` syntax (Mustache)\n  * Add the `production` label\n  * Save\n\n\n\nEither way, the prompt appears in **Prompts** , versioned and labeled `production`:\n\n**3 — Use the prompt from code:**\n\nSet `USE_LANGFUSE_PROMPT=true` in `.env` and `create_agent()` in agent.py fetches the prompt from Langfuse; otherwise it uses the hardcoded template:\n\n\n\n    def create_agent(langfuse, model, tools, user_tier, account_id, reference_date, session_manager=None) -> tuple:\n        if langfuse is not None and os.getenv(\"USE_LANGFUSE_PROMPT\", \"false\").lower() == \"true\":\n            ...  # fetch the versioned prompt via langfuse.get_prompt(...)\n        else:\n            ...  # use the hardcoded template\n        ...\n\n\nIt returns `prompt_obj`, which api.py will pass back to Langfuse to link the prompt version to the trace:\n\n\n\n    span.update(input=request.message, output=response.answer, prompt=prompt_obj)\n\n\nThe trace then shows a `Prompt: banking-sentinel-system - v1` tag linking to the exact version used:\n\nBenefits: version history, compare prompt versions across experiments, iterate without redeploying, A/B test prompts.\n\n##  CI/CD\n\nFor this PoC, CI runs three sequential jobs that gate on each other — each stage must pass before the next starts:\n\n  1. **Build** — installs dependencies, builds the package, runs unit tests\n  2. **Standalone Evals** — runs Strands native evaluations in embedded mode (no Langfuse). Fails if any score drops below a threshold.\n  3. **Langfuse Evals** — runs Langfuse experiments and reports results to the dashboard. Fails if any score drops below a threshold. For this PoC we start Langfuse with Docker just for the CI run, but a real setup would point at a shared, always-on instance.\n\n\n\nThis means a code or prompt change that degrades agent quality will fail CI before it can reach production.\n\nLangfuse acts as an active quality gate:\n\n  * Traces, scores, and experiment history accumulate across every PR and deploy — production traces surface regressions and datasets grow from real failures.\n  * A deployment job runs only after all eval jobs pass, so CI blocks deploys that would degrade quality — with score thresholds tuned per metric as you build up baseline data.\n\n\n\nThis is the continuous AI Engineering Loop as documented by Langfuse:\n\n##  Documentation\n\n  * Strands Agents docs\n  * Langfuse docs\n  * Langfuse × Strands Agents integration\n  * Strands Agents quick start\n  * Langfuse tracing\n  * Strands Evals SDK\n  * Langfuse datasets\n  * Langfuse model-based evaluations\n  * Langfuse user feedback\n  * Langfuse annotation queues\n  * Langfuse prompt management\n\n\n\nHappy GenAI coding! 💙",
  "title": "Strands Agents + Langfuse Evaluations"
}