{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiayxklrquevr25qohhypo4zunvcnmlwtiupqcfze7svzom7azdoje",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3moylmau4png2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreifoyfjq7flwracbcadf3encqcg3kb7j6ogivlhsjpvh2qejelqipu"
    },
    "mimeType": "image/webp",
    "size": 86512
  },
  "path": "/javaking1129/running-a-langgraph-react-agent-in-production-openai-compatible-api-multi-model-gateway--emi",
  "publishedAt": "2026-06-23T22:57:54.000Z",
  "site": "https://dev.to",
  "tags": [
    "langchain",
    "llm",
    "python",
    "kubernetes",
    "@tool",
    "@router.post",
    "@lru_cache"
  ],
  "textContent": "Most LangGraph content stops at the notebook. You build a cute ReAct loop, it answers one question, and the article ends before the hard part: _how do you actually serve this thing, swap models without a rewrite, and see what it's doing when it misbehaves?_\n\nThis post walks through a small but **production-shaped** LangGraph deployment: a RAG ReAct agent that\n\n  * exposes an **OpenAI-compatible HTTP API** , so any OpenAI client (Open WebUI, the `openai` SDK, LibreChat) can talk to it unchanged,\n  * routes every model call through a **gateway** so switching from a hosted API to self-hosted vLLM is a config change, not a code change, and\n  * gets **full tracing** — node transitions, tool calls, and LLM calls in one trace — by adding a single callback.\n\n\n\nEvery snippet below is real code from a working service. Roughly 150 lines of Python is all it takes.\n\n##  The shape of the thing\n\n\n    OpenAI client (Open WebUI, openai SDK)\n            │  POST /v1/chat/completions\n            ▼\n    FastAPI router ──► LangGraph StateGraph ──► LLM Gateway ──► model (hosted API today, vLLM tomorrow)\n            │                   │\n            │                   └──► ToolNode ──► Qdrant (RAG)\n            │\n            └──► Langfuse callback (one trace per request)\n\n\nThe contract with the outside world is **just the OpenAI API**. Everything interesting — the graph, RAG, tracing — lives behind that boundary. That single decision is what lets an off-the-shelf chat UI drive a custom agent with zero adapter code.\n\n##  1. The ReAct graph\n\nThe graph is deliberately tiny: one `agent` node that reasons, one `tools` node that retrieves, and a conditional edge that loops between them until the model stops asking for tools.\n\n\n\n    # app/graph/builder.py\n    from langgraph.graph import END, StateGraph\n    from langgraph.prebuilt import ToolNode, tools_condition\n\n    def build_graph():\n        g = StateGraph(AgentState)\n        g.add_node(\"agent\", agent_node)\n        g.set_entry_point(\"agent\")\n\n        # ReAct: if the model emits tool_calls, go to `tools`; otherwise END.\n        g.add_node(\"tools\", ToolNode(TOOLS))\n        g.add_conditional_edges(\"agent\", tools_condition)\n        g.add_edge(\"tools\", \"agent\")\n        return g.compile()\n\n\n`tools_condition` and `ToolNode` are LangGraph prebuilts that do the unglamorous work: inspect the last message for `tool_calls`, route accordingly, execute the tools, and append `ToolMessage`s back into state. You wire the loop; they run it.\n\nState is a single shared message log with a reducer that _appends_ rather than replaces:\n\n\n\n    # app/graph/state.py\n    from typing import Annotated, TypedDict\n    from langchain_core.messages import BaseMessage\n    from langgraph.graph.message import add_messages\n\n    class AgentState(TypedDict, total=False):\n        messages: Annotated[list[BaseMessage], add_messages]\n\n\n`add_messages` is the reducer. Every node returns `{\"messages\": [...]}` and LangGraph merges it into the running log — no manual list-shuffling, and it's what makes the agent⇄tools loop accumulate context correctly.\n\nThe agent node binds the tools and calls the model. Note `bind_tools` is conditional — flip RAG off and the exact same node degrades to a plain single-shot chat call:\n\n\n\n    # app/graph/nodes/agent.py\n    async def agent_node(state: AgentState) -> dict:\n        llm = get_llm()\n        if get_settings().rag_enabled:\n            llm = llm.bind_tools(TOOLS)\n        messages = [SystemMessage(content=SYSTEM_PROMPT), *state[\"messages\"]]\n        response = await llm.ainvoke(messages)\n        return {\"messages\": [response]}\n\n\nAnd the tool itself is an ordinary `@tool`-decorated function. The docstring is not documentation — it's the prompt the model reads to decide _when_ to call it:\n\n\n\n    # app/graph/tools.py\n    @tool\n    def search_docs(query: str) -> str:\n        \"\"\"Search internal docs for content relevant to the question.\n        When the user asks about the project/system/docs, call this first.\"\"\"\n        hits = get_vector_store().similarity_search(query, k=get_settings().rag_top_k)\n        blocks = [\n            f\"[{i}] (source: {doc.metadata.get('source', 'unknown')})\\n{doc.page_content.strip()}\"\n            for i, doc in enumerate(hits, 1)\n        ]\n        return \"\\n\\n\".join(blocks) or \"No relevant documents found.\"\n\n\nReturning a `[1] (source: ...)` structure isn't cosmetic — it's how the model can cite sources in its final answer, which is the difference between a demo and something people trust.\n\n##  2. The OpenAI-compatible surface\n\nHere's the lever that makes everything else cheap: the agent speaks OpenAI's wire format. The router turns an incoming `/v1/chat/completions` request into graph input and the graph's output back into an OpenAI response.\n\n\n\n    # app/api/router.py\n    @router.post(\"/v1/chat/completions\")\n    async def chat_completions(req: ChatCompletionRequest):\n        graph = get_graph()\n        inputs = {\"messages\": to_langchain_messages(req.messages)}\n        config: dict = {}\n\n        if not req.stream:\n            result = await graph.ainvoke(inputs, config=config)\n            text = extract_final_text(result.get(\"messages\", []))\n            return make_completion(text, settings.served_model_name)\n\n        return StreamingResponse(\n            graph_to_openai_sse(graph, inputs, settings.served_model_name, config=config),\n            media_type=\"text/event-stream\",\n        )\n\n\nBecause the response matches OpenAI's schema (including SSE streaming chunks), **Open WebUI thinks it's talking to OpenAI**. You point its `openaiBaseUrl` at this service and your custom RAG agent shows up as a selectable model. No frontend work.\n\n##  3. One gateway, many models\n\nLangGraph nodes never name a provider. They call one factory:\n\n\n\n    # app/llm/client.py\n    from langchain_openai import ChatOpenAI\n\n    def get_llm(model=None, temperature=None, streaming=True) -> ChatOpenAI:\n        s = get_settings()\n        return ChatOpenAI(\n            base_url=f\"{s.litellm_url}/v1\",   # gateway, not a provider\n            api_key=s.litellm_key,\n            model=model or s.default_model,\n            temperature=s.default_temperature if temperature is None else temperature,\n            streaming=streaming,\n        )\n\n\nThe `base_url` points at a **LiteLLM gateway** , not at any specific vendor. LiteLLM exposes an OpenAI-compatible endpoint and fans out to whatever its `model_list` says — a hosted API today, self-hosted vLLM tomorrow. Migrating off a paid API to an in-cluster GPU model becomes a _gateway config edit_ ; this Python file never changes.\n\nThere's one deliberate escape hatch — when the gateway is down locally, point straight at Ollama's OpenAI-compatible endpoint:\n\n\n\n        if s.chat_provider.lower() == \"ollama\":\n            return ChatOpenAI(base_url=f\"{s.ollama_url}/v1\", api_key=\"ollama\",\n                              model=model or s.ollama_chat_model, ...)\n\n\nSame `ChatOpenAI` class, different `base_url`. The OpenAI-compatible interface shows up _three_ times in this architecture — inbound API, gateway, and local fallback — and that consistency is the whole trick.\n\n##  4. Tracing in one line\n\nA multi-node graph with a tool loop is opaque when it goes wrong. Did the model skip the tool? Retrieve garbage? Loop twice? Langfuse's LangChain callback captures the entire run — every node transition, tool call, and LLM call — as a single nested trace.\n\nThe integration is genuinely one object:\n\n\n\n    # app/obs/langfuse.py\n    from functools import lru_cache\n\n    @lru_cache\n    def get_langfuse_handler():\n        s = get_settings()\n        if not (s.langfuse_public_key and s.langfuse_secret_key):\n            return None  # no keys → tracing silently disabled (safe for local/POC)\n        from langfuse.langchain import CallbackHandler\n        return CallbackHandler()\n\n\n> Heads-up for the SDK version churn: on **Langfuse SDK v3+** the import is `from langfuse.langchain import CallbackHandler`, and the handler reads `LANGFUSE_PUBLIC_KEY` / `LANGFUSE_SECRET_KEY` / `LANGFUSE_HOST` from the environment — you don't pass keys to the constructor anymore. This tripped up a lot of v2 tutorials.\n\nThen attach it per request via the graph `config` — which is also where you stamp user/session metadata so traces are filterable in the Langfuse UI:\n\n\n\n    # app/api/router.py\n    handler = get_langfuse_handler()\n    if handler is not None:\n        config[\"callbacks\"] = [handler]\n        config[\"metadata\"] = {\n            \"langfuse_user_id\": req.user or \"anonymous\",\n            \"langfuse_session_id\": getattr(req, \"chat_id\", None) or \"no-session\",\n            \"langfuse_tags\": [\"my-agent\", settings.served_model_name],\n        }\n\n\nPassing the handler through `config[\"callbacks\"]` (rather than baking it into the LLM client) means it propagates down the _entire_ graph automatically. One request → one trace → every step visible.\n\n##  What this buys you\n\nConcern | How it's handled | Why it scales\n---|---|---\nFrontend integration | OpenAI-compatible API | Any OpenAI client works unchanged\nModel choice | LiteLLM gateway behind `ChatOpenAI` | Swap providers via config, not code\nAgent logic | LangGraph `StateGraph` + prebuilts | ReAct loop in ~10 lines, extensible to multi-agent\nObservability | Langfuse callback via graph `config` | One trace per request, zero per-node wiring\nLocal dev | Ollama fallback through same interface | No gateway needed to hack offline\n\nNone of these pieces is exotic. The point is the **seams** : an OpenAI boundary on the outside, a gateway boundary on the model side, and a callback boundary for observability. Get the seams right and the agent in the middle stays small and swappable.\n\nThe same skeleton extends cleanly to a supervisor/worker multi-agent graph, a Postgres checkpointer for persistent threads, and an in-cluster vLLM model — each is an additive change behind one of those seams. But that's a follow-up post.\n\n_Built with LangGraph, LangChain, LiteLLM, Qdrant, and Langfuse. If you're running LangGraph in production and want to compare notes on deployment patterns, reach out._",
  "title": "Running a LangGraph ReAct Agent in Production: OpenAI-Compatible API + Multi-Model Gateway + One-Line Tracing"
}