{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreie6pmfoyur7zwhaiki3sxin4yx4fyqgd6sjavxh3z7gmgglhxk5xi",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3moo3wn7vrel2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreia4fqtz6oogcjae3hv6obynn2g5ad77h7stwvycm43thr3xg7tw2i"
    },
    "mimeType": "image/webp",
    "size": 92924
  },
  "path": "/alex_spinov/your-llm-judge-costs-more-than-the-agent-gate-it-in-40-lines-cc7",
  "publishedAt": "2026-06-19T19:30:16.000Z",
  "site": "https://dev.to",
  "tags": [
    "ai",
    "finops",
    "python",
    "agents",
    "Dev.to",
    "TechCrunch",
    "Linux Foundation",
    "pre-execution gate",
    "success gate",
    "waste-after-failure",
    "sliding-window spend guard"
  ],
  "textContent": "**LLM judge cost is the share of your eval bill spent grading agent output instead of producing it.** To control it, run a 40-line offline pre-gate that triages every span with four deterministic rules and escalates only the uncertain tail to the expensive judge. On one trace this cut judge cost share from 50% to 16%.\n\n**LLM judge cost** is the line item nobody puts on the FinOps dashboard. You add an LLM-as-judge to grade every agent span, you sleep better, and three weeks later the eval layer is quietly billing a third of what the agent itself costs. This post measures that share of your bill spent _judging_ instead of _doing_ , with a 40-line offline meter, and shows the one move that drops it from 50% to 16% on the same trace.\n\n> **AI disclosure:** I drafted this with an AI writing assistant. The tool, both fixtures, and every number below come from a real local run of `judge_gate.py` on Python 3.13.5, no network, no API key. I ran it, checked the exit codes, hashed the output twice to confirm it's deterministic, and edited every line myself before publishing.\n\nHere's the sentence that set me off. Sattyam Jain wrote it on Dev.to on June 12, in a post arguing you should stop running an LLM judge on every agent call: _\"if your monitor exceeds ~20–25% of production cost, you built the wrong monitor.\"_ (Dev.to) That's a great rule of thumb. It's also unfalsifiable until you can put a number on _your_ monitor. His post sketches the tiered architecture (cheap deterministic heuristics first, expensive judge last) but ships no code you can run against your own trace. So I wrote the missing 40 lines.\n\nThe timing isn't an accident. The token bill is coming due across the whole industry right now. TechCrunch reported on June 5 that _\"Uber blew through its entire 2026 AI coding budget by April,\"_ and that a Priceline employee saw _\"a routine Cursor contract renewal came back 4–5x more expensive.\"_ (TechCrunch) Two days earlier the Linux Foundation announced its _intent to launch the Tokenomics Foundation_ — open standards for AI cost management, because, in Jim Zemlin's words, _\"tokens have become the new unit of technology spend.\"_ (Linux Foundation) Everyone's auditing what the agent spends. Almost nobody's auditing what the _watchdog_ spends.\n\nAnd the watchdog is an LLM call too. You priced the agent. Did you price the thing watching the agent?\n\n##  TL;DR\n\n  * An LLM judge on every span isn't rigor — it's a second agent you forgot to budget. Price it before it surprises you.\n  * `judge_gate.py` is a 40-line, offline, keyless, zero-network script. Feed it a JSONL trace; four deterministic rules triage each span as OK / BAD / UNCERTAIN, and only UNCERTAIN ones would reach the expensive judge.\n  * On a well-instrumented 50-span trace it resolved **68% cheaply** and sent only **32% to the judge → 16% judge cost share** (exit 0, PASS). On the same agent logged as free text, **100% escalated → 50% cost share** (exit 1, FAIL).\n  * The judge is never actually called. It's _priced_ via configurable `--judge-price` and `--prod-cost` flags. Substitute your own rates; I ship neutral placeholder units.\n  * Exit code is a CI gate: 0 if judge cost share ≤ budget (default 0.25), 1 if over, 2 on bad input. Deterministic — byte-identical across runs.\n\n\n\nThis is the next piece in a series on **controlling agents before they execute, not after**. The pre-execution gate gates the agent's _action_. The success gate decides _what_ to verify in a result. This one is a level up the stack: it doesn't gate the agent at all. It gates the _judge_ — and asks how much that judge is allowed to cost.\n\n##  What \"judge cost share\" actually means\n\nHere's the failure mode I keep seeing. Someone reads that agents silently fail (true) and bolts on an LLM-as-judge to grade every step. Every span: a second model call, often a frontier model, sometimes with a chunky rubric prompt. It works. It catches things. Then the finance person asks why the eval bill is the same order of magnitude as the agent bill, and the honest answer is \"because we run a full second model over every single thing the first one does.\"\n\nThe number that matters is a ratio. Call it **judge cost share** : the cost of the judging layer divided by the cost of the production run it's judging.\n\n\n    judge_cost_share = (judge_calls × judge_price) / prod_cost\n\n\nIf that's 8%, fine — cheap insurance. If it's 50%, you didn't add a monitor, you added a co-pilot you're paying full freight for and calling overhead. The whole game is shrinking `judge_calls`: the number of spans that _actually need_ a human-grade judgment, versus the spans a dumb deterministic rule can settle for free.\n\nMost spans don't need a judge. A tool either got called or it didn't. A JSON output either parses or it doesn't. A 200 with an empty body is wrong no matter how confident the prose around it sounds. You don't need a frontier model to know `[]` is not a successful invoice send. You need an `if` statement.\n\n##  The fix: triage every span, escalate only the uncertain tail\n\nThe pre-gate is a function. It looks at one span and returns one of three verdicts:\n\n  * **OK** — cheaply, provably fine. Don't pay to judge it.\n  * **BAD** — cheaply, provably broken. Don't pay to judge it either; you already know.\n  * **UNCERTAIN** — the cheap rules abstain. _This_ is the only span the expensive judge should ever see.\n\n\n\nFour rules carry almost all the weight. They're the deterministic heuristics Sattyam Jain pointed at (\"did the claimed gate execute?\") turned into code:\n\n  1. **Claim-vs-evidence.** The span says it called `send_email`, but `tools_called` doesn't contain `send_email`. Claim without evidence → BAD. (This is the same idea as the success gate's middle check, reused here as a free triage rule.)\n  2. **Output schema.** The output isn't even a JSON object — it's a raw string, or it's missing. → BAD.\n  3. **200-with-empty-payload.** Status says success, body is empty. The classic silent lie. → BAD.\n  4. **Duplicate retry.** This span's argument hash equals the previous span's. A byte-identical retry — the waste-after-failure loop signature. → BAD.\n\n\n\nIf none of those fire and the span has a clean `ok: true` + 200, it's **OK**. Otherwise the rules abstain and it's **UNCERTAIN** — escalate. Here's the whole triage:\n\n\n\n    def triage(span):\n        \"\"\"Return (verdict, rule). UNCERTAIN means 'a human-grade LLM judge is needed'.\"\"\"\n        out = span.get(\"output\")\n        if not isinstance(out, dict):                      # output not valid JSON object\n            return \"BAD\", \"schema:not-an-object\"\n        if span.get(\"claimed_tool\") and span[\"claimed_tool\"] not in span.get(\"tools_called\", []):\n            return \"BAD\", \"claim-without-evidence\"         # said it called X, trace has no X\n        if span.get(\"status\") == 200 and not out:          # 200 OK with empty payload\n            return \"BAD\", \"200-empty-payload\"\n        if span.get(\"arg_hash\") and span[\"arg_hash\"] == span.get(\"prev_arg_hash\"):\n            return \"BAD\", \"duplicate-span\"                 # byte-identical retry of prior call\n        if out.get(\"ok\") is True and span.get(\"status\") == 200:\n            return \"OK\", \"clean-success\"                   # explicit ok + 200, no contradiction\n        return \"UNCERTAIN\", \"needs-judge\"                  # cheap rules abstain -> escalate\n\n\nThat's it. No network, no key, no model. The judge layer is _priced_ , not called: I count the UNCERTAIN spans and multiply by a price you supply on the command line. I refuse to hardcode a vendor rate — those go stale in a month and I'd rather be honestly empty than confidently wrong about someone's bill.\n\n##  The run: 32% to the judge, not 100%\n\nI built two traces of the same 50-span agent — a support-desk bot doing searches, record updates, email sends, classifications, and reply drafts.\n\nThe first, `trace_gated.jsonl`, is **well-instrumented** : each span logs the tool it claimed, the tools actually called, a structured output (an `ok` flag where the verdict is clear-cut, a `confidence` value or label where it isn't), and an argument hash. The second, `trace_naive.jsonl`, is the _same agent_ logging only free-text outputs like `{\"text\": \"email sent\"}`, the way a lot of agents actually log in the wild. Same work. Different telemetry.\n\nHere's the verbatim output. I didn't touch it:\n\n\n\n    $ python3 judge_gate.py fixtures/trace_gated.jsonl --judge-price 1 --prod-cost 100\n    spans total:        50\n    resolved by gate:   34 (68.0%)  [OK=29 BAD=5]\n    sent to LLM judge:  16 (32.0%)\n    judge cost share:   16.0% of prod cost (judge_price=1.0, prod_cost=100.0, budget=25%)\n    verdict: PASS - judge layer within budget\n    $ echo $?\n    0\n\n    $ python3 judge_gate.py fixtures/trace_naive.jsonl --judge-price 1 --prod-cost 100\n    spans total:        50\n    resolved by gate:   0 (0.0%)  [OK=0 BAD=0]\n    sent to LLM judge:  50 (100.0%)\n    judge cost share:   50.0% of prod cost (judge_price=1.0, prod_cost=100.0, budget=25%)\n    verdict: FAIL - judge layer over budget\n    $ echo $?\n    1\n\n\nRead the two side by side. Same agent, same fifty spans, same `--judge-price 1 --prod-cost 100`. The well-instrumented trace sends **16 spans** to the judge and lands at **16% cost share: a PASS, exit 0**. The free-text trace can't resolve a single span cheaply, sends all **50** , and lands at **50%: a FAIL, exit 1** , tripping Sattyam Jain's \"wrong monitor\" line by a mile.\n\nThe lever isn't a fancier judge. It's whether your trace carries the four cheap facts a rule can read. Of the 16 spans that did escalate in the gated run, most are genuinely subjective: ambiguous contract summaries (`confidence: 0.45`), hedged reply drafts (`\"I cannot find the order, but it is probably fine.\"`), borderline intent labels. A handful escalate for a humbler reason — they carry no `ok` flag for a cheap rule to confirm, so the gate abstains instead of guessing. Either way, that's the tail you _want_ a human-grade judge on. The other 34? Five were provably broken (one duplicate retry, two claims with no matching tool call, one 200 with an empty body, one non-object output) and the rest were clean successes. None of those needed a model to adjudicate.\n\nI want to be precise about a number I almost fudged. The cost figures are **placeholder units** (`judge_price=1`, `prod_cost=100`). I am not telling you a judge call costs a dollar or that your run costs a hundred of anything. Plug in your real per-call judge price and your real run cost. The _rate_ , 32% vs 100% of spans escalating, is the part that's mine: measured, reproducible. The dollars are yours.\n\n##  Am I just moving the bug into the gate?\n\nFair objection, and it's the one I'd raise. If the cheap rules are wrong, you've replaced a $50 judge bill with a 16% bill _and_ a stack of bad verdicts. So: how good can a cheap layer actually be?\n\nTwo recent papers say: surprisingly good, on the parts that matter. In _Cheap Reward Hacking Detection_ (arXiv:2606.08893, June 8), Belenky, Itria and Johns put a linear probe on a small transformer encoder and detected reward hacking at **AUC 0.9467, TPR 0.8296 at 5% FPR, at \"roughly four orders of magnitude lower per-trajectory cost\"** than an LLM-as-judge baseline. And _Goal-Autopilot_ (arXiv:2606.11688) reports a gated finite-state machine that _\"forbids any terminal 'done' claim whose falsifiable gate did not actually execute and pass,\"_ cutting fabrication on SWE-bench Lite from **33.7% to 0.67%**. Those are _their_ numbers on _their_ setups, not mine. I'm citing them as evidence that a cheap deterministic layer catches most of what a dear one catches, not as my own result.\n\nMy four `if` statements are cruder than a trained probe. They don't need to be clever. They need to be _right when they're confident and silent when they're not_ — which is the whole point of the UNCERTAIN bucket. A rule that isn't sure doesn't guess. It escalates. The judge still grades the hard 32%. You just stopped paying it to rubber-stamp the easy 68%.\n\n##  What this is not\n\n  * **Not an eval suite.** It doesn't score answer quality. It decides _which spans deserve a judge_ , then prices that layer. Correctness of the hard tail is still the judge's job.\n  * **Not a runtime cap.** It reads a finished trace and fails CI. If you need to _block_ a runaway loop mid-flight, that's a sliding-window spend guard, a different tool.\n  * **Not a verdict on confidence fields.** Honest limitation: my gate ignores a span's self-reported `confidence`. One span in the fixture says `confidence: 0.95, \"no ambiguity\"` and still got escalated, because I refuse to trust a model's own confidence as a cheap signal — that's the kind of self-assessment that lies. If you trust yours, add a fifth rule. I didn't.\n  * **Not a license to skip the judge.** The judge gets the genuinely uncertain spans. The argument is against running it on the _obvious_ ones, not against running it at all.\n\n\n\n##  Run it on your own trace\n\nExport 40–60 spans of a real agent run to JSONL with six fields per span (`status`, `claimed_tool`, `tools_called`, `output`, `arg_hash`, and `prev_arg_hash` carrying the previous span's hash so the duplicate-retry rule can fire), point `judge_gate.py` at it, and pass your real `--judge-price` and `--prod-cost`. If your judge cost share comes back under 10%, ignore me; your monitor's fine. If it comes back at 40%, you've found a line item.\n\nOne thing I genuinely don't know yet and would put real money on being argued in the comments: where the honest threshold is. Sattyam Jain says 20–25%. I shipped a default of 25%. But for a low-stakes summarizer, even 10% might be waste, and for an agent that moves money, maybe 40% is cheap. The budget is a `--flag` precisely because I don't think there's one right answer.\n\nSo I'll ask you: what's the judge cost share on a real eval pipeline you've shipped — and where would _you_ set the budget before it counts as the wrong monitor?\n\n_I publish one runnable FinOps tool for AI agents at a time, with the real run log attached. Follow for the next number from the next trace — and drop your judge cost share in the comments, I read every one._",
  "title": "Your LLM Judge Costs More Than the Agent. Gate It in 40 Lines."
}