{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreifkispknqtnrdj43f3excxmuiqtwvj7ogsynumfqngoyjshy2fg6y",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mp3wzwe6dka2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreia2exyr5pywjo2vhqjarmzi5fa5cekxjlxovtb6qahz5uxelft5yy"
    },
    "mimeType": "image/webp",
    "size": 78144
  },
  "path": "/zoetaka38/when-ai-reviews-ais-code-youve-built-an-infinite-loop-heres-how-we-stopped-it-4g1n",
  "publishedAt": "2026-06-25T07:35:28.000Z",
  "site": "https://dev.to",
  "tags": [
    "ai",
    "agents",
    "codereview",
    "architecture",
    "https://www.codens.ai/en/"
  ],
  "textContent": "While building an AI code review product (Orange Codens), there was one design problem that dominated everything else: when the reviewer is AI and the fixer is AI, you can very easily build an infinite loop.\n\nWalk it through. Orange reviews a PR and finds a problem. It doesn't stop at the comment — it hands the fix off to another Codens (Purple / Red) to produce a fix PR. But that fix PR is also a PR. Orange reviews it. New code, new findings. Hand off again. Fix. Review again.\n\nA human reviewer stops at \"good enough.\" AI doesn't, and every loop costs LLM tokens. So \"it terminates\" and \"cost is bounded\" have to be guaranteed by the architecture, not by how smart the model is. Here's what we wired into Orange, with the real code and the actual non-convergence incidents we hit.\n\n##  The structure that loops\n\nBuilt naively, here's how it spins:\n\n\n\n    PR open\n      → Orange review → finding\n        → handoff → Purple/Red opens a fix PR\n          → that fix PR is open\n            → Orange review → new finding\n              → handoff → … (forever)\n\n\nThere isn't one place to cut it. \"When do we hand off\", \"whose PRs are eligible\", \"do we dispatch the same finding twice\", \"when does the review itself converge\" — each needs its own mechanism. In order:\n\n##  Cut 1: handoff fires at merge, not when a finding is created\n\nThis is the big one. We don't dispatch a fix task the moment we find something. We dispatch only when the PR is **merged**.\n\nThe webhook handler:\n\n\n\n    _REVIEWED_ACTIONS = {\"opened\", \"synchronize\", \"reopened\"}\n\n    async def execute(self, payload):\n        action = payload.get(\"action\", \"\")\n        pr = payload.get(\"pull_request\") or {}\n\n        if action == \"closed\":\n            # the merge-gated handoff trigger point\n            merged = bool(pr.get(\"merged\"))\n            return WebhookResult(\n                outcome=\"handoff\",\n                handoff_repository_id=repository.repository_id,\n                handoff_pr_number=int(pr.get(\"number\", 0)),\n                handoff_merged=merged,\n                handoff_pr_author=str((pr.get(\"user\") or {}).get(\"login\", \"\")),\n            )\n        ...\n\n\nBefore merge, it only posts GitHub suggestions. It wakes Purple/Red only on the `closed` event with `merged=true`. And a PR closed without merging **discards** its queued handoffs:\n\n\n\n    if not merged:\n        # the code never lands on main, so drop queued handoffs\n        discarded = 0\n        for f in open_findings:\n            if f.handoff_queued_at is not None:\n                f.handoff_queued_at = None\n                await finding_repo.update(f)\n                discarded += 1\n        return {\"merged\": False, \"discarded\": discarded}\n\n\nWhat changes: **an unmerged PR has zero downstream cost.** Experimental PRs, drafts, rejected PRs — however many findings they collect, not a single fix task fires unless the code reaches main. Reviews can run all they want, but handoff (the operation that spends money and creates new PRs) always passes through merge — a human (or an explicit gate) decision.\n\n##  Cut 2: PRs authored by a bot are excluded from auto-handoff\n\nThis is the direct break in the loop. Fix PRs are opened by `purple-codens[bot]` or `red-codens[bot]`. A PR authored by one of those bots is excluded from auto-handoff.\n\n\n\n    _CODENS_BOT_LOGINS = {\"purple-codens[bot]\", \"red-codens[bot]\", \"orange-codens[bot]\"}\n\n    def _should_handoff(self, finding, policy, is_bot_pr):\n        # a human-queued finding dispatches regardless of mode/bot\n        if finding.handoff_queued_at is not None:\n            return True\n        if is_bot_pr:\n            return False  # auto-dispatching on a bot PR is the loop\n        if policy.mode == HandoffMode.FULL_AUTO:\n            return True\n        if policy.mode == HandoffMode.THRESHOLD_AUTO:\n            threshold = policy.auto_threshold_severity or \"high\"\n            meets_severity = finding.severity.rank >= Severity(threshold).rank\n            in_category = (\n                not policy.auto_categories or finding.category.value in policy.auto_categories\n            )\n            return meets_severity and in_category\n        return False\n\n\nOrange still _reviews_ a bot's fix PR (we want to confirm quality), but it does **not auto-spawn the next fix task** from it. That's the fundamental cut.\n\nNote the first branch. A `handoff_queued_at` finding (one a human manually queued with \"fix this\") dispatches even on a bot PR, even when mode is off. **Explicit human intent overrides the loop guard.** Automatic chaining is stopped, but a human saying \"this particular finding on this bot PR is worth fixing\" gets through. Automatic vs. manual cleanly splits the safe default from the escape hatch.\n\n##  Cut 3: findings from verify runs aren't handed off\n\nClosing another loop path. Purple runs its own verify cycle (implement → test → fix). Findings from Orange reviewing those runs are excluded from handoff.\n\n\n\n    # exclude findings from purple_verify runs (loop guard)\n    purple_verify_run_ids = {\n        r.review_run_id for r in runs if r.triggered_by == TriggeredBy.PURPLE_VERIFY\n    }\n\n    eligible = [\n        f for f in open_findings\n        if f.review_run_id not in purple_verify_run_ids\n        and f.handed_off_task_id is None          # no double-dispatch (idempotency)\n        and self._should_handoff(f, policy, is_bot_pr)\n    ]\n\n\n`handed_off_task_id is None` matters too: a finding handed off once is never dispatched again. It prevents the quiet divergence of two fix tasks spawning for the same problem.\n\n##  Incident 1: findings in the same file deadlocked each other's fix PRs\n\nNow the part where it was built to spec and still failed to converge.\n\nIn an E2E run, `app/api/notes/search.py` had 3 findings (SQL injection, a hardcoded AWS key, and one more). Dispatched naively as \"one task per finding,\" each opened a separate fix PR.\n\nHere's the trap. When the SQLi fix PR tries to merge, Orange reviews it and raises a carry-over REQUEST_CHANGES: \"the AWS-key finding in the same file is still unresolved.\" The AWS-key fix PR is blocked for the same reason as long as the SQLi remains. **All three fix PRs became unmergeable, each blocked by the others' unfixed findings.**\n\nThe fix: at handoff time, coalesce findings of \"same file × same target\" into one task.\n\n\n\n    # coalesce by (target_codens, file_path). Multiple findings in the same file\n    # going to the same service become ONE task so the fix PR resolves them\n    # together. Otherwise each finding's fix PR is blocked by orange's carry-over\n    # review of the other unfixed findings in that file.\n    groups: dict[tuple, list[Finding]] = {}\n    singles: list[Finding] = []\n    for f in eligible:\n        target = self._target_for(f, policy)\n        if target == TargetCodens.PURPLE and f.file_path:\n            groups.setdefault((target, f.file_path), []).append(f)\n        else:\n            singles.append(f)\n\n\nFix one file's problems together, in one PR. Obvious in hindsight, but \"parallelize naively per finding\" puts the reviewer (Orange itself) in a position to block its own fixes with its own other findings — a self-deadlock. Get the unit of parallelism wrong and review and fix jam against each other.\n\n##  Incident 2: new findings on new code keep the review from converging\n\nAnother one. When Orange reviewed a bot's fix PR, it kept raising new high-severity findings on the newly written code each round (missing tests, style, etc.) — 4 consecutive REQUEST_CHANGES rounds, then escalation at the cap (max_review_iterations=5). Non-convergence.\n\nThe fix changed the decisive-review logic for bot fix PRs:\n\n  * carry-over (unresolved findings from the previous review) at severity ≥ high → block (confirm they were actually fixed)\n  * new findings block **only if they're blockers** (the fix introduced a new critical problem)\n  * new findings below critical are recorded inline / in the body but the PR is APPROVED\n\n\n\nBehavior on human PRs (COMMENT only, never auto-block) is unchanged. Only for bot fix PRs did we make the judgment favor convergence.\n\nThe point: **a perfectionist reviewer never converges.** If you raise a new minor finding at full marks on every round, the PR never closes. Anchor on \"was the previous serious finding resolved\", record new minor ones but let them through. Designing the review bar so the review can stop.\n\n##  Takeaway: termination is guaranteed by the architecture, not the model\n\nWhen AI is both author and reviewer, the architecture's job is to guarantee termination and bound cost. No amount of model intelligence guarantees that. The wiring does.\n\nThe five cuts in Orange:\n\n  * Handoff fires at merge, not at finding-time (unmerged PRs have zero downstream cost)\n  * Bot-authored PRs are excluded from auto-handoff (the direct loop cut)\n  * Findings from verify runs aren't handed off\n  * A handed-off finding is never dispatched twice (idempotency)\n  * A human-queued finding crosses the guard (escape hatch)\n\n\n\nAnd two we added after hitting them:\n\n  * Findings in the same file coalesce into one task (avoid self-deadlock)\n  * Bot fix PR review blocks only on carry-over high + new blockers (favor convergence)\n\n\n\nRun \"AI writes code\" in production and you always hit \"who reviews it.\" If you let AI review it too, you have to design the stopping mechanism as part of the deal — otherwise it fixes forever, the cost diverges, or it deadlocks itself.\n\nOrange Codens builds all of this into the product.\n\nhttps://www.codens.ai/en/",
  "title": "\"When AI reviews AI's code, you've built an infinite loop. Here's how we stopped it.\""
}