{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiamnu5isqi7xmxeadtrke3or7tzwqk63qaluye6pbhsp4kpe4z65a",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpmpx3ywfmz2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreidzromiweqrmn7xipodzxya3ipi5bth37yznoysl73xzdeckgfzzy"
    },
    "mimeType": "image/webp",
    "size": 79748
  },
  "path": "/zoetaka38/our-ai-autopilot-merged-nothing-for-3-days-the-culprit-was-a-review-freshness-check-1240",
  "publishedAt": "2026-07-01T23:15:59.000Z",
  "site": "https://dev.to",
  "tags": [
    "ai",
    "devops",
    "github",
    "programming"
  ],
  "textContent": "At Codens, we run our development on AI agents around the clock: a PRD becomes tasks, agents implement them and open PRs, an AI reviewer approves, and approved PRs merge automatically. Humans mostly read the reviewed output and the notifications. That was the idea — until one morning the dashboard showed **zero completed tasks for three days straight**.\n\n\"Autopilot is down.\"\n\nExcept it wasn't. Every component checked out healthy. Plans were being generated. Tasks were being submitted. Agents were opening PRs. The AI reviewer had **already approved** them. CI was green. The only thing that never happened was the merge.\n\nThe culprit was a **freshness check** in the code that polls PR reviews — a defensive check we had added ourselves, to prevent a different incident from recurring. This post is the story of how that played out, plus a lesson I think generalizes well beyond AI agents: **do you treat external state as events, or as a snapshot?**\n\n##  Background: the wait_review step\n\nOur workflow engine drives each task through fixed steps like `implement → create_pr → wait_ci → wait_review → merge_pr`. We learned early on that letting the AI decide \"what to do next\" makes operations impossible to reason about, so transitions are hard-coded in the engine.\n\n`wait_review` is a rendezvous point: it polls GitHub PR reviews and advances once approvals are in. Inside it, there was this check:\n\n\n\n    # simplified\n    fresh_reviews = [\n        r for r in pr_reviews\n        if r.submitted_at > workflow.review_initiated_at\n    ]\n    if any(r.state == \"APPROVED\" for r in fresh_reviews):\n        advance_to_merge()\n\n\nOnly look at reviews **newer** than `review_initiated_at`. Seems reasonable.\n\n##  Why the freshness check existed\n\nIt was added to prevent a real incident:\n\n  1. A reviewer APPROVES\n  2. The agent pushes follow-up commits\n  3. The reviewer submits CHANGES_REQUESTED\n  4. The poll picks up the **stale APPROVE** and merges anyway\n\n\n\nMerging on an outdated approval after changes were requested is clearly bad, so we added \"only trust reviews newer than when we started waiting.\" At the time, this was the correct fix.\n\n##  Re-arming kills standing approvals\n\nThe problem was **when** `review_initiated_at` gets updated.\n\nThe engine has a recovery path for Spot instance reclamation and process restarts: it picks up interrupted workflows and **re-enters** the current step. Re-entering `wait_review` **re-armed`review_initiated_at` to the current time** — regardless of whether any new commits existed.\n\nWhich makes this timeline possible:\n\n\n\n    10:00  Reviewer APPROVES (submitted_at = 10:00)\n    10:30  Spot reclamation → workflow recovery re-enters the step\n           → review_initiated_at re-armed to 10:30\n    10:31  Poll: submitted_at(10:00) > review_initiated_at(10:30) is false\n           → \"no new reviews\"\n    Since the reviewer has already approved, no new review\n    will ever arrive → the workflow waits forever\n\n\nThe approval **still exists on GitHub** , plain as day — a standing approval. But the filter's reference time moved past it, so the engine forever concludes \"no review yet.\" A perfect deadlock.\n\nThe nasty part: it's **probabilistic**. If recovery re-entry happens before the approval, nothing goes wrong. Only workflows where re-entry lands after the approval silently freeze. On Spot-heavy infrastructure, re-entries are frequent enough that these accumulated over days, and only surfaced as \"zero dones.\"\n\n##  How we actually diagnosed it\n\nGetting from \"Autopilot is down\" to the root cause came down to one habit: **don't look at the endpoint, look at the distribution.**\n\n  * Watching only the done count (the endpoint), everything looks uniformly stuck\n  * Aggregating in-flight workflows by step showed **28 abnormally parked at`wait_review`**\n  * Cross-referencing their PRs: **all approved, CI green, unmerged**\n\n\n\nAt that point \"the engine can't see approvals\" is established, and the rest is just reading the poll code. Counting _where the bodies pile up_ beats hunting for a dead component — because there was no dead component. A live process faithfully executing a wrong rule will never show up on a health check.\n\n##  The fix: evaluate a snapshot, not events\n\nThe essence of the fix fits in one sentence: instead of asking **\"did a new review arrive?\"** , compute **\"what is the effective review state right now?\"** on every poll.\n\n\n\n    # keep only each reviewer's latest review\n    latest_by_reviewer = {}\n    for r in sorted(pr_reviews, key=lambda r: r.submitted_at):\n        latest_by_reviewer[r.user.login] = r\n\n    effective_states = [r.state for r in latest_by_reviewer.values()]\n\n    if \"CHANGES_REQUESTED\" in effective_states:\n        hold()\n    elif \"APPROVED\" in effective_states:\n        advance_to_merge()\n\n\nThis is literally the semantics GitHub's own UI uses: the review panel shows each reviewer's _latest_ state, and _when_ the approval happened plays no role in the effective verdict. Which lands on an unglamorous but solid conclusion: **if you gate on an external system's state, adopt that system's own semantics.**\n\nWhat about the original incident (merging on a stale approval after changes were requested)? Per-reviewer-latest handles it for free: the reviewer who requested changes _is_ in CHANGES_REQUESTED state, so an older APPROVE can never win. The freshness check survives only in its legitimate, narrow role — debouncing re-review after the same reviewer requested changes and new commits landed.\n\nAfter deploying, the `wait_review` backlog dropped from 28 to 8, and **15 tasks completed in 25 minutes** — three days of approved PRs draining the moment the predicate was fixed.\n\n##  Wave two: failures must propagate\n\nWe thought that was the end. The next day, the same symptom came back — different cause: **dependency chains**.\n\nSome tasks had been marked failed at `wait_review` during the bug window, and their **downstream tasks** sat blocked on those dependencies. The auto-recovery job only did \"move blocked back to pending,\" while the scheduler's start condition remained \"all dependencies done\" — so tasks with a failed ancestor bounced between pending and blocked forever. A no-op retry loop.\n\nThe fix was to go fail-fast: when a dependency fails, don't leave descendants ambiguously blocked — **explicitly mark them FAILED and propagate it downstream**. A visible failure gets fixed with one action (\"retry the failed ancestor\"). An invisible blocked task never gets retried, because nobody knows it's waiting.\n\n##  Takeaways\n\nThese generalize to any system that polls external state at a rendezvous point, AI agents or not:\n\n  1. **Evaluate external state as a snapshot, not as events.** Gating on \"anything newer than time T?\" means one mistake in managing T permanently drops facts that are already true (standing approvals). Recomputing \"the effective state right now\" is idempotent and recovery-proof.\n  2. **Re-arming a timestamp means erasing every fact before it.** If retry/recovery/re-entry paths casually set `*_initiated_at = now()`, everything established before that instant vanishes behind the filter. Re-arm only when the premise actually changed — e.g., new commits landed.\n  3. **Yesterday's safeguard is today's prime suspect.** The freshness check was a correct response to a real incident — its scope was just too broad. When you add a defense, also write down _what stops working if this defense misfires_ ; you'll check it first next time something stalls.\n  4. **Diagnose \"it's stuck\" by step-level backlog distribution, not endpoint metrics.** All-green health checks can't detect a live process that's wrong on purpose. Counting how many items are parked at each rendezvous point finds it in minutes.\n  5. **Propagate failures.** A dependency failure left as \"blocked\" is an unbounded silent wait. Fail-fast propagation turns recovery into a single retry.\n\n\n\nIf your autopilot includes auto-merge, \"three days of zero dones\" isn't an inconvenience — it's a production outage for your development line. Don't trust the green health checks; put rendezvous-point backlogs on your dashboard. That's the monitoring this incident permanently added to ours.",
  "title": "Our AI autopilot merged nothing for 3 days — the culprit was a review 'freshness' check"
}