{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreibqrynfcbvxbixznwlrzxxb2leojphcedsxwxgwizluizhltiqlzm",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mppulozzmcq2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreidrt44vtwtwpf4aka6utmieguapax36dfso4bytbfpvia7iurkhsi"
    },
    "mimeType": "image/webp",
    "size": 59848
  },
  "path": "/gabrielholmesqa/how-to-know-which-tests-a-pr-will-break-before-ci-runs-44b8",
  "publishedAt": "2026-07-03T05:00:53.000Z",
  "site": "https://dev.to",
  "tags": [
    "testing",
    "qa",
    "automation",
    "devops",
    "testward.app/diff-scanner",
    "Testward"
  ],
  "textContent": "Every team has lived this: a pull request looks clean, gets approved, merges — and twenty minutes later the test suite is red. The change was in `src/`. The failure is in `tests/`. And nothing in the review surfaced the connection.\n\n##  Why code review misses it\n\nCode review shows you the diff. It does not show you what _depends on_ the diff. A reviewer reading a one-line selector rename has no way to know that three end-to-end specs select on that exact attribute — especially if those specs live in a folder they never open, or a repo they don't have checked out.\n\nSo the breakage is discovered by the most expensive possible detector: a full CI run, after merge, by whoever is on call for the red build. The person with the most context — the author, mid-review — never saw it coming.\n\n##  The three ways teams try to close the gap\n\n**1. Run the whole suite on every PR.** Correct, but slow and expensive, and it tells you _after_ the fact — you still wait for the red. On large suites it's minutes-to-hours of feedback latency per push.\n\n**2. Coverage-based test selection.** Map code to the tests that execute it and run only those. Powerful for unit tests, but it needs instrumentation, breaks down for E2E (where the \"coverage\" is a running browser), and says nothing across repo boundaries.\n\n**3. Static impact analysis at review time.** Read the diff, extract what it changes that tests depend on — selectors, routes, ids, visible text — and match those against the test files. This is cheap, needs no instrumentation, and runs the moment the PR opens.\n\n##  What \"what tests depend on\" actually means\n\nEnd-to-end and integration tests are coupled to the app through a small, identifiable set of anchors:\n\n  * **Selectors / test ids** — `data-testid`, `data-cy`, `getByRole`, CSS, XPath\n  * **Routes** — URL literals the test navigates to or asserts on\n  * **Visible text & labels** — button copy, headings, aria-labels the test queries\n  * **Element ids and names** — the locators page objects hang off\n\n\n\nIf a PR changes one of these, any test referencing it is a break candidate. That's a tractable matching problem you can run in seconds — no test execution required.\n\nWant to see it on a real diff? I built a free scanner that runs this extraction entirely in your browser (nothing gets uploaded): testward.app/diff-scanner\n\n##  The cross-repo blind spot\n\nHere's where it gets interesting for QA teams specifically. Many of us keep automation in a **dedicated repo** — separate from the app it tests. Good hygiene, but it severs the only signal: the frontend dev's PR is green (no E2E tests live there), and the automation repo goes red hours later with no link back to the cause.\n\nCoverage tools, affected-test selectors, and CI test-splitting all operate within a single repo and a single test run. None of them can say \"a change _here_ breaks a test _there_.\"\n\n##  How I automated this\n\nI'm a QA automation engineer, and this loop ate enough of my afternoons that I built a GitHub App for it — Testward. On every PR it:\n\n  1. Reads the diff and extracts the anchors the change touches.\n  2. Scans your test files — same repo, or a separately-linked automation repo (one line of config).\n  3. Runs an LLM confirmation pass that names the specific specs likely to break and why.\n  4. Posts one sticky comment: risk level, affected specs, reasons.\n\n\n\nThe reviewer sees the consequence next to the cause, while the author still has the context to fix it — or update the test in the same PR.\n\nKnown limitation worth being upfront about: dynamic selectors (``row-${id}``) can't be matched statically — that's the gap I'm working on. If your suite leans on template-literal test ids, the anchor model will miss those.\n\n##  Takeaway\n\nYou don't need to run tests to know a PR endangers them. The dependency between app code and test code is mostly _textual_ — selectors, routes, labels — and you can check it at review time, when the fix is cheapest. Whether you script it yourself or use a tool, moving that signal left is one of the highest-leverage things a QA team can do.\n\n_Questions or war stories about PRs silently breaking your suite? I'd genuinely like to hear them — especially from teams whose automation lives in a separate repo._",
  "title": "How to know which tests a PR will break — before CI runs"
}