{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiei75layzl6mula6g62vjlbwxbbivpn6ecafb6o23m2brrxv2tv7a",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mokjv37agvv2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreidhamslfoud2vpgt7ktktq53inmbedojgngya6m5v7hfxgpuqudcq"
    },
    "mimeType": "image/webp",
    "size": 74008
  },
  "path": "/kazutaka-dev/why-vitest-changed-misses-some-tests-and-how-runtime-coverage-fixes-it-jjm",
  "publishedAt": "2026-06-18T09:03:23.000Z",
  "site": "https://dev.to",
  "tags": [
    "javascript",
    "testing",
    "typescript",
    "node",
    "testpick",
    "https://github.com/TwistTheoryGames/testpick",
    "https://www.npmjs.com/package/testpick",
    "@vite-ignore"
  ],
  "textContent": "Your CI re-runs the entire test suite on every push. But a one-line change can\nonly break a handful of tests. The rest is wasted compute — and wasted minutes\nyou spend waiting.\n\nThe usual fix is \"run only the tests affected by my change.\" Vitest and Jest\nship this:\n\n\n\n    vitest --changed\n    jest --onlyChanged\n    jest --findRelatedTests <files>\n\n\nThese are great. But they share a blind spot, and that blind spot can make them\n**skip a test that your change actually breaks** — the worst possible failure\nfor a test selector. Let's look at exactly where they break, and a different\napproach that doesn't.\n\n##  How the built-ins decide\n\n`vitest --changed` (and `jest --findRelatedTests`) walk the **static import\ngraph**. They parse your modules, follow `import`/`require` statements, and find\nwhich test files transitively import the changed file.\n\nFor the common case this works beautifully. Vitest even uses Vite's module graph,\nso it's fast and precise — and it handles _literal_ dynamic imports too, because\nVite globs them:\n\n\n\n    // Vite resolves this fine — it globs ./*.js into the graph\n    const mod = await import(`./${name}.js`);\n\n\nSo far so good. The problem starts when the import target isn't statically\nanalyzable.\n\n##  The case they miss: a runtime-computed path\n\nPlenty of real code loads modules by a path that is **data** , not a literal:\nplugin registries, dependency injection, feature flags, anything table-driven.\n\n\n\n    // src/plugins/loader.ts\n    const REGISTRY: Record<string, string> = {\n      feat: \"../features/feat.ts\",\n    };\n\n    export async function load(name: string) {\n      const path = REGISTRY[name];               // the path is data\n      return import(/* @vite-ignore */ path);    // Vite can't analyze this\n    }\n\n\n\n    // src/loader.test.ts\n    import { load } from \"@/plugins/loader\";\n\n    test(\"dynamic feature\", async () => {\n      const m = await load(\"feat\");\n      expect(m.feat()).toBe(\"F\");                 // exercises features/feat.ts\n    });\n\n\nThere is **no static import edge** from `loader.test.ts` to `features/feat.ts`.\nThe only thing that connects them is what happens at runtime.\n\nNow change `features/feat.ts` and ask the built-ins what to run:\n\n\n\n    $ vitest related src/features/feat.ts --run\n    No test files found, exiting with code 0\n\n\n\n    $ jest --findRelatedTests src/features/feat.js\n    No tests found, exiting with code 1\n\n\nBoth say **\"nothing to run\"** — and they're wrong. `loader.test.ts` exercises\nthat file and would catch a regression. A selector you trust would have just let\na real breakage through.\n\n##  The fix: select from what actually ran\n\nStatic analysis answers \"what _could_ be imported.\" Coverage answers \"what _was_\nexecuted.\" If you record, per test file, which source files it actually touched\nat runtime, the registry/DI case stops being special — the edge shows up because\nthe code really ran.\n\nThat's the idea behind testpick,\na small CLI I built. Two commands:\n\n\n\n    npx testpick map     # learn the test → source map once (from runtime coverage)\n    npx testpick run     # from then on, run only what your diff can break\n\n\nOn the same change:\n\n\n\n    $ testpick run\n    testpick: 1/3 test file(s) affected by 1 change(s).\n     ✓ src/loader.test.ts\n\n\nIt picks `loader.test.ts` because, when the map was built, that test's run\nactually executed `features/feat.ts`. (This holds for Jest too — same example,\nsame result.)\n\n##  Building the map without it being slow\n\nThe obvious objection: collecting per-test coverage sounds expensive. The naive\nway — run each test file in its own process with coverage — pays the runner's\nstartup cost N times.\n\ntestpick does it **single-pass** : it shards the test files across your cores and\nruns each shard as _one_ serial runner process (`vitest run` /\n`jest --runInBand`), using V8 precise coverage. It snapshots cumulative coverage\nafter each test file and diffs it against the previous snapshot — whatever went\nup was exercised by that file.\n\nOn a 23-test fixture (8-core machine):\n\nStrategy | Wall-clock | Total CPU\n---|---|---\nprocess-per-file (parallel) | 4.9s | ~30s\nsingle-pass (sharded) | **2.7s** | **~16s**\n\n~1.8x faster wall-clock and ~4x less CPU — and the resulting map came out\n**identical** to the per-file method. On CI boxes with fewer cores, the\nsingle-pass win is bigger.\n\nThe map is also incremental: it hashes test files and only re-measures the ones\nthat changed, so refreshing it is nearly free.\n\n##  The part that matters most: never skip silently\n\nA test selector is only useful if you can trust it not to drop a test you needed.\ntestpick's rule is: **when in doubt, run more, never less.**\n\n  * A changed file the map doesn't know about (a brand-new file, a config) → it runs **everything** by default.\n  * `testpick explain` prints exactly why each test was selected or skipped — no black box.\n  * There's an optional `--ai` mode that asks an LLM to narrow down unmapped changes, but even it can only _add_ tests; if the model is unsure, it still falls back to running everything.\n\n\n\n##  Being honest: it's complementary, not \"better\"\n\nI don't want to oversell this. A coverage map and a static module graph catch\n_different_ things:\n\n  * **Coverage wins** on runtime-computed couplings (registry/DI) and on trimming imported-but-never-executed modules.\n  * **The static graph wins** when a change touches a branch your recorded run hasn't exercised yet — coverage hasn't \"seen\" it.\n\n\n\nThat asymmetry is exactly why the safety fallback above matters, and why testpick\nworks the same way for Jest, where there's no Vite graph to lean on.\n\n##  Monorepos\n\ntestpick detects workspaces (`workspaces` in package.json, or\n`pnpm-workspace.yaml`) and treats each package as its own unit — its own runner,\nits own map. A repo with a Vitest package and a Jest package both works:\n\n\n\n    $ testpick run\n    Monorepo: 2 package(s).\n    [packages/api]  3/41 test file(s) affected by 2 change(s).\n    [packages/web]  1/120 test file(s) affected by 1 change(s).\n\n\nChanging one package doesn't run another's tests. A change _outside_ every\npackage (root config, lockfile) is treated as potentially affecting everything,\nso it safely runs all packages.\n\n##  Try it\n\n\n    npx testpick map\n    npx testpick run\n\n    # in CI:\n    npx testpick run --base origin/main\n\n\n  * GitHub: https://github.com/TwistTheoryGames/testpick\n  * npm: https://www.npmjs.com/package/testpick\n\n\n\nIt's MIT, plain Node (no build step, no runtime deps), Vitest + Jest. It's early\n(v0.1), and I'd genuinely love feedback — especially numbers from real, large\nsuites and monorepos. If `vitest --changed` already covers your case perfectly,\ngreat; if you've ever been bitten by a dynamically-loaded module slipping through\nselection, this is for you.",
  "title": "Why `vitest --changed` misses some tests (and how runtime coverage fixes it)"
}