Best way to run tests on file save

ghcid is indeed a good way of doing this; the --test argument can be used to run arbitrary GHCi commands after (re-)loading.

The way I like to structure things:

• All the “real” code lives in a library (myproject:lib:myproject)
• The “application” (myproject:exe:myproject) is just a thin wrapper that says something like main = MyProject.Main.main
• The testsuite has its own main
• An “outer” ghcid job loads and watches the libary part, reloading as usual, and restarting (--restart) whenever any of the files change that ghci itself doesn’t watch (myproject.cabal, cabal.project, etc.).
• The “outer” ghcid then runs (via --test) an “inner” ghcid that loads and watches the testsuite: --test ':! ghcid --test Main.main -c cabal repl myproject:test:myproject-tests'

Put together, it looks something like this:

#!/bin/sh
ghcid \
    --restart myproject.cabal \
    --restart cabal.project \
    --test ':! ghcid --test "Main.main" -c cabal repl myproject:test:myproject-tests' \
    -c 'cabal repl myproject:lib:myproject'

Depending on which files you change, this will (usually) do the right thing:

• If any of your test source files change, it will reload the testsuite (but not the library) and re-run tests. This is very fast.
• If any of the library sources change, it will reload the library, restart the inner ghci, load the testsuite, and re-run tests. This is still pretty fast: the testsuite will be restarted, but the library is reloaded, and typically none of the testsuite itself has changed, so the restart can use cached build artifacts.
• If cabal.project or myproject.cabal changes, it restarts everything, loads the library, then the testsuite, then runs the tests. This will take a bit longer (similar to plain cabal test), but since you don’t change those files frequently, that’s OK (and you can’t really avoid it anyway).

To restrict which tests are run, I use environment variables - I usually use tasty, which looks at environment variables such as TASTY_PATTERN to determine which tests to run. You can also use environment variables to change other testing options, such as the size and number of quickcheck tests, shrinking steps, success/failure reporting, etc.

With the above setup, this requires restarting the ghcid job to pick up those changed env vars; you can expand the pattern to use an env file, restarting the outer ghcid when that file changes, and re-sourcing the env file before restarting the inner ghcid, but I usually don’t bother, because I don’t really change those test patterns a lot.