> 6 minutes is spent skipping tests in CLI (skipping individual tests takes a long time)

[mpickering]

If you look at a CI log you will see for the lib-suite test section:

480 tests, 362 skipped, 0 unexpected passes, 0 unexpected fails.

Then if you look at a skipped test, they take > 1s to conclude to skip.

PackageTests\UserConfig\cabal.test.hs                                                                             SKIP no cabal-install (1.24s)

So if 360 tests each take 1 second to skip, that is 6 minutes wasted.

Some possible improvements:

• Investigate whether running the test can be made faster
• Don't separately run lib-suite and cli-suite, so less tests are skipped each time.

[ulysses4ever]

Exact timing depends on the platform I guess, because on Ubuntu runners I see more like ~30 sec to skip, so it's more like 3 minutes rather than 6 there, but that doesn't matter: the whole approach does feel wasteful and could be improved.

Merging jobs feels a little wrong because at one point people were quite excited about splitting, I think. I don't recall the actual arguments for it, but I'm a little afraid of undoing that work.

[phadej]

actual arguments for it,

Different support windows for Cabal the lib and cabal-install. IMO the test-suite is the only thing keeping the development together, they are quite independent components (and would be better if they were more independent).
(EDIT: e.g. stack is developed independently of Cabal, but it does rely on Cabal - it's "just" a lib).

The shared part as far as I remember is using cabal act-as-setup and ./Setup to run the same stuff.

OTOH, the true cabal-install tests (using v2-build etc) are so different, that might be an argument towards splitting the test-suites apart.

[jasagredo]

I didn't check the exact timings now, but I hope I made (#10186) this faster by skipping before setting up the tests environment. I think it should particularly benefit Windows.