Don't fail silently when running a junit4 test in junit5 test suite w/o vintage

[JohnnyMorganz]

I am recently working on migrating a project from Maven to Bazel. As part of this, it has most modules which use junit4, but some which use junit5, and this ruleset really helps for the latter (thank you!).

However, we do have some modules which used both junit4 and junit5 classes (which was possible with maven-surefire-plugin). When migrating to Bazel, we use the junit5_test_suite rule, and we can either update all these classes to use JUnit 5, or use the vintage runner [which is all fine :)]

But, my main problem comes when using the junit5_test_suite but forgetting to also set up the vintage runner. Somewhat surprisingly, neither junit5 nor the bazel test runner fails in this case, and instead silently passes (with 0 test cases run). This seems like a potential concern, if developers assume their test is running but actually it wasn't but just passed anyway.

I did see that there is actually an open issue in junit5 to better report errors for invalid setups (junit-team/junit5#1223 and junit-team/junit5#242), but unfortunately there doesn't seem to be much movement there. I was curious if this was something detect-able via the the bazel test runner. Something along the lines of how bazel errors with "no test targets found, but testing was requested", except for the junit5 test runner instead.

Totally reasonable if not possible, but was interested to know.
Thanks for your time!

[illicitonion]

Hello! Totally reasonable feature request - I'm not sure concretely how we'd go about this; I guess it would involve re-implementing some of the class scanning to try to detect junit4 @Test annotations in the case that no tests were run? Which sounds pretty fragile and fiddly... If there's a way we can do this re-using existing upstream components, it seems very reasonable to do; if not, if it doesn't make the code much more complicated I'd happily review a PR for this.