Retrofit JUnit 5 tags/JUnit 4 Categories to the JUnit Vintage Engine

[vasilmkd]

At JetBrains, we're facing an interesting problem. The IntelliJ IDEA test framework was historically written in JUnit 3, with some compatibility code for JUnit 4. For this reason, we are heavy users of https://github.com/sbt/junit-interface. We take full advantage of its ability to execute JUnit 3 and JUnit 4 tests. We also take advantage of its ability to specify JUnit 4 categories. We rely on this to split our unit and integration tests in the CI to run in parallel on multiple machines and reduce overall CI time.

The new IDEA test framework is provided in JUnit 5, so we'd like to transition our tests to take advantage of it. For that, we will need sbt-jupiter-interface. I already said in #74 that I attempted to use sbt-jupiter-interface in the Scala plugin sbt project, coupled with JUnit Vintage Engine to be able to run the old JUnit 3 and 4 tests without changes.

I think it has already been documented in other issues that sbt-junit-interface relies on the JUnit Platform for discovering and running tests. In this regard, it differs from junit-interface, which is a bespoke implementation of a "JUnit test runner", which had some interesting implementation details.

This has to do with JUnit 4 categories and how junit-interface is able to "retrofit" them to JUnit 3 tests. Because junit-interface is a bespoke implementation, it decides to honor categories during the test discovery and filtering process and apply categories to all discovered tests, regardless if they are JUnit 4 or JUnit 3 (which predates categories and isn't required to support them).

sbt-jupiter-interface on the other hand simply invokes the JUnit Platform, which knows how to run all JUnit engines. The JUnit 5 engine takes care of running JUnit 5 tests, while the JUnit Vintage Engine takes care of running JUnit 3 and 4 tests. However, the Vintage Engine only supports JUnit 4 categories for JUnit 4 tests, not for JUnit 3 tests, so this is a "regression", or rather, loss of existing functionality, when compared to junit-interface

Is there a possibility that sbt-jupiter-interface can build an additional filter that would retrofit JUnit 5 tags/JUnit 4 categories onto JUnit 3 tests? Or, alternatively, offer a way to provide additional test filters (sbt's class name filters here are insufficient).

As an aside, I also understand that JUnit 3 and 4 tests can simply be run with junit-interface and it can happily coexist with sbt-jupiter-interface in the same sbt project, but this doesn't quite work in practice, given that categories/tags need to be applied to the testOnly -- additional args command, where additional args are unique to each sbt test framework and it is a bit of a hassle to provide specific arguments to specific sbt test frameworks without resorting to modifying the build files, i.e. from the sbt console.

Let me know if you want me to go into more detail, or provide additional examples, or links to JUnit documentation.

Any thoughts or advice are very much appreciated. Thank you!