Potential problem with junit platform and maven surefire 'rerunFailingTestsCount'

[shandy1987]

Hello,

I have a ui/api automated test suite using java/selenium/cucumber.

Maven surefire plugin 3.5.0 executes the tests.

Is anybody aware of an issue where if you are running:
junit platform greater than 1.10.2 (it works on 1.10.2)

Under maven surefire when rerunFailingTestsCount is invoked after your initial cucumber test run, it is rerunning all your tests again and not just the failed ones?

[mpkorstanje]

Exactly which Cucumber and JUnit dependencies and versions are you using? Can you show us a reproducer by forking https://github.com/cucumber/cucumber-java-skeleton?

[shandy1987]

Hello I'm using:

cucumber-java
cucumber-junit-platform-engine
cucumber-picocontainer
cucumber-junit

They are all version 7.19.0

My maven surefire / cucumber configuration parameters are:

<configuration>
  <parallel>methods</parallel>
  <rerunFailingTestsCount>2</rerunFailingTestsCount>
  <perCoreThreadCount>false</perCoreThreadCount>
  <argLine>-Xmx1024m</argLine>
  <properties>
        <configurationParameters>
            cucumber.publish.quiet=true
            cucumber.execution.parallel.enabled=true
            cucumber.execution.parallel.config.strategy=fixed
            cucumber.execution.parallel.config.fixed.parallelism=${threadCount}
            cucumber.execution.parallel.config.fixed.max-pool-size=${threadCount}
            <!--ignore unresolved maven property threadCount-->
            cucumber.plugin=pretty,html:target/cucumber-reports/testsLastRun.html,json:target/cucumber-reports/testsLastRun.json
        </configurationParameters>
  </properties>
</configuration>

My junit platform dependencies are:

junit-platform-suite
junit-platform-console

They are currently version 1.10.2, as anything greater than this recreates the problem.

The other junit dependency is:

junit-jupiter

version 5.11.0

I'm executing my tests with:

mvn clean test -DthreadCount=3 -Dcucumber.filter.tags="@test"

Will look into fork when I get some spare time.

Thanks

[mpkorstanje]

Can't reproduce the problem based on what you've provided. So a minimal reproducer would be appreciated.

But the reproduction conditions look suspiciously like a variation of cucumber/cucumber-jvm#2911.
So you could use cucumber.junit-platform.naming-strategy=long and make sure all features names are unique, and within a feature that all scenario names are unique. If that solves the problem it is somewhere in Maven.

[shandy1987]

Just tried cucumber.junit-platform.naming-strategy=long and still getting the same issue with the later versions of junit-platform. Will look into fork and try to recreate with minimal setup.

[shandy1987]

Here is a basic repo to clone and recreate the problem. Make sure you have chrome version 129.

https://github.com/shandy1987/cucumber-java-skeleton2

[mpkorstanje]

You've used

@ConfigurationParameter(key = FEATURES_PROPERTY_NAME, value = "src/test/resources/featureFiles")`

This results in a warning

Sep 26, 2024 9:40:56 PM io.cucumber.junit.platform.engine.DiscoverySelectorResolver warnWhenCucumberFeaturesPropertyIsUsed
WARNING: Discovering tests using the cucumber.features property. Other discovery selectors are ignored!

This is a work around for the limited JUnit 5 support in Maven and Gradle. Please request/upvote/sponsor/ect better support for JUnit 5 discovery selectors. For details see: https://github.com/cucumber/cucumber-jvm/pull/2498

If you are using the JUnit 5 Suite Engine, Platform Launcher API or Console Launcher you should not use this property. Please consult the JUnit 5 documentation on test selection.

You should use the discovery selector annotations from JUnit 5 instead. For the possible options see: org.junit.platform.suite.api.

Additionally @SelectClasspathResource("javaFiles") seems out of place. You probably meant to use:

@Suite
@IncludeEngines("cucumber")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "javaFiles")
@SelectPackages("featureFiles")

This wasn't a problem until you upgraded and pulled in the fix for #3928. Prior to that, the @ConfigurationParameter annotations were ignored on a rerun.

[mpkorstanje]

You are also mixing different versions of JUnit Jupiter 5.11.0 and the JUnit Platform 1.10.2. Their minor and patch versions should stay in sync.

To avoid any problems from this all together you might also want to consider using the bill of materials to manage the versions of JUnit and Cucumber.

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.junit</groupId>
                <artifactId>junit-bom</artifactId>
                <version>5.11.1</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>io.cucumber</groupId>
                <artifactId>cucumber-bom</artifactId>
                <version>7.19.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

This means you can omit all version from every org.junit and every io.cucumber dependency.

[mpkorstanje]

Out of curiosity, could you tell me where you got the suggestion from to use @ConfigurationParameter(key = FEATURES_PROPERTY_NAME, value = "src/test/resources/featureFiles") ?

[shandy1987]

Thank you, I will make all of them changes tomorrow. I can’t quite remember where I got that configuration parameter from. It was a while ago now.

[shandy1987]

All changes made and working. Thanks again!