Second time coverage gives 0% coverage

[olatheander]

I noticed a peculiar behavior; I fire up sbt from the shell and the first time I collect the coverage with clean;coverage;test;coverageAggregate I get about 37% coverage on my example project but if I run it again immediately after I get 0% and also for all consecutive runs. I obviously expected to get the same coverage every time.

If I exit sbt and run coverage again I get the same behavior; first run gives correct coverage but consecutive runs give 0%.

My test project can be found here: https://github.com/olatheander/scala-code-coverage-test

It seems to be a Scala 3 issue, I created an equivalent Scala 2 project and then it worked as expected.

Below is the output of the first run followed by a second run:

sbt:Scala Code Coverage Test> clean;coverage;test;coverageAggregate
[success] Total time: 0 s, completed 18 nov. 2023 13:38:41
[info] Defining ThisBuild / coverageEnabled
[info] The new value will be used by Compile / compile / scalacOptions, libraryDependencies
[info] Reapplying settings...
[info] set current project to Scala Code Coverage Test (in build file:/home/olathe/Git/scala-code-coverage-test/)
[info] compiling 1 Scala source to /home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/classes ...
[info] compiling 1 Scala source to /home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/test-classes ...
com.example.MySuite:
  + Person test 0.036s
  + Person test 2 0.0s
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2
[success] Total time: 4 s, completed 18 nov. 2023 13:38:46
[info] Aggregating coverage from subprojects...
[info] Found 1 subproject scoverage data directories [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-data]
[info] Generating scoverage reports...
[info] Written Cobertura report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/coverage-report/cobertura.xml]
[info] Written XML coverage report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-report/scoverage.xml]
[info] Written HTML coverage report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-report/index.html]
[info] Statement coverage.: 37.50%
[info] Branch coverage....: 100.00%
[info] Coverage reports completed
[info] Aggregation complete. Coverage was [37.50]
[error] Coverage is below minimum [37.50% < 70.00%]: Stmt:Total
[info] All done. Coverage was stmt=[37.50%] branch=[100.00%]
[success] Total time: 0 s, completed 18 nov. 2023 13:38:46
sbt:Scala Code Coverage Test> clean;coverage;test;coverageAggregate
[success] Total time: 0 s, completed 18 nov. 2023 13:38:48
[info] Defining ThisBuild / coverageEnabled
[info] The new value will be used by Compile / compile / scalacOptions, libraryDependencies
[info] Reapplying settings...
[info] set current project to Scala Code Coverage Test (in build file:/home/olathe/Git/scala-code-coverage-test/)
[info] compiling 1 Scala source to /home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/classes ...
[info] compiling 1 Scala source to /home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/test-classes ...
com.example.MySuite:
  + Person test 0.004s
  + Person test 2 0.0s
[info] Passed: Total 2, Failed 0, Errors 0, Passed 2
[success] Total time: 1 s, completed 18 nov. 2023 13:38:49
[info] Aggregating coverage from subprojects...
[info] Found 1 subproject scoverage data directories [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-data]
[info] Generating scoverage reports...
[info] Written Cobertura report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/coverage-report/cobertura.xml]
[info] Written XML coverage report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-report/scoverage.xml]
[info] Written HTML coverage report [/home/olathe/Git/scala-code-coverage-test/target/scala-3.3.1/scoverage-report/index.html]
[info] Statement coverage.: 0.00%
[info] Branch coverage....: 0.00%
[info] Coverage reports completed
[info] Aggregation complete. Coverage was [0.00]
[error] Coverage is below minimum [0.00% < 70.00%]: Branch:Total
[info] All done. Coverage was stmt=[0.00%] branch=[0.00%]
[success] Total time: 0 s, completed 18 nov. 2023 13:38:49
sbt:Scala Code Coverage Test>

Scala version: 3.3.1
sbt-coverage version: 2.0.9

[olatheander]

After investigating a bit more; after doing a coverage;test after sbt startup there is scoverage.measurements.**** file created e.g:

$ cd target/scala-3.3.1
$ tree scoverage-data/                                                                                                                                                                                                                                                                                                                           scoverage-data/
├── scoverage.coverage
└── scoverage.measurements.58cb6c3d-238f-4782-90d9-f698d1b8bf19.187

but after a clean and consecutive coverage;test this file is not created anymore, only the scoverage.coverage file is recreated. Without the measurements file, the coverage is reported to be 0%.

I verified this by hard-coding the measurements value in loadCoverage in which I added a log statement to collect the values:

  private def loadCoverage(
      crossTarget: File,
      log: Logger,
      sourceRoot: File
  ): Option[Coverage] = {
     ...
     ..
      log.info(s"Reading scoverage measurements...")
      val measurementFiles = IOUtils.findMeasurementFiles(dataDir)
      // val measurements = IOUtils.invoked(measurementFiles)
      val measurements = Set((0, ""), (2, ""), (1, ""))
      log.info(s"measurements: ${measurements}")
      coverage.apply(measurements)
      Some(coverage)

    } else {
      None
    }
  }

With the hard-coded Set() the reported coverage was 37.50% every time despite the measurements file was missing (obviously since I hard-coded the values).

However, this makes me believe that perhaps it's more of a dotty issue than a sbt-coverage issue?

[dillius]

I also have seen this issue with a recent little project trying with Scala 3.3.3, ZIO 2.0.19, and Scoverage 2.1.0.

On initial setup and run I was getting accurate coverage numbers, but after trying to add some exclusions I began to repeatedly get 0% coverage results. Reverted to original scoverage changeset, and still getting 0%.