analyzer: Empty binaryRemoteArtifact for Gradle project's dependency causes duplicate package error and prohibits result file creation

[devcooch]

1. I use Maven and Gradle package managers together (e.g. in config: enabledPackageManagers: [Maven, Gradle])
2. both Maven and Gradle projects are using same dependency package
3. I run analyzer on the folder with these two project

Something causes binaryArtifact to be empty in Gradle project, which in turn causes two entries in Set where only binaryArtifact differs (empty and non-empty).
This in turn creates duplicate packages error, e.g.:

Exception in thread "main" java.lang.IllegalArgumentException: Unable to create the AnalyzerResult as it contains packages and projects with the same ids: [[Package(id=Identifier(type=Maven, namespace=org.hamcrest, name=hamcrest-core, version=1.3), purl=pkg:maven/org.hamcrest/[email protected], cpe=null, authors=[Joe Walnes, Nat Pryce, Steve Freeman, Neil Dunn, Tom Denley], declaredLicenses=[New BSD License], declaredLicensesProcessed=ProcessedDeclaredLicense(spdxExpression=BSD-3-Clause, mapped={New BSD License=BSD-3-Clause}, unmapped=[]), concludedLicense=null, description=This is the core API of hamcrest matcher framework to be used by third-party framework providers. This includes the a foundation set of matcher implementations for common operations., homepageUrl=https://github.com/hamcrest/JavaHamcrest/hamcrest-core, binaryArtifact=RemoteArtifact(url=https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3.jar, hash=Hash(value=42a25dc3219429f0e5d060061f71acb49bf010a0, algorithm=SHA-1)), sourceArtifact=RemoteArtifact(url=https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3-sources.jar, hash=Hash(value=1dc37250fbc78e23a65a67fbbaf71d2e9cbc3c0b, algorithm=SHA-1)), vcs=VcsInfo(type=Git, [email protected]:hamcrest/JavaHamcrest.git, revision=, path=), vcsProcessed=VcsInfo(type=Git, url=ssh://[email protected]/hamcrest/JavaHamcrest.git, revision=, path=), isMetadataOnly=false, isModified=false), Package(id=Identifier(type=Maven, namespace=org.hamcrest, name=hamcrest-core, version=1.3), purl=pkg:maven/org.hamcrest/[email protected], cpe=null, authors=[Joe Walnes, Nat Pryce, Steve Freeman, Neil Dunn, Tom Denley], declaredLicenses=[New BSD License], declaredLicensesProcessed=ProcessedDeclaredLicense(spdxExpression=BSD-3-Clause, mapped={New BSD License=BSD-3-Clause}, unmapped=[]), concludedLicense=null, description=This is the core API of hamcrest matcher framework to be used by third-party framework providers. This includes the a foundation set of matcher implementations for common operations., homepageUrl=https://github.com/hamcrest/JavaHamcrest/hamcrest-core, binaryArtifact=RemoteArtifact(url=, hash=Hash(value=, algorithm=)), sourceArtifact=RemoteArtifact(url=https://repo.maven.apache.org/maven2/org/hamcrest/hamcrest-core/1.3/hamcrest-core-1.3-sources.jar, hash=Hash(value=1dc37250fbc78e23a65a67fbbaf71d2e9cbc3c0b, algorithm=SHA-1)), vcs=VcsInfo(type=Git, [email protected]:hamcrest/JavaHamcrest.git, revision=, path=), vcsProcessed=VcsInfo(type=Git, url=ssh://[email protected]/hamcrest/JavaHamcrest.git, revision=, path=), isMetadataOnly=false, isModified=false)], [Package(id=Identifier(type=Maven, namespace=net.bytebuddy, name=byte-buddy, version=1.9.3), purl=pkg:maven/net.bytebuddy/[email protected], cpe=null, authors=[Rafael Winterhalter], declaredLicenses=[The Apache Software License, Version 2.0], declaredLicensesProcessed=ProcessedDeclaredLicense(spdxExpression=Apache-2.0, mapped={The Apache Software License, Version 2.0=Apache-2.0}, unmapped=[]), concludedLicense=null, description=Byte Buddy is a Java library for creating Java classes at run time.
        This artifact is a build of Byte Buddy with all ASM dependencies repackaged into its own name space., homepageUrl=http://bytebuddy.net/byte-buddy, binaryArtifact=RemoteArtifact(url=https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.9.3/byte-buddy-1.9.3.jar, hash=Hash(value=f32e510b239620852fc9a2387fac41fd053d6a4d, algorithm=SHA-1)), sourceArtifact=RemoteArtifact(url=https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.9.3/byte-buddy-1.9.3-sources.jar, hash=Hash(value=ef8bdb760633510eed72e262193d6afbc451cc72, algorithm=SHA-1)), vcs=VcsInfo(type=Git, [email protected]:raphw/byte-buddy.git, revision=byte-buddy-1.9.3, path=), vcsProcessed=VcsInfo(type=Git, url=ssh://[email protected]/raphw/byte-buddy.git, revision=byte-buddy-1.9.3, path=), isMetadataOnly=false, isModified=false), Package(id=Identifier(type=Maven, namespace=net.bytebuddy, name=byte-buddy, version=1.9.3), purl=pkg:maven/net.bytebuddy/[email protected], cpe=null, authors=[Rafael Winterhalter], declaredLicenses=[The Apache Software License, Version 2.0], declaredLicensesProcessed=ProcessedDeclaredLicense(spdxExpression=Apache-2.0, mapped={The Apache Software License, Version 2.0=Apache-2.0}, unmapped=[]), concludedLicense=null, description=Byte Buddy is a Java library for creating Java classes at run time.
        This artifact is a build of Byte Buddy with all ASM dependencies repackaged into its own name space., homepageUrl=http://bytebuddy.net/byte-buddy, binaryArtifact=RemoteArtifact(url=, hash=Hash(value=, algorithm=)), sourceArtifact=RemoteArtifact(url=https://repo.maven.apache.org/maven2/net/bytebuddy/byte-buddy/1.9.3/byte-buddy-1.9.3-sources.jar, hash=Hash(value=ef8bdb760633510eed72e262193d6afbc451cc72, algorithm=SHA-1)), vcs=VcsInfo(type=Git, [email protected]:raphw/byte-buddy.git, revision=byte-buddy-1.9.3, path=), vcsProcessed=VcsInfo(type=Git, url=ssh://[email protected]/raphw/byte-buddy.git, revision=byte-buddy-1.9.3, path=), isMetadataOnly=false, isModified=false)]]
	at org.ossreviewtoolkit.analyzer.AnalyzerResultBuilder.build(AnalyzerResultBuilder.kt:49)
	at org.ossreviewtoolkit.analyzer.AnalyzerState.buildResult(Analyzer.kt:270)
	at org.ossreviewtoolkit.analyzer.Analyzer.analyzeInParallel(Analyzer.kt:180)
	at org.ossreviewtoolkit.analyzer.Analyzer.analyze(Analyzer.kt:132)
	at org.ossreviewtoolkit.cli.commands.AnalyzerCommand.run(AnalyzerCommand.kt:222)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:211)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
	at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:400)
	at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:397)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:415)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:440)
	at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:75)

To simplify debugging, I've create a test project where issue is reproducible with current HEAD and the issue above is visible:
https://github.com/devcooch/ort-gradle-maven-issue

[sschuberth]

To simplify debugging, I've create a test project where issue is reproducible with current HEAD and the issue above is visible:
https://github.com/devcooch/ort-gradle-maven-issue

Thanks for the reproducer! I can confirm the issue on main, but I can also confirm the issue to be gone with the GradleInspector 😸

[devcooch]

Oh, sounds interesting!

[devcooch]

I tried this gradle-inspector-model-builder branch and still fails on this test repository, I guess more branches are needed..

[sschuberth]

I tried this gradle-inspector-model-builder branch and still fails on this test repository, I guess more branches are needed..

No, but you need to explicitly enable GradleInspector and disable Gradle as described in the dedicated README.md.

[devcooch]

@sschuberth I am trying to run locally using this instruction. I tried modifying ~/.ort/config/config.yaml and through command-line, but new package manager somehow is not found. Am I missing something? I did git pull today and I assume the code is already merged.

The run configuration looks like this:

Maybe you have some idea or hint why GradleInspector package manager could not be found?

[sschuberth]

~/.ort/config/config.yaml

Note that the file needs to end with "yml", not "yaml". Also see #6848.

I did git pull today and I assume the code is already merged.

It is. But it might be that you're running into an issue that's only resolved with #6845.

[devcooch]

@sschuberth the extension was correct, I quoted it in incorrectly. Restarting IDE and rerunning Gradle sync multiple times apparently helped and it's visible now.

Easy way to prove that it's IDE issue is to check source tree:

"Properly" working plugins will have blue square (module).

I checked and now combination Maven+GradleInspector works fine on my example.
Thanks!

Should I close this issue?

[sschuberth]

Should I close this issue?

If the current implementation works for you, then yes, please go ahead.

For "real" cases of "packages and projects with the same ids" we already have #6465.

[devcooch]

Yes, #6465 is the one I observe now again. Back to different binary/source URLs topic caused by artifactory mirrors :)