Nextflow executions fail with message 'Picked up JAVA_TOOL_OPTIONS'

[pditommaso]

In some environments, NF execution fails with a message similar to the following

Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
USAGE: java <options> -jar ../../../.nextflow/framework/20.08.0-edge/nextflow-20.08.0-edge-one.jar

This is caused by the JAVA_TOOL_OPTIONS variable that interferes with the Capsule loader Java version detection.

The error stack trace reports

CAPSULE EXCEPTION: Could not parse version line: Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
java.lang.IllegalArgumentException: Could not parse version line: Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
	at Capsule.parseJavaVersionLine(Capsule.java:4168)
	at Capsule.getActualJavaVersion(Capsule.java:4175)
	at Capsule.getJavaVersion(Capsule.java:4100)
	at Capsule.chooseJavaHome0(Capsule.java:2567)
	at Capsule.chooseJavaHome(Capsule.java:2559)
	at Capsule.chooseJavaHome(Capsule.java:2559)
	at Capsule.chooseJavaHome(Capsule.java:2559)
	at Capsule.getJavaHome(Capsule.java:2543)
	at Capsule.resolve00(Capsule.java:3486)
	at Capsule.resolve0(Capsule.java:3402)
	at Capsule.resolve0(Capsule.java:3402)
	at MavenCapsule.resolve0(MavenCapsule.java:235)
	at Capsule.resolve(Capsule.java:3375)
	at Capsule.processOutgoingPath(Capsule.java:3542)
	at Capsule.buildJavaProcess(Capsule.java:2217)
	at Capsule.buildProcess0(Capsule.java:1530)
	at Capsule.buildProcess(Capsule.java:1521)
	at Capsule.buildProcess(Capsule.java:1521)
	at Capsule.buildProcess(Capsule.java:1521)
	at Capsule.prelaunch0(Capsule.java:1494)
	at Capsule.prelaunch(Capsule.java:1487)
	at NextflowLoader.prelaunch(NextflowLoader.java:42)
	at Capsule.prepareForLaunch(Capsule.java:1339)
	at Capsule.launch(Capsule.java:1268)

A possible patch consists of patching the Capsule loader. A simple workaround is unsetting the JAVA_TOOL_OPTIONS env variable.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mboisson]

Can this be fixed ?

[pditommaso]

Use NXF_OPTS instead of JAVA_TOOL_OPTIONS.

[mboisson]

Hem... no. JAVA_TOOL_OPTIONS is a documented java-wide setting. We are not using JAVA_TOOL_OPTIONS for nextflow, we are using JAVA_TOOL_OPTIONS for Java... This is a global setting for all java executions on our cluster, because otherwise Java crashes because it is not smart enough to detect how much memory it can actually use and attempts to use crazy amounts of memory.

This should be supported by any tool that uses java.

[pditommaso]

Consider submitting a PR https://github.com/nextflow-io/capsule

[mboisson]

Unfortunately, I am not at all fluent in Java, so I can't. I can open an issue on the other repository, if you think this is where the bug is located.

[mboisson]

Mmm, apparently, we can not open issues on the capsule repository

[mboisson]

For what it's worth, I can solve this by patching the binary-bash nextflow script: ComputeCanada/easybuild-computecanada-config@c4135d2

[pditommaso]

As of version 21.12.1-edge the launcher script allows the use of the NXF_JVM_ARGS variable to provide custom Java options without interfering with the pre-loading / version detection mechanism.

[nh13]

@pditommaso this is still an issue if we want to set JAVA_TOOL_OPTIONS system-wide (and for processes).

1. Is the preferred solution for processes to use the env scope? Edit: one can also use the -e option to nextflow run.
2. For other software, I suppose I can store JAVA_TOOL_OPTIONS , unset it, run nextflow, then restore JAVA_TOOL_OPTIONS, which is not ideal.

[pditommaso]

TLDR; for processes use env.JAVA_TOOL_OPTIONS in the nextflow config file. For nextflow use instead the variable NXF_JVM_ARGS (requires version 21.12.1-edge or later).

[mboisson]

The point remains that NextFlow should not barf because an environment variable it does not use is defined (JAVA_TOOLS_OPTIONS)

[pditommaso]

Fair enough 👉 30956af

[nh13]

@pditommaso thats a really unwelcome fix, since then anything run after Nextflow will not pick up JAVA_TOOL_OPTIONS!!!

See: puniverse/capsule#138. It shouldn’t be hard to fix capsule instead.

[mboisson]

@pditommaso thats a really unwelcome fix, since then anything run after Nextflow will not pick up JAVA_TOOL_OPTIONS!!!

See: puniverse/capsule#138. It shouldn’t be hard to fix capsule instead.

Actually, unless people use source on nextflow (which they probably shouldn't anyway), it will only unset JAVA_TOOL_OPTIONS within the nextflow process and subprocesses.

That is essentially what we are doing in this:
ComputeCanada/easybuild-computecanada-config@c4135d2

except that we also define export NFX_OPTS=\$JAVA_TOOL_OPTIONS before unsetting JAVA_TOOL_OPTIONS.

[nh13]

Ah, so it will still remain set outside Nextflow? If so, then false alarm.

[pditommaso]

It's ok

» export JAVA_TOOL_OPTIONS=foo
» ./nextflow info
  Version: 22.07.1-edge build 5726
  Created: 13-07-2022 10:53 UTC (12:53 CEST)
  System: Mac OS X 12.2.1
  Runtime: Groovy 3.0.10 on OpenJDK 64-Bit Server VM 17.0.2+8
  Encoding: UTF-8 (UTF-8)

» echo $JAVA_TOOL_OPTIONS
foo

the problem is that there was a typo 🤦 👉 0e7b416