Setting default JAVA_TOOL_OPTIONS

[mboisson]

Java has common memory issues, documented here:
https://docs.computecanada.ca/wiki/Java#Memory_Issues

The question is, would anything break if we set this by default ?

export _JAVA_OPTIONS="-Xms256m -Xmx2g"

[poquirion]

This overwrites the jvm options entered on the command line, so yes, it would break things. However It could be good to have it by default on the login nodes only, not on the compute nodes. But I do not know how difficult it would be to implement it like that.

Also note that as soon as we will have cgroups implemented on the login nodes instead of relying on ulimit RAM confinement, this java memory problem will go away.

[ostueker]

Indeed, if _JAVA_OPTIONS takes precedence over command line options, we shouldn’t set it by default.

This thread on StackOverflow discusses this variable and also comes to the conclusion that _JAVA_OPTIONS takes precedence over command line options and further more is undocumented.

However there is also the (documented) variable JAVA_TOOL_OPTIONS that has lower precedence than command line options that could be used for our purpose.

@poquirion : Setting _JAVA_OPTIONS only on login nodes won’t work, because environment variables from the user’s sessions will propagate to the jobs.

[mboisson]

Thanks @ostueker and @poquirion, that's good feedback. I would never have imagined an environment variable overridding command line options.

Changing the issue to define JAVA_TOOL_OPTIONS instead.

[mboisson]

I tested by installing rJava on a login node, which fails with a Java OOM by default. Both

JAVA_TOOL_OPTIONS="-Xmx2g"

or

_JAVA_OPTIONS="-Xmx2g"

work to resolve the OOM. Setting -Xms is not necessary (but is probably not hurtful). I therefore suggest setting

JAVA_TOOL_OPTIONS="-Xmx2g"

The question is then where to set it. Two options:

1. In each of the java modules (this requires re-deploying every java module).
2. Globally, in the CCconfig.lua. This would avoid changing the behavior on non-CC systems. https://github.com/ComputeCanada/software-stack-custom/blob/main/modules/CCconfig.lua
3. In the gentoo and nixpkgs modules. This changes it everywhere but avoid redeploying the java modules.

[mboisson]

This is now deployed. I redeployed the java modules, option 1. It seemed like the cleanest.

[mboisson]

Apparently, this causes this issue:
nextflow-io/nextflow#1716

[clemgoub]

Hello! My student and I are indeed encountering this problem now on beluga. Our Nextflow pipeline stopped working with the same error as nextflow-io/nextflow#1716

CAPSULE EXCEPTION: Could not parse version line: Picked up JAVA_TOOL_OPTIONS: -Xmx2g (for stack trace, run with -Dcapsule.log=verbose)
USAGE: <options> ../../cvmfs/soft.computecanada.ca/easybuild/software/2020/Core/nextflow/20.10.0/bin/nextflow

Actions:
  capsule.version - Prints the capsule and application versions.
  capsule.modes - Prints all available capsule modes.
  capsule.jvms - Prints a list of all JVM installations found.
  capsule.help - Prints this help message.

Options:
  capsule.mode=<value> - Picks the capsule mode to run.
  capsule.reset - Resets the capsule cache before launching. The capsule to be re-extracted (if applicable), and other possibly cached files will be recreated.
  capsule.log=<value> (default: quiet) - Picks a log level. Must be one of none, quiet, verbose, or debug.
  capsule.java.home=<value> - Sets the location of the Java home (JVM installation directory) to use; If 'current' forces the use of the JVM that launched the capsule.
  capsule.java.cmd=<value> - Sets the path to the Java executable to use.
  capsule.jvm.args=<value> - Sets additional JVM arguments to use when running the application.
Unable to initialize nextflow environment

Setting JAVA_TOOL_OPTIONS="-Xmx2g" did not change the outcome. Unfortunately, I am not familiar enough with Java and Nextflow, so I am willing to hear from you! Thanks!

Clément

[mboisson]

@clemgoub, for now, the only workaround is to unset JAVA_TOOL_OPTIONS. This environment variable is required for other Java usage. Nextflow is bugged in not supporting this.

[clemgoub]

Thank you @mboisson; I tried to add JAVA_TOOL_OPTIONS='' to my sbatch file; I also added export JAVA_TOOL_OPTIONS='' and sourced my ~/.bashrc with no success. $JAVA_TOOL_OPTIONS is empty when echoed in my work environment though.
Would you mind walking me through the proper procedure? -- Thanks again!

EDIT: FIXED. Following @robsyme advice, adding this to the sbatch file did the trick:

export NXF_OPTS=$JAVA_TOOL_OPTIONS
unset JAVA_TOOL_OPTIONS

[mboisson]

This will be fixed once I reinstall nextflow and patch it in place:
ComputeCanada/easybuild-computecanada-config@c4135d2

[clemgoub]

Excellent, thanks all for the support!

[mboisson]

This is now fixed.

[clemgoub]

Thanks a lot @mboisson!