Equivalent of module-hide --hidden-loaded?

[haampie]

Does Lmod have an equivalent of https://modules.readthedocs.io/en/latest/modulefile.html#mfcmd-module-hide?

We're generating reasonably large dependency graphs where say 30 modules are hidden, and only 2 are exposed to the user (under module avail).

Modules are hidden using an rc file:

hide_version("zlib-ng/2.1.4-xyz")

in case of Lmod, and

module-hide --soft --hidden-loaded zlib-ng/2.1.4-gcc-13.2.0-xyz

in case of "Environment Modules".

Currently after module load pkg:

• Lmod: module list shows only 32 modules, many of which don't mean much to the user
• Environment Modules: module list shows only 2 modules that are meaningful to the user

Does this functionality exist in Lmod too?

[rtmclay]

I have created a branch named IS690-hide. This branch support Tmod commands in .modulerc files

module-hide ...
module-forbid ...

And Lmod now supports the following commands in .modulerc.lua files and $LMOD_MODULERC files:

hide{name="",kind="soft or hard",  user={"user1",...}, group={}, notuser={}, notgroup={},before="YYYY-MM-DD",after="YYYY-MM-DD"}
forbid{name="", user={"user1",...}, group={}, notuser={}, notgroup={},before="YYYY-MM-DD",after="YYYY-MM-DD",message="",nearlymessage=""}

The documentation for this feature does not exist yet but the Tmod documentation describes what Lmod is doing.
There are minor differences. Namely that a user can get help from forbidden modules.

This branch has also added marking or decorations to terse output:

% module --terse --show_hidden avail
/path/to/lmod/IS690-hide/rt/modulerc/mfD/Core:
B/
B/1.0 <H>
C/
C/2.0
C/3.0 <F>
H/
H/1.0 <H>
cluster/
cluster/a
cluster/b
cluster/.defaultCluster <H> <F>

These decorations can be turned off by setting

export LMOD_TERSE_DECORATIONS=no

At the latest, this branch will be merged into the main branch on Nov. 15th 2024. Just before Supercomputing 2024. Please test this branch before then. Thanks!

[haampie]

Looking good. Only issue I found so far is that the tcl flag is --hidden-loaded instead of --hidden_load which the branch assumes.

[rtmclay]

I have updated this branch with this change. Thanks for testing this out.

[xdelaruelle]

I have updated this branch with this change. Thanks for testing this out.

Hello @rtmclay,

Option on my side is --hidden-loaded not --hidden-load. It would help @haampie a lot to have the exact same option whatever the module tool used (simpler Tcl modulefile generation).

Cheers,
Xavier

[boegel]

@rtmclay The <F> and <H> markers in the output of module --terse --show_hidden avail are a breaking chang, if they're enabled by default, can that be reconsidered? EasyBuild is going to trip over them (probably) since it parses the output of module --terse avail.

Also, if you're using markers anyway, why not make them more meaningful like <HIDDEN> and <FORBIDDEN>?
That doesn't make it less machine-readable imho.

[rtmclay]

@xdelaruelle Thanks for pointing out that --hidden-loaded matches. I have updated the branch to match.

@boegel: Thanks for the feedback. Yes. the and markers are a breaking change. These are the same markers that Tmod produces so I'll stay with those. Also setting:

export LMOD_TERSE_DECORATIONS=no

turns them off. This can also be turned off in /etc/lmod/lmod_config.lua with

cosmic:value("LMOD_TERSE_DECORATIONS","no")

when this branch becomes main

Also note that the Lmod hide{} now uses hidden_loaded instead of hidden_load.

[boegel]

@rtmclay OK, understood. Tracking that for EasyBuild in easybuilders/easybuild-framework#4637

A couple of related questions:

• Is $LMOD_TERSE_DECORATIONS a new configuration option (currently only in the IS690-hide branch?
• How does Lmod deal with unknown $LMOD_XXX environment variables to control configuration options, are they silently ignored? (I want to know whether I can set $LMOD_TERSE_DECORATIONS already, without caring about the Lmod version being used)
• Do you already know which Lmod version this will land in? Would it be Lmod 8.8.0?

[rtmclay]

Yes. LMOD_TERSE_DECORATIONS is new to the IS690-hide branch. Lmod does not care if you set $LMOD_TERSE_DECORATIONS now. Lmod only cares about the ones that are defined and doesn't care about unknown ones.

Lmod does error out if you try to set an unknown cosmic:value() to an unknown option.

Yes, this will be Lmod 8.8