RFC: remove track_features to weigh down newer sysroot

[minrk]

Summary

I think we should remove track_features from this package to allow the latest supported sysroot to be installed by default by the solver, instead relying on pinning such as c_stdlib_version. It seems the c_stdlib migration means we can do that soon, if not now.

Action items:

• identify specifically what is preventing the removal of track_features (e.g. c_stdlib migration propagation, docs for outside-conda-forge packaging)
• meet the above conditions (or determine we shouldn't do this at all)
• if/when met, remove track_features in this repo

Background

This discussion has been spread out over various issues and merged PRs without resolution as part of the c_stdlib and cos7 migrations. I'm trying to consolidate this discussion to a single topic so it can come to a resolution independent of all the other related issues where it has come up without being resolved.

Refs:

• "Getting rid of the sysroot hack is indeed one of the goals of the whole stdlib effort"
• "it will break non-conda build users"
• Drop cos6 builds conda-forge-pinning-feedstock#6070
• RFC: add strong_constrains on sysroot_{{ target_platform }}? #63
• mpifort 5.0.1 fails with undefined refs to memcpy@GLIBC_2.14 and clock_gettime@GLIBC_2.17 openmpi-feedstock#143
• implementation of conda_2_28 (aka centos 8) conda-forge.github.io#1941
• META: {{ stdlib("c") }} migration conda-forge.github.io#2102

Key points:

• the version of sysroot at build time governs the minimum version of the glibc runtime requirement. The lower the sysroot, the more widely distributable the results will be.
• c_stdlib_version is how we specify the sysroot version in conda-forge infrastructure (starting in March, 2024)
• conda-forge compilers and sysroot are also used at runtime in user environments
• track_features is used to "weigh down" the sysroot_linux packages, so that we can influence the 'default' sysroot version installed when no pinning is applied
• if track_features is removed, the latest sysroot compatible with the host environment would be installed when no pinning is applied

Reasons to remove track_features

• In most cases, it makes sense to get the latest available/compatible versions of every package by default. This makes the most sense and is least surprising for users. Special cases are surprising
• Runtime compilation should not be assumed to be for distribution purposes, which is the principle reason for weighing down by default
• Using latest available versions tends to yield best results for local compilation
• In the absence of run_exports/run_constrained on sysroot (RFC: add strong_constrains on sysroot_{{ target_platform }}? #63), default sysroot is often older than the oldest glibc supported by dependencies
• Workflows intended for distribution can set target minimum versions explicitly instead of implicitly, as conda-forge does now via c_stdlib_version

Reasons to keep track_features

• in the absence of pinning, compilation products will be less redistributable by default
• it is a change in default behavior, which can always be disruptive
• That would break non-conda-build users of our compilers since they would be getting 2.28 and would not be able to run executables made with our compilers if they have __glibc<=2.27

Rest

As I understand the various issues, removing the track_features changes the "maximum compatibility" sysroot/glibc behavior from opt-out to opt-in, but breaks nothing, and simplifies things for users and packagers alike.

Is there anything I'm missing? @isuruf made comments suggesting that things would be 'broken' by doing this, but I don't see how this is the case. The main situations I can see are summarized as:

• conda-forge (handled explicitly by c_stdlib_version, not affected)
• runtime compilation for local use (e.g. R/Julia packages, pip installs, code generation via fenics, etc.), where newer version is preferable and weighed-down 2.12 default has been causing problems (mostly gone with 2.17, but the same situation will presumably return as more packages require 2.28)
• packaging for distribution outside conda-forge tooling (may prefer older sysroot, but should do so explicitly instead of implicitly). I believe this is both conda-build outside conda-forge and non-conda-build.

Which, to me, is a general improvement over the situation currently produced by track_features in all situations.

[minrk]

I noticed that sysroot doesn't have runtime constraints on __glibc, which means it is possible to install sysroot 2.28 on a 2.17 system. Lacking this constraint is required to build targeting 2.28 on a 2.17 system (this necessarily means packages cannot be tested, like cross compilation). I'm not sure if that is meant to be a supported situation or not. (edit: this is intentional).

For removing track_features to be a complete solution, sysroot would need a runtime constraint on __glibc that matches its target (#69), however:

this would prevent us from building for cos9 using a cos8 image. I think this is a use case we have typically supported.

I'm not entirely sure how this is true, but if so removing track_features is not a complete solution. The desired goal remains the same. I probably shouldn't have started this with a specific solution in mind, but rather higher-level goals:

• user environments with conda compilers should get recent, but compatible sysroot by default. There is no such package right now. add run_constrained to target glibc #69 plus removing track_features makes sysroot that package.
• environment combinations that are meant to work shouldn't be cut off (which add run_constrained to target glibc #69 does, so no good there)
• packaging with conda-build should set targets explicitly e.g. with c_stdlib_version
• packaging outside conda-build should set targets explicitly e.g. with sysroot_linux...==2.27

The exact technical solution is less relevant, and may need one or more additional (meta-)packages. My main goal is confining the weighing-down of sysroot to conda-forge builds, as it isn't appropriate elsewhere.

[minrk]

@isuruf suggests:

Instead of making things uninstallable, a compromise would be to have multiple variant meta-packages where the highest priority variant is installable only on __glibc>=2.28 and the lowest priority version is installable only on __glibc>=2.17.

which I think looks a bit similar to what we have now, but instead of weighing down each sysroot > 2.17 in inverse version order, we have:

• sysroot x.y with >=__glibc x.y
• sysroot x.y without __glibc (or with lowest supported __glibc), weighed down equally with e.g. sysroot_without_glibc feature

(and apply this to all sysroot versions), such that:

• sysroot==x.y will always install, even if there is no matching __glibc
• sysroot will select the highest sysroot x.y with satisfiable __glibc by default
• sysroot>=x.y will install highest satisfiable sysroot if there is one, otherwise highest available sysroot.

(if that is in fact how weighing-down features behaves)

That leaves only one situation that we haven't technically had an issue with (I think), but might want to be able to express explicitly: "runtime-compilation", which should exclude the unsatisfiable sysroots, e.g. a metapackage that only depends directly on an exact sysroot and matching glibc bound:

- sysroot x.y
- __glibc >= x.y

I'm not sure this is worth having or not.

[mbargull]

We discussed this and related topics at today's core meeting; see conda-forge/conda-forge.github.io#2267 .

I gave Bioconda as an example for those downstream users (previously mentioned at conda-forge/conda-forge-pinning-feedstock#6070 (comment) ) whose recipes have not yet been adjusted to use the stdlib('c') Jinja function.
For those -- and others we don't necessarily know of -- we should have some kind of deprecation phase if/whenever we'd decided to remove the track_features down weighing to not break downstream.

---

Furthermore (and this is a rough paraphrasing from memory, so feel free to correct me), in the meeting @isuruf pointed out that we should take care to understand the different uses/intended use cases thoroughly so we can come up with solutions that accomplish the intended goals (without breaking others or missing the actual goals).
One example given was the want to link to a newer glibc than 2.17 -- which often times means the one from the machine's sysroot but not necessarily the latest one we package.

[minrk]

+100%, absolutely. If I have time, I'll try to propose a solution and what will happen in as many real situations as I can come up with, which should help with deciding when the deprecation window should end and what needs to be communicated and where