Split off akmods into it's own layer

[castrojo]

The nvidia builds are becoming unwieldly (15m for each one and going up over time), if we were paying for build time it'd look like this:

We should consider splitting the akmods build steps into their own layer so we can build centrally but install it as many times as it's needed to the subsequent nvidia images.

[bsherman]

Not to derail the core issue here, but restructuring nvidia akmod builds may be a good time to consider how other akmod layers could be built. ( see: ublue-os/main#46 )

My thought is there could eventually be a list of akmods which need to be built, and none of that build time should run more than once daily.

I imagine building all the akmod RPMs once, and then COPYing them in for later install like is now done with config.
COPY --from=ghcr.io/ublue-os/akmods:latest /rpms /tmp/akmods-rpms

[dhoell]

That basically what I'm trying to do now. Except with the addition to build every akmod in separate container. So like
COPY --from=ghcr.io/ublue-os/akmod-nvidia:latest /var/cache /var/cache
COPY --from=ghcr.io/ublue-os/akmod-zfs:latest /var/cache /var/cache
...

But maybe I make a secondary image which just combines every built kmod rpm, this would reduce complexity downstream, if you can just
COPY --from=ghcr.io/ublue-os/akmod-cache:latest /var/cache /var/cache
and know that you have every akmod you could get.

My goal would be to make it possible that everyone can write an independent some_akmod-build.sh without worrying about signing or whatever, we can just copy it into the repo, use a containerfile template for it and it will work.

[Zoialord]

I think bsherman has what sounds like a good plan. I gotta ask though are these builds rebuilding the akmods every time no matter what changes are made?

It's just that would seem a bit of a waste when they really only need to be rebuilt with kernel updates/changes. If that is the case it would make even more sense to split them off. But I may be on the totally wrong track here.

[bsherman]

This issue is really meant to address that specific concern for nvidia drivers, and I'm just trying to piggy back on that to get those other drivers added in too. 😁

Currently, each variant build of the ublue-os/nvidia images rebuilds the akmods separately for that variant, but they only need to be built once for the specific kernel version. (same would be true for other akmods like xone and xpadneo).

It could probably get fancier and check for changes in kernel versions or something, but simply doing this build once per day would be a huge improvement.

[joshua-stone]

I think it'd be wise to have a ublue-os/akmods repo where various akmods can be built and signed to act as a build cache, and they'd share similar build patterns and follow best practices.

One reason this wasn't addressed earlier is that it wasn't straightforward to guarantee that kernels on the build root and the downstream image would match. This should be significantly easier now that ublue-os/main exists to "gate" builds and normalize the build environment.

[dhoell]

I'm currently making good progress on a separate akmod build layer, changing the build order from something like this:

to this

This enables akmods in the main image. Currently all akmods (well nvidia and v4l2loopback) get build within the same containerfile, I would like to split that too, so we have one containerfile for nvidia akmods, one for xpadneo akmods, one for zfs akmods, etc...
The containerfiles would be templateable, and any build related complexity contained to the build script specific to that container and akmod.

This makes it easier to write new akmods, as you don't have to shove everthing inbetween existing scripts and seperates build dependencies, really which shouldn't be a problem, but you never know...
As a bonus it would also reduce build time by eliminating redundant builds :)

[joshua-stone]

This looks sound to me. I started working on this recently by generalizing nvidia's build.sh for eventually living in akmods, with the major technical challenge being to ensure that package versions match between build time and install time as well as provide a generalized package resolving logic that's easy for downstream containers to consume. This can be quite frustrating with the way RPMFusion works with resolving package versions leading to xorg-x11-drv-nvidia-kmodsrc mismatching:

$ podman run -ti --entrypoint rpm kmod-test:latest -qa 'xorg-x11-drv-nvidia*'
xorg-x11-drv-nvidia-cuda-libs-520.56.06-1.fc37.x86_64
xorg-x11-drv-nvidia-libs-520.56.06-1.fc37.x86_64
xorg-x11-drv-nvidia-kmodsrc-525.89.02-1.fc37.x86_64
xorg-x11-drv-nvidia-power-520.56.06-1.fc37.x86_64
xorg-x11-drv-nvidia-520.56.06-1.fc37.x86_64
xorg-x11-drv-nvidia-cuda-520.56.06-1.fc37.x86_64
xorg-x11-drv-nvidia-devel-520.56.06-1.fc37.x86_64

Another challenge is Nvidia akmods conflicting with each other if more than one version is installed at a time. So we'd have to designate one container for all akmods that can be build together, and have one container for each Nvidia akmod.

A theoretical akmods.json so far looks something like this:

{
    "37": {
        "nvidia": {
            "520": [
                "xorg-x11-drv-nvidia",
                "xorg-x11-drv-nvidia-cuda-libs",
                "xorg-x11-drv-nvidia-kmodsrc",
                "xorg-x11-drv-nvidia-libs",
                "xorg-x11-drv-nvidia-power"
            ]
        }
    },
    "all": {
        "VirtualBox": {
            "*": [
                "VirtualBox-server",
                "liblzf",
                "libtpms"
            ]
        },
        "xtables-addons": {
            "*": [
                "xtables-addons"
            ]
        },
        "crystalhd": {
            "*": [
                "crystalhd",
                "crystalhd-firmware",
                "libcrystalhd",
                "unifdef"
            ]
        },
        "ndiswrapper": {
            "*": [
                "ndiswrapper",
                "perl-File-Copy"
            ]
        },
        "v4l2loopback": {
            "*": [
                "v4l2loopback"
            ]
        },
        "wl": {
            "*": [
                "broadcom-wl"
            ]
        },
        "nvidia": {
            "525": [
                "egl-gbm",
                "egl-wayland",
                "nvidia-settings",
                "xorg-x11-drv-nvidia",
                "xorg-x11-drv-nvidia-cuda-libs",
                "xorg-x11-drv-nvidia-libs",
                "xorg-x11-drv-nvidia-power"
            ]
        }
    }
}

And one could then run something like:

$ podman build --file akmods.Containerfile --build-arg AKMODS="nvidia=520,v4l2loopback='*'"

[dhoell]

New build process is coming along nicely.

Currently it pulls from quary.io/fedora-ostree-desktop/{silverblue,....} to build main (exactly the main we have now), uses base-main to build (in parallel) akmod-nvidia, akmod-v4l2loopback, {other akmods go here}, then uses {silverblue,....}-main and those akmods (except nvidia) to build {silverblue,....}-extended, and then uses {silverblue,...}-extended and akmod-nvidia to build {silverblue,...}-nvidia.

It's all in one repo, so we can use a properly set up (needs:) actions file to guarantee that the akmods get build with exactly the same image as where they will be installed 100% of the time. https://github.com/dhoell/akmods if you want to have a look

Also we get a bit of parallelism for free, since {silverblue,...}-extended can start building before akmod-nvidia is finished, saving a totally relevant ~2 min of build time 😄

I have not enabled all spins/versions/drivers in build.yml, wanted to get the basic process running first, before I do the rest I'd love to have some feedback from you, is this direction worth pursing?

[castrojo]

From chat: what if we could also set the exact nvidia version as a label so we know what the kernel and driver version are?

[bsherman]

This isn't done, but it is being worked on in #71

some preliminary PRs have been merged and the current ones need some tweaking but we are moving forward here.

[almereyda]

As this discussion did not allow the deeplink into the main project with an abbreviated syntax, here is the original target for others to follow:

• AKMods/Nvidia Split main#71

[bsherman]

As of August 25, 2023 with the merging of ublue-os/hwe#146 ... this proposal is completed.