kernel-devel and kernel-headers matching kernel-core not available

[mtalexan]

Describe the bug

When running either rpm-ostree update or ujust update, I get the report that there's nothing to update.
However I need to build a custom out of tree kernel driver for the lighting on my laptop keyboard (Asus Predator device), so I need the kernel-devel installed. I tried this and then discovered it didn't match the current kernel-core that was installed by default.

After installing kernel-core and kernel-devel:

$ rpm -q kernel-core
kernel-core-6.9.12-205.fsync.fc40.x86_64

$ rpm -q kernel-devel
kernel-devel-6.10.12-200.fc40.x86_64

$ rpm -q kernel-headers
kernel-headers-6.10.3-200.fc40.x86_64

And when I try to install the matching version for kernel-core explicitly:

$ rpm-ostree install kernel-devel-$(uname -r) kernel-headers-$(uname -r)
Checking out tree 5b620b6... done
Enabled rpm-md repositories: copr:copr.fedorainfracloud.org:matte-schwartz:sunshine copr:copr.fedorainfracloud.org:rodoma92:kde-cdemu-manager copr:copr.fedorainfracloud.org:rodoma92:rmlint copr:copr.fedorainfracloud.org:rok:cdemu updates fedora rpmfusion-free-updates-testing rpmfusion-free-updates rpmfusion-free code updates-archive
Importing rpm-md... done
rpm-md repo 'copr:copr.fedorainfracloud.org:matte-schwartz:sunshine' (cached); generated: 2024-09-13T09:52:44Z solvables: 2
rpm-md repo 'copr:copr.fedorainfracloud.org:rodoma92:kde-cdemu-manager' (cached); generated: 2024-04-20T12:39:00Z solvables: 12
rpm-md repo 'copr:copr.fedorainfracloud.org:rodoma92:rmlint' (cached); generated: 2024-05-10T10:16:28Z solvables: 4
rpm-md repo 'copr:copr.fedorainfracloud.org:rok:cdemu' (cached); generated: 2024-09-18T12:52:44Z solvables: 23
rpm-md repo 'updates' (cached); generated: 2024-10-11T03:33:46Z solvables: 28845
rpm-md repo 'fedora' (cached); generated: 2024-04-14T18:51:11Z solvables: 74881
rpm-md repo 'rpmfusion-free-updates-testing' (cached); generated: 2024-10-09T14:33:29Z solvables: 2
rpm-md repo 'rpmfusion-free-updates' (cached); generated: 2024-10-09T14:33:22Z solvables: 164
rpm-md repo 'rpmfusion-free' (cached); generated: 2024-04-20T12:11:51Z solvables: 422
rpm-md repo 'code' (cached); generated: 2024-10-11T21:14:29Z solvables: 459
rpm-md repo 'updates-archive' (cached); generated: 2024-10-10T02:51:28Z solvables: 47039
error: Packages not found: kernel-devel-6.9.12-205.fsync.fc40.x86_64, kernel-headers-6.9.12-205.fsync.fc40.x86_64

This is a brand new install of the latest Bazzite, with the only change to the repo being the addition of the VSCode RPM repo. There is no matching kernel-devel or kernel-headers for the fsync kernel-core available in any repos.

What did you expect to happen?

I expected there to be a kernel-devel and kernel-headers matching the default shipped kernel-core. On Bazzite this is a custom UBlue fsync version (apparently) and doesn't keep up with the latest from upstream Fedora, so UBlue would be responsible for providing the associated kernel-devel and kernel-headers to match their custom kernel-core.

Output of rpm-ostree status

State: idle
Deployments:
● ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-nvidia-open:stable
Digest: sha256:f0a715ec44064ebfa14504db0c72b3790542ce3bcd3d7f3b9fb068e6a04197e7
Version: 40.20240930.0 (2024-09-30T15:25:37Z)
LayeredPackages: code grubby kernel-devel kernel-devel-uname-r sunshine zsh

ostree-image-signed:docker://ghcr.io/ublue-os/bazzite-nvidia-open:stable
Digest: sha256:f0a715ec44064ebfa14504db0c72b3790542ce3bcd3d7f3b9fb068e6a04197e7
Version: 40.20240930.0 (2024-09-30T15:25:37Z)
LayeredPackages: code grubby kernel-devel sunshine zsh

Hardware

$ inxi
CPU: 14-core (6-mt/8-st) 12th Gen Intel Core i9-12900H (-MST AMCP-)
speed/min/max: 771/400/4900:5000:3800 MHz
Kernel: 6.9.12-205.fsync.fc40.x86_64 x86_64 Up: 27m
Mem: 8.37/31.04 GiB (27.0%) Storage: 92.53 GiB/-4784 Procs: 460 Shell: Zsh
inxi: 3.3.34

$ cat /sys/devices/virtual/dmi/id/product_name
Predator PH315-55s

Extra information or context

The particulars of the device don't really matter, the problem is that there seems to be no repos to provide updates or support for the custom fsync kernel-core that ships with Bazzite.
It's reasonable that the kernel-header and kernel-devel matching the kernel-core aren't installed by default in the Bazzite ostree refs, and it's certainly easier to distribute the custom kernel-core purely by including it directly there rather than providing it via an RPM repo on copr or something, but some method of getting the kernel-devel and kernel-headers matching it are a minimum necessity of having a custom kernel in the first place.

[mtalexan]

Incidentally, this brought up basically the same topic and was closed almost immediately without any resolution whatso ever: #899

[mtalexan]

I'm not at all sure where the fsync kernel-core being used is coming from, but at a rough guess it looks like maybe this copr could add what's needed, presuming it keeps old enough builds that it would still have the 6.9 kernel (the oldest kernel version Fedora itself still supports). https://copr.fedorainfracloud.org/coprs/sentry/kernel-fsync/

[mtalexan]

Yeah ok. This is a clear initial comfig bug in UBlue. Any ostrees shipping with the fsync kernel require the copr repo to be included in the /etc/yum.repos.d/ files, and the default kernels from the fefora-updates.repo to be disabled. Exactly as the https://copr.fedorainfracloud.org/coprs/sentry/kernel-fsync/ explains. If you want to add a ujust command to toggle the Fedora kernels back on that could make sense so people could then install newer kernels without fsync patches, but the default has to be to use the matching kernel packages from the copr.

[mtalexan]

I see why the copr repo isn't installed and setup as the default. The copr only keeps the most recent build, none of the past ones, which means it has only the 6.11 kernel right now. The current latest UBlue/Bazzite-provided kernel-core-6.9.12-205.fsync.fc40.x86_64 is relatively old, and is not supported by the copr anymore (probably hasn't been for >1 year).

I guess if UBlue is going to ship with the fsync kernel and not keep to the bleeding edge like the upstream copr supports, it's going to have to figure out how to support a cached version of all the associated kernel packages so it has the older fsync versions available. Maybe UBlue needs to setup it's own copr just to host cached copies of things?

In the meantime, it's not possible to build kernel drivers for any fsync UBlue images as far as I can tell. The minimum necessary kernel-devel doesn't exist anymore and (apparently) wasn't cached before it was removed from the upstream copr. From all appearances, it looks like even the UBlue akmods layer isn't functional for fsync kernels (it doesn't actually appear to be functional for anything to be honest, relying on GitHub releases of container images to keep cached copies of the kernel RPMs but the most recent release being from April 2024 and therefore way out of date).

[mtalexan]

Looking more at the UBlue akmods, it appears there's actually nightly updates to the akmods container images that just aren't published as releases. So doing this would get the set of RPMs from the latest for the fsync kernel:

$ podman pull ghcr.io/ublue-os/fsync-kernel:40
...
$ podman container create --name=foo ghcr.io/ublue-os/fsync-kernel:40
...
$ podman container cp foo:/tmp/rpms $(pwd)
...
$ ls rpms/
kernel-6.11.2-201.fsync.fc40.x86_64.rpm        kernel-devel-matched-6.11.2-201.fsync.fc40.x86_64.rpm  kernel-modules-core-6.11.2-201.fsync.fc40.x86_64.rpm
kernel-core-6.11.2-201.fsync.fc40.x86_64.rpm   kernel-headers-6.11.2-201.fsync.fc40.x86_64.rpm        kernel-modules-extra-6.11.2-201.fsync.fc40.x86_64.rpm
kernel-devel-6.11.2-201.fsync.fc40.x86_64.rpm  kernel-modules-6.11.2-201.fsync.fc40.x86_64.rpm        kernel-uki-virt-6.11.2-201.fsync.fc40.x86_64.rpm

This is running into a similar issue as the copr though, it only has the latest available kernel version.

[m2Giles]

You are currently using the fsync-ba kernel.

[mtalexan]

You are currently using the fsync-ba kernel.

Does the -ba part of this mean something?
And if so, how can you tell when the reported kernel version from both inxi and kernel-core exactly match with what the copr fsync packages/kernel report?

[antheas]

-ba zzite

We are currently in the process of moving building the kernel to github. This means that for you, you'll have access to all the versions for a long time.

https://github.com/hhd-dev/kernel-bazzite

With a changelog as well.

[antheas]

Looking more at the UBlue akmods, it appears there's actually nightly updates to the akmods container images that just aren't published as releases. So doing this would get the set of RPMs from the latest for the fsync kernel:

$ podman pull ghcr.io/ublue-os/fsync-kernel:40
...
$ podman container create --name=foo ghcr.io/ublue-os/fsync-kernel:40
...
$ podman container cp foo:/tmp/rpms $(pwd)
...
$ ls rpms/
kernel-6.11.2-201.fsync.fc40.x86_64.rpm kernel-devel-matched-6.11.2-201.fsync.fc40.x86_64.rpm kernel-modules-core-6.11.2-201.fsync.fc40.x86_64.rpm
kernel-core-6.11.2-201.fsync.fc40.x86_64.rpm kernel-headers-6.11.2-201.fsync.fc40.x86_64.rpm kernel-modules-extra-6.11.2-201.fsync.fc40.x86_64.rpm
kernel-devel-6.11.2-201.fsync.fc40.x86_64.rpm kernel-modules-6.11.2-201.fsync.fc40.x86_64.rpm kernel-uki-virt-6.11.2-201.fsync.fc40.x86_64.rpm
This is running into a similar issue as the copr though, it only has the latest available kernel version.

There exist these things called tags. You can get all previous versions. See https://github.com/ublue-os/kernel-cache/pkgs/container/fsync-kernel

[mtalexan]

Looking more at the UBlue akmods, it appears there's actually nightly updates to the akmods container images that just aren't published as releases. So doing this would get the set of RPMs from the latest for the fsync kernel:
$ podman pull ghcr.io/ublue-os/fsync-kernel:40
...
$ podman container create --name=foo ghcr.io/ublue-os/fsync-kernel:40
...
$ podman container cp foo:/tmp/rpms $(pwd)
...
$ ls rpms/
kernel-6.11.2-201.fsync.fc40.x86_64.rpm kernel-devel-matched-6.11.2-201.fsync.fc40.x86_64.rpm kernel-modules-core-6.11.2-201.fsync.fc40.x86_64.rpm
kernel-core-6.11.2-201.fsync.fc40.x86_64.rpm kernel-headers-6.11.2-201.fsync.fc40.x86_64.rpm kernel-modules-extra-6.11.2-201.fsync.fc40.x86_64.rpm
kernel-devel-6.11.2-201.fsync.fc40.x86_64.rpm kernel-modules-6.11.2-201.fsync.fc40.x86_64.rpm kernel-uki-virt-6.11.2-201.fsync.fc40.x86_64.rpm
This is running into a similar issue as the copr though, it only has the latest available kernel version.

There exist these things called tags. You can get all previous versions. See https://github.com/ublue-os/kernel-cache/pkgs/container/fsync-kernel

In Github, you can only see container tags if it was listed as a release, or if it's still present and someone gives you a link. If you don't have permissions to run builds, you also can't see a list of them. That's what releases are intended to show. So currently there are no public lists of any container builds at all in that repo.
By reading the Dockerfile code and README I was able to manually reconstruct what tag was pulled during the build for the kernel-archive image, and where files in it lived. Which is where I found the build (GitHub will show you the info about the digest matching the image tag if you go there in a browser instead of using docker pull). The content matching the current latest non-testing bazzite ostree release (which has the 6.9 fsync kernel still) doesn't match with the akmods build for the same configuration description (fsync-kernel:40), which is why I assumed it was a nightly.

If you're talking about git tags instead, then sure if there were some way to relate that to an image digest for the kernel-archive, then I would. But all akmod builds use a moving tag (e.g. 40) and don't track what the digest was at the moment it resolved the tag. And remember, builds lists can only be seen if you personally have permission to run them.

[antheas]

In Github, you can only see container tags if it was listed as a release, or if it's still present and someone gives you a link. If you don't have permissions to run builds, you also can't see a list of them. That's what releases are intended to show. So currently there are no public lists of any container builds at all in that repo. By reading the Dockerfile code and README I was able to manually reconstruct what tag was pulled during the build for the kernel-archive image, and where files in it lived. Which is where I found the build (GitHub will show you the info about the digest matching the image tag if you go there in a browser instead of using docker pull). The content matching the current latest non-testing bazzite ostree release (which has the 6.9 fsync kernel still) doesn't match with the akmods build for the same configuration description (fsync-kernel:40), which is why I assumed it was a nightly.

If you're talking about git tags instead, then sure if there were some way to relate that to an image digest for the kernel-archive, then I would. But all akmod builds use a moving tag (e.g. 40) and don't track what the digest was at the moment it resolved the tag. And remember, builds lists can only be seen if you personally have permission to run them.

Uh yes it does. The last kernel in stable was fsync-ba, and you can see that the last tag in fsync-ba-kernel is the one currently in bazzite.

As for what the digest was, the images are tagged using the kernel version. So, seeing that in bazzite stable you could derive that the current image is 6.9.12-210.fsync.fc40.x86_64.

Below is an image of the kcache in incognito mode. I can see all of the tagged versions.

I am not going to say this is ideal and it is something that we are moving away from.

Here is the repo:
https://github.com/ublue-os/kernel-cache/pkgs/container/fsync-ba-kernel

We moved to fsync-ba in August due to 6.10 being completely broken. We are ready to move on to 6.11 now with a cleaner build system.

[mtalexan]

In Github, you can only see container tags if it was listed as a release, or if it's still present and someone gives you a link. If you don't have permissions to run builds, you also can't see a list of them. That's what releases are intended to show. So currently there are no public lists of any container builds at all in that repo. By reading the Dockerfile code and README I was able to manually reconstruct what tag was pulled during the build for the kernel-archive image, and where files in it lived. Which is where I found the build (GitHub will show you the info about the digest matching the image tag if you go there in a browser instead of using docker pull). The content matching the current latest non-testing bazzite ostree release (which has the 6.9 fsync kernel still) doesn't match with the akmods build for the same configuration description (fsync-kernel:40), which is why I assumed it was a nightly.
If you're talking about git tags instead, then sure if there were some way to relate that to an image digest for the kernel-archive, then I would. But all akmod builds use a moving tag (e.g. 40) and don't track what the digest was at the moment it resolved the tag. And remember, builds lists can only be seen if you personally have permission to run them.

Uh yes it does. The last kernel in stable was fsync-ba, and you can see that the last tag in fsync-ba-kernel is the one currently in bazzite.

As for what the digest was, the images are tagged using the kernel version. So, seeing that in bazzite stable you could derive that the current image is 6.9.12-210.fsync.fc40.x86_64.

Below is an image of the kcache in incognito mode. I can see all of the tagged versions.

I am not going to say this is ideal and it is something that we are moving away from.

Here is the repo: https://github.com/ublue-os/kernel-cache/pkgs/container/fsync-ba-kernel

We moved to fsync-ba in August due to 6.10 being completely broken. We are ready to move on to 6.11 now with a cleaner build system.

Oh you meant the kernel image tag is fsync-ba-kernel, I didn't understand that part.

Oh, I see what you mean, if I open the image name without a version tag in a browser, it redirects to the image itself to see the version tags. I've only ever followed to a specific version tag. I guess I'm just used to the pretty layout from registries like Quay.io or hub.Docker.com.

OK, that helps get me over the hump then. It might be worth adding something to grab these packages automatically as a ujust command, then all the digging on these particulars don't matter as much.

[castrojo]

skopeo inspect is also good for tag sleuthing: https://github.com/containers/skopeo/blob/main/docs/skopeo-inspect.1.md

[mtalexan]

skopeo inspect is also good for tag sleuthing: https://github.com/containers/skopeo/blob/main/docs/skopeo-inspect.1.md

I did just find that yesterday (unrelated actually), but apparently it wouldn't have helped in this case because I didn't even have the right basename.