Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

figure out which CDTs we can drop for cos8, maybe drop them all? #66

Open
beckermr opened this issue Aug 23, 2023 · 36 comments
Open

figure out which CDTs we can drop for cos8, maybe drop them all? #66

beckermr opened this issue Aug 23, 2023 · 36 comments

Comments

@beckermr
Copy link
Member

We need to figure out which ones we need to keep vs ones we can build?

@h-vetinari
Copy link
Member

h-vetinari commented Nov 30, 2023

@conda-forge/core, I started with creating a list (based on the README, modulo aarches) and collecting some information (mainly based on the excellent pkgs.org).

If you think additional information would be useful (and don't care if the table formatting looks bad), please tell me what else I should add or remove. If/once people agree on a format I can fill in the rest.

PS. The separation into various areas is somewhat subjective, happy to reshuffle if people have better suggestions.

@h-vetinari
Copy link
Member

h-vetinari commented Nov 30, 2023

X11

Name
(cos6/cos7)
Name
(alma8)
Version
(cos7)
Version
(alma8)
Feedstock? Comment Keep as CDT
✅ / ❌ / ❔
libice yes
libice-devel
libsm yes
libsm-devel
libx11 yes
libx11-common
libx11-devel
libxau yes
libxau-devel
libxcb yes
libxcb-devel
libxcomposite yes
libxcomposite-devel
libxcursor yes
libxcursor-devel
libxdamage yes
libxdamage-devel
libxext yes
libxext-devel
libxfixes yes
libxfixes-devel
libxft yes
libxft-devel
libxi yes
libxi-devel
libxinerama yes
libxinerama-devel
libxkbcommon yes
libxkbcommon-devel
libxrandr yes
libxrandr-devel
libxrender yes
libxrender-devel
libxscrnsaver yes
libxscrnsaver-devel
libxshmfence no?
libxshmfence-devel
libxt yes
libxt-devel
libxtst yes
libxtst-devel
libxxf86vm yes
libxxf86vm-devel
xcb-util yes
xcb-util-devel
xcb-util-image yes
xcb-util-image-devel
xcb-util-keysyms yes
xcb-util-keysyms-devel
xcb-util-renderutil yes
xcb-util-renderutil-devel
xcb-util-wm yes
xcb-util-wm-devel
xorg-x11-proto-devel maybe
xorg-x11-server-common
xorg-x11-server-xvfb
xorg-x11-util-macros

GNOME

Name
(cos6/cos7)
Name
(alma8)
Version
(cos7)
Version
(alma8)
Feedstock? Comment Keep as CDT
✅ / ❌ / ❔
at-spi at-spi2-core(?) 1.32.0 2.28.0 maybe Libname changed
libspi.so->
libatspi.so
at-spi-devel at-spi2-core-devel(?) 1.32.0 2.28.0
atk same 2.28.1 2.28.1 maybe
atk-devel same 2.28.1 2.28.1
gconf2 same 3.2.6 3.2.6
gdk-pixbuf2 same 2.36.12 2.36.12 yes
gdk-pixbuf2-devel same 2.36.12 2.36.12
glib-networking same 2.56.1 2.56.1 yes
glib2 same 2.56.1 2.56.4 yes
glib2-devel same 2.56.1 2.56.4
gtk2 same 2.24.31 2.24.32 yes
gtk2-devel same 2.24.31 2.24.32
gtkmm24 same 2.24.5 2.24.5
gtkmm24-devel same 2.24.5 2.24.5
libbonobo
libbonobo-devel
libidl
libidl-devel
orbit2
orbit2-devel
webkitgtk
webkitgtk-devel

Other Graphics (DRM, Mesa, OpenGL, etc.)

Name
(cos6/cos7)
Name
(alma8)
Version
(cos7)
Version
(alma8)
Feedstock? Comment Keep as CDT
✅ / ❌ / ❔
cairo same 1.15.12 1.15.12 yes
cairo-devel same 1.15.12 1.15.12
libdrm yes
libdrm-devel
libglvnd ❌**
libglvnd-core-devel ❌**
libglvnd-devel ❌**
libglvnd-egl ❌**
libglvnd-gles ❌**
libglvnd-glx ❌**
libglvnd-opengl ❌**
mesa-dri-drivers ❌**
mesa-dri1-drivers ❌**
mesa-khr-devel ❌**
mesa-libegl ❌**
mesa-libegl-devel ❌**
mesa-libgbm
mesa-libgl maybe ❌**
mesa-libgl-devel ❌**
mesa-libglapi ❌**
pixman yes
pixman-devel

** needs: conda-forge/staged-recipes#25919

Other Hardware & Kernel Interfaces

Name
(cos6/cos7)
Name
(alma8)
Version
(cos7)
Version
(alma8)
Feedstock? Comment Keep as CDT
✅ / ❌ / ❔
alsa-lib yes Sound
alsa-lib-devel
cups-devel same 1.6.3 2.2.6
cups-libs same 1.6.3 2.2.6 Printer
ibacm yes Infiniband
libibcm yes
libibcm-devel yes
libibmad yes
libibmad-devel yes
libibumad yes
libibumad-devel yes
libibverbs yes
libibverbs-devel yes
libnl yes Kernel Netlib Sockets
libnl-devel
libnl3 yes Same libnl feedstock
libnl3-cli
libnl3-devel
librdmacm yes RDMA
librdmacm-devel yes
numactl yes
numactl-devel yes
numactl-libs yes
opensm
opensm-devel
opensm-libs
pciutils
pciutils-devel
pciutils-libs
rdma-core yes
rdma-core-devel yes

System & Security Libs

Name
(cos6/cos7)
Name
(alma8)
Version
(cos7)
Version
(alma8)
Feedstock? Comment Keep as CDT
✅ / ❌ / ❔
centos-release No direct match?
chkconfig same 1.7.6 1.19.2
kernel-headers yes,
per arch
kmod no System kernel modules (would stick to CDT)
libselinux conclusion
libselinux-devel
libpam no see here
libpam-devel
libsepol conclusion
libsepol-devel
libudev yes not found on pkgs.org for cos7?
libudev-devel
p11-kit yes
p11-kit-trust
systemd yes
systemd-devel
systemd-libs

Formats, Parsing & Images

This whole section should be replaceable with our own builds.

Name
(cos6/cos7)
Name
(alma8)
Version
(cos7)
Version
(alma8)
Feedstock? Comment Keep as CDT
✅ / ❌ / ❔
expat same 2.1.0 2.2.5 yes
expat-devel same 2.1.0 2.2.5
libjpeg-turbo same 1.2.90 1.5.3 yes
libpng same 1.5.13 1.6.34 yes Upstream provides several versions
libpng-devel same 1.5.13 1.6.34
libtiff yes
libuuid yes
libuuid-devel
libxml2 yes
libxml2-devel
pcre yes

Fonts & Language

Name
(cos6/cos7)
Name
(alma8)
Version
(cos7)
Version
(alma8)
Feedstock? Comment Keep as CDT
✅ / ❌ / ❔
fontconfig same 2.13.0 2.13.1 yes
fontconfig-devel same 2.13.0 2.13.1
freetype same 2.8 2.9.1 yes
freetype-devel same 2.8 2.9.1
fribidi same 1.0.2 1.0.4 yes
harfbuzz yes
libthai
pango yes
pango-devel

Web & Browser

Name
(cos6/cos7)
Name
(alma8)
Version
(cos7)
Version
(alma8)
Feedstock? Comment Keep as CDT
✅ / ❌ / ❔
ca-certificates same yes
libsoup yes HTTP
libsoup-devel
nspr yes
nss yes
nss-softokn
nss-softokn-freebl
nss-util

Rest

Name
(cos6/cos7)
Name
(alma8)
Version
(cos7)
Version
(alma8)
Feedstock? Comment Keep as CDT
✅ / ❌ / ❔
copy-jdk-configs same 3.3 4.0
java-1.7.0-openjdk
java-1.7.0-openjdk-headless
javapackages-tools
jpackage-utils
pkgconfig yes Should be able
to use our own
python-javapackages

@hmaarrfk
Copy link
Contributor

x11 can probably fully dropped. We've dropped all x11 from qt main's repo:
https://github.com/conda-forge/qt-main-feedstock/blob/main/recipe/meta.yaml#L69

@h-vetinari
Copy link
Member

@conda-forge/core, checking in here again - could people with opinions propose some ✅ / ❌ in the "keep as CDT?" column? (perhaps with a little abbreviation for your name, e.g. "Alice: ❌" or "Bob: ✅")

Especially the ❌'s would be important, but also stuff we absolutely cannot move away from CDTs.

x11 can probably fully dropped.

I mean we're all waiting for wayland to fully take over (right? 😉), though I guess you don't mean that we won't need x11 anymore (as a random example, there are >30 recipes still using the libx11 CDT...), but rather that we should be able to move to xorg-libx11 etc.?

@isuruf
Copy link
Member

isuruf commented Feb 20, 2024

All of expat, expat-devel, libjpeg-turbo, libpng, libpng-devel, libtiff, libuuid, libuuid-devel, libxml2, libxml2-devel, pcre can be dropped. Are they directly used in recipes or are they added to recipes as an indirect dependency?

@h-vetinari
Copy link
Member

h-vetinari commented Feb 20, 2024

All of expat, expat-devel, libjpeg-turbo, libpng, libpng-devel, libtiff, libuuid, libuuid-devel, libxml2, libxml2-devel, pcre can be dropped.

Thanks for the input! I updated the table above.

Are they directly used in recipes or are they added to recipes as an indirect dependency?

This query shows 26 recipes which use at least one of these directly, mostly expat. Another 8 feedstocks have at least one of them in yum_requirements.txt.

@kkraus14
Copy link

kkraus14 commented Feb 21, 2024

With the rdma-core feedstock (https://github.com/conda-forge/rdma-core-feedstock) most of the hardware / kernel interfaces are covered. Filling in the table now.

May need to do some work to split outputs from rdma-core if needed.

@isuruf
Copy link
Member

isuruf commented Feb 21, 2024

This query shows 26 recipes which use at least one of these directly, mostly expat. Another 8 feedstocks have at least one of them in yum_requirements.txt.

It's not immediately clear if they are direct or indirect because some CDTs don't have dependencies correctly and the maintainers have to add them to the recipe even though they are indirect.

@h-vetinari
Copy link
Member

Thanks a lot @kkraus14!

@isuruf I guess the simplest way to find out is to submit PRs which replace these cdts with our packages?

@h-vetinari
Copy link
Member

Speaking about HW interfaces, we can probably also drop alsa-lib for https://github.com/conda-forge/alsa-lib-feedstock/.

Also, I'm pretty sure everything under "Fonts & Language" could be replaced (perhaps with the exception of libthai that we don't have yet)

@jakirkham
Copy link
Member

Added a few more. Not positive whether we can drop the associated CDTs. So added question marks for now

@h-vetinari
Copy link
Member

Tentatively marked the following as ❌ (principally because it's stuff we have feedstocks for, which should just work 🤞):

  • gdk-pixbuf2, glib-networking, glib2, gtk2
  • cairo, libdrm, pixman
  • alsa-lib, libnl
  • kernel-headers
  • Everything under Fonts (except libthai)
  • Everything under Web that has a clear feedstock
  • pkgconfig (should be possible to use our own I believe)

Plus added some more feedstock matches, resp. candidates. Seems we have pretty much all of X11 already built separately (principally by @pkgw it seems 🙏)

Some near matches:

  • Not sure how much of the mesa-* CDTs is covered by our mesalib.
  • I'm guessing our at-spi2-core and at-spi2-atk cover most of what gtk/gnome should need
  • our libnl seems to cover at least v1 of the netlib interfaces; not sure about libnl3 though (CC @nehaljwani)

@jakirkham
Copy link
Member

jakirkham commented Mar 12, 2024

The conda-forge libnl is on version 3. So I think yes. Also confirmed it has the CLI bits

Mesa is complicated. Maybe we can look at gstreamer, opencv or qt-main to test removing the associated CDTs

@h-vetinari
Copy link
Member

This was again discussed in the core call today:

Overall, we're getting to the point where we've come to a decision (if preliminary) for ~80% of CDTs, so we should be able to restart working on the alma enablement (which is painful due to changed repo structure / urls), rebuild perhaps slightly more CDTs than necessary, and then try to move feedstocks over to using our own builds, and only falling back to CDTs if that runs into issues.

@jaimergp
Copy link
Member

jaimergp commented May 1, 2024

@jakirkham
Copy link
Member

Based on discussion at the core meeting it sounds like we are ok with the current list of CDTs

So we should be good to start producing these

cc @beckermr (for vis)

@isuruf
Copy link
Member

isuruf commented May 1, 2024

That's not true. Please see the meeting notes.

@h-vetinari
Copy link
Member

That's not true. Please see the meeting notes.

In the call, we did discuss how we're ready to start building all CDTs that aren't descoped in the mega-list above. We can still reduce that number later on if we prefer. The only open question was the effort around libglvnd, which shouldn't change the overall calculus AFAICT (except that we can leave out yet more packages once that becomes available). Perhaps you could update the matrix above with "❌**" or something like that, where the asterisks indicate that we can drop them once libglvnd becomes available?

@jakirkham
Copy link
Member

jakirkham commented May 8, 2024

Right it was my understanding we were done updating the table and ready to make CDTs

Could I ask that we spend the next week updating anything that still needs updates in the table?

Would like to get consensus for moving forward on this issue in the next core meeting. So we can start making these

@h-vetinari
Copy link
Member

Thanks @isuruf for filling out the table! Is there something particular about mesa-libgbm that makes it stand out compared to all the other mesa libs, or was it just a simple editing oversight that it didn't get a "❌**"?

@h-vetinari
Copy link
Member

Also, I've now marked all the X11-related packages for which we have feedstocks as ❌

@jakirkham
Copy link
Member

In today's core meeting, we decided to go ahead with any that have a ✅

@jakirkham
Copy link
Member

A user recently wrote up a list of X11 packages we are missing: conda-forge/staged-recipes#26241

@ehfd
Copy link
Member

ehfd commented Aug 12, 2024

https://github.com/conda-forge/libglvnd-feedstock is implemented. All OpenGL CDT packages may be dropped completely.

libgbm is part of Mesa, so it is required from there.

However, as a substitute for Mesa CDTs, it is a good idea to implement Mesa with libglvnd linking support as well.

@jakirkham
Copy link
Member

jakirkham commented Aug 13, 2024

https://github.com/conda-forge/libglvnd-feedstock is implemented. All OpenGL packages may be dropped completely.

Thanks Seungmin! 🙏

However, as a substitute for Mesa CDTs, it is a good idea to implement Mesa with libglvnd as well.

So would this mean updating the mesalib feedstock? If so, that appears not to use libglvnd currently (from a CDT or otherwise). Perhaps because we have EGL disabled in mesalib

If this would be a new feedstock, maybe it would be a good idea to capture what is needed in a new staged-recipes package request issue

@h-vetinari
Copy link
Member

We have about 130 feedstocks that use a yum-requirement that's either mesa-* or libgl-related. Sounds like we should maybe have a dedicated migrator (once an update mesa is in place?).

@ehfd
Copy link
Member

ehfd commented Aug 13, 2024

libglvnd does not require mesa during package build, and this is a key point this feedstock exists. However, during runtime, if libglvnd doesn't find a vendor library, it will error out.
If there is a GPU and its driver, it will not be a problem. But it is a potential issue without any GPUs existing.
Either way, the feedstock is always better than the CDTs inserted in build dependencies, sharing the same caveats that have never been resolved in Conda.

Basically, Mesa is only a requirement in some situations, which has never been fulfilled in Conda.

Moreover, as stated, libgbm is part of Mesa so that should be handled.

@isuruf
Copy link
Member

isuruf commented Aug 13, 2024

We don't need mesa in most packages that use mesa CDTs. Just replace with libgl-devel or libegl-devel.

@jakirkham
Copy link
Member

Just replace with libgl-devel or libegl-devel.

Initial attempt at such a change happening in PR: conda-forge/qt-main-feedstock#284

@ehfd
Copy link
Member

ehfd commented Aug 14, 2024

xorg-libxv, xorg-libxxf86dga, xorg-libxshmfence are new feedstocks or replacement feedstocks for CDTs.

Other information on X11 packages and their revamp: conda-forge/staged-recipes#26241

xorg-libxcomposite is implemented so this can be changed to a yes, and xorg-x11-proto-devel is equal to https://anaconda.org/conda-forge/xorg-xorgproto so this is also a yes.

@h-vetinari
Copy link
Member

Especially, I see that many feedstocks use CDTs and yum_requirements.txt for libxshmfence and libxxf86dga1

I cannot find a single feedstock in conda-forge that uses libxxf86dga1 - which ones are you talking about? libxshmfence at least occurs in 4 feedstocks - that does sound worthwhile to add.

@ehfd
Copy link
Member

ehfd commented Aug 21, 2024

Looks like it is mostly only for mplayer/mpv, xdpyinfo (xorg-x11-utils), etc. That do not have feedstocks.
@h-vetinari

xorg-libxv, xorg-libxxf86dga, xorg-libxshmfence are implemented and free for use in various feedstocks.

@ehfd
Copy link
Member

ehfd commented Aug 23, 2024

While libxv doesn't have used feedstocks because of no CDTs, it is an optional dependency that can be enabled for better performance on many X11 aspects in existing feedstocks such as GStreamer's xvimagesink, ffmpeg, zbar, etc. through build flags.

xorg-libxv, xorg-libxxf86dga, xorg-libxshmfence are implemented and free for use in various feedstocks.

These were important milestones for Xwayland in the future.

@h-vetinari
Copy link
Member

First batch of builds landed in #70! 🥳

@h-vetinari
Copy link
Member

Interested parties here can work towards removing usage of CDTs that we haven't lifted to alma8 (and yum_requirements that go along with it, especially those with a different name in alma8) across recipes.

We discussed in the core-call today that we'll probably do a separate migrator for that as well, but the less it has to chew through, the better. :)

Here's some searches to get you started (lots of X11, mesa, alsa etc. to be removed in both cases):

The only CDTs that can stay (if the feedstock still requires it) should be the following:

allowlists:
alma8:
- audit-libs
- audit-libs-devel
- cracklib
- cracklib-devel
- cracklib-dicts
- kmod
- kmod-devel
- kmod-libs
- libpwquality
- libpwquality-devel
- libselinux
- libselinux-devel
- libsepol
- libsepol-devel
- mesa-libgbm
- mesa-libgbm-devel
- pam
- pam-devel
centos7:

@hmaarrfk
Copy link
Contributor

hmaarrfk commented Sep 29, 2024

I think there is a question about the CDT builds as part of the tk + freetype.
conda-forge/tk-feedstock#63

Without them we would may have circular dependencies.

edit: would -> may. I haven't done the full research to check that we do in fact get circular dependencies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

8 participants