eopkg: Handle the py2-based pisi to py3-based eopkg migration as robustly as possible

[ermo]

Motivation

Py2 was designated EOL in 2020. We need to move off py2 for our core tooling.

Blocked-by

Currently, the actual py2 to py3 migration is blocked by the following tasks:

• Retiring py2-based solus-sc (current focus)
  • Getting the eopkg py2 PackageKit backend working "well enough" (Done)
    Note that the PackageKit eopkg4 branch we use for our development is intended to work on both py2 and py3, so finishing this also unblocks py3 PK integration.
  • Promoting GNOME Software to non-experimental status (Done)
  • Promoting KDE Discover 6 to non-experimental status (Done)
    • Building the ability for KDE Discover 6 PK integration to replace LDM (Confirmed working, PR here (?), needs upstreaming at some point)
  • Move the deprecation process out of the solus-sc repo and into the monorepo. (Done)
  • Get a handle on how much documentation we need for the release to be able to not include Solus SC on the ISO
  • Get metrics for how many of our apps that need, but don't yet have, appstream metadata (no appstream data == app "doesn't exist" from the PoV of G-S and Discover)
    • Decide how large a percentage of coverage is "enough" based on those metrics
• Develop a replacement for the baselayout package (Done)
• Figure out how to get rid of any remaining comar use cases (TBD)
• Extensive testing of the full set of upgrade starting point combinations (TBD)
• Build out an epoch bump process (In progress)
  • Ensure that we have a good handle on the required eopkg functionality, such as being able to upgrade first eopkg, then restart, then upgrade baselayout, then restart, then upgrade any system.base packages, then restart and finally upgrading remaining packages in one go for reliability reasons. (In progress, eopkg tracker issue here)

Design Considerations

The transition off of depending on python2 for Solus tooling will be split up into four phases (1+2 before the /usr merge epoch bump, 3+4 after the /usr merge epoch bump).

The transition is currently planned to look like this:

1. Land unified eopkg4 py3-based package

Status: DONE.

Get the unified eopkg4 (py3) package landed in the Solus repo, which includes a nuitka-compiled eopkg.bin executable that's written in python3, but due to the magic of the nuitka compiler, only depends on glibc.

2. Make some clever adjustments to the eopkg4 and pisi solus packages

Status: DONE. PR here.

In the 2nd phase, we update the eopkg4 (py3), pisi (py2) and solbuild packages to:

• Make the pisi package (needed for Solus SC 3rd party) use .py2 extensions for all executable package manager scripts.
• Make the eopkg4 and the python-eopkg subpackage use .py3 extensions for its scripts. In addition the eopkg package will have its component changed from system.utils to system.base (it only depends on glibc).

This makes it possible to have the eopkg4 package (contains eopkg.bin), the python-eopkg package (contains eopkg.py3, lseopkg->lseopkg.py3, and uneopkg->uneopkg.py3), and the pisi package (contains eopkg.py2 and a bunch of other scripts with the .py2 extension) all be co-installable while being able to compare functionality for them.

3. Interlude -- /usr-merge and 4.6 ISO release

Status: DONE (/usr-merge script in prod, 4.6 ISO released)

Tracker issues/PRs:

• usysconf-epoch script

The short version is that the 4.6 ISO release will need to contain co-installed eopkg4 (py3) and pisi (py2) packages. The ISO script was recently updated to do a /usr-merge run during the ISO generation itself, thus ensuring that all ISO created from now on are /usr-merged.

We also got rid of the requirement for AppArmor patches to be present, which means that we can begin working on ISO generation automation in earnest.

• Solus SC (including 3rd Party) will still work, because it uses pisi py2 APIs. Solus SC will be available alongside the new GNOME Software and KDE Discover software centers (which will in turn be using the python3 eopkg PK backend).
• (?) solbuild will be updated to call eopkg.py3 build for 3rd party packaging wrapper operations (upstream solbuild repo has a draft PR up, which will be landed at the requisite time)

Prerequisites for phase 4 readiness

• Make solbuild use eopkg.bin (pisi/eopkg: Add versioned FilesDB and LazyDB instances #4063 and solbuild: Drop fakeroot use, use eopkg.bin #3850)
• Make sure packages are suitably asciified, as this will help with latin1 -> utf-8 migration (https://github.com/getsolus/packages/tree/main/a/asciify)
• Finish AppStream generation automation ()
• Ensure decent AppStream coverage for packages (https://github.com/getsolus/packages/blob/main/common/Scripts/check_appstream_progress.py)
• Fix remaining issues in eopkg4 (https://github.com/getsolus/eopkg/issues)
  • Build a better compat/upgrade story when moving from eopkg.py2 to eopkg.py3 (pisi/eopkg: Add versioned FilesDB and LazyDB instances #4063)
• Ensure that ferryd can remove repos (Ensure that repository removal functionality works and is hooked up to ferryctl ferryd#34)
• Update py2 package recipes to not be in system.devel or system.base (for epoch bump removal purposes)

4. Epoch bump, remove py2 (including pisi and Solus SC), switch to utf-8 package strings

Status: TBD

• In this release, the eopkg4 package will take over ownership of the eopkg and eopkg-cli symlinks, which will both point to eopkg.bin
• Documentation and ancillary files from the pisi package will be moved to the eopkg4 package (solus packaging work is ready in draft form in existing package recipes)
• eopkg4 (.bin/.py3) will switch to use utf-8 encoding exclusively (no more weird string hacks!)
• Python 2 packages (including pisi, Solus SC + Solus Update Checker) will not be present in the repo and will be removed by eopkg.bin

This will be deployed in the same way as the usr-merge as a script, but this script will only run after a successful /usr-merge run. Because we set things up so that /usr-merged systems are guaranteed to have eopkg4 installed as part of system.base, this phase will use eopkg.bin to remove the pisi and solus-sc packages (and their dependencies) from the system during early boot via a systemd oneshot script.

The reason we can do it this way, is that we were recently lucky enough to figure out how to fix eopkg4 to use the protocol=2 py2 compatible pickle cache version. This means that people who did their last update to their systems with Solus SC (py2), will still be left with a readable pickle cache when first the /usr-merge script and then the epoch bump script run using eopkg.bin.

Testing the process

Status: TBD

It is worth mentioning that ^ can all be tested in a VM by using e.g. a Solus 4.5 ISO as the starting point, then making a snapshot of it before any updates are applied, at which point the testing process is trivially repeatable by rolling back to this initial snapshot.

Here, the "process" is "update through phase 1 and 2, do the phase 3 /usr-merge system and the phase 4 epoch jump, then watch the system still work".

What then?

At this point (post the /usr-merge and phase 4 epoch bump), Solus is using eopkg.bin (py3) and utf-8 string in packages, and solbuild is using nuitka-compiled py3 eopkg.bin and ypkg binaries. Python2 packages are no longer present in the repo.

Any ISO released after phase 4 is completed will contain no python2 packages.

Originally, I had proposed to do a second epoch jump after phase 4, but sheepman pointed out that this might not be necessary, which led us to the process described above.

sheepman
I would be comfortable with a warning (multiple month period) that the SC will soon stop working and that you will need to be on discover/g-s to do GUI updates after , and remove py2+pisi+sc on that date. Systems that people attempt to update with SC after that date will temporarily break and need to be rescued by completing the update on the CLI, and that's just what you get for not updating for 6mos
ermo
If we make this process work with a 3 to 6 month deprecation cycle for Solus SC, that means that we're looking towards the end of Q4 '24 or the end of Q1 '25 (depending on when we begin phase 3, which would have to start with the deprecation notice).

Solus Packaging Context

The pisi package (before phase 1+2) contains the following binaries in /usr/bin:

$ eopkg info -f pisi |gawk '/bin/ { print $1 }' |tr -d ',' |xargs ls -l
-rwxr-xr-x 1 root root 2388 Apr  5 12:41 /usr/bin/check-newconfigs.py
lrwxrwxrwx 1 root root    9 Apr  5 19:35 /usr/bin/eopkg -> eopkg-cli
-rwxr-xr-x 1 root root 2178 Apr  5 12:41 /usr/bin/eopkg-cli
-rwxr-xr-x 1 root root 1526 Apr  5 12:41 /usr/bin/lseopkg
lrwxrwxrwx 1 root root    7 Apr  5 19:35 /usr/bin/lspisi -> lseopkg
-rwxr-xr-x 1 root root 5085 Aug 28  2023 /usr/bin/revdep-rebuild
-rwxr-xr-x 1 root root 1485 Apr  5 12:41 /usr/bin/uneopkg
lrwxrwxrwx 1 root root    7 Apr  5 19:35 /usr/bin/unpisi -> uneopkg

All (or at least most) of these binaries would need to be replaced by a combination of pure py3 executables (with the extension .py3) and nuitka-compiled py3 executables (with the extension .bin), which would in turn be symlinked to the correct executable artefact names per above.

Proposed Solution for making eopkg4 the "main" eopkg package (phase 4)

In packaging terms, this implies that the pisi package will be the "backup" package that is co-installable with (and depends on) eopkg4 (and not the other way around):

• eopkg4:
  • nuitka-compiled /usr/bin/eopkg.bin
  • /usr/share/PackageKit/helpers/eopkg/eopkgBackend.bin
  • /usr/share/defaults/eopkg
  • /usr/share/man/man1/eopkg.1
  • /usr/bin/eopkg symlink pointing to /usr/bin/eopkg.bin
  • /usr/bin/eopkg-cli symlink pointing to /usr/bin/eopkg.bin
• python-eopkg:
  • /usr/lib/python3.11/site-packages/eopkg*
  • /usr/lib/python3.11/site-packages/pisi/
  • /usr/bin/check-newconfigs.py3 (TBD)
  • /usr/bin/check-newconfigs.py symlink pointing to /usr/bin/check-newconfigs.py3 (TBD)
  • /usr/bin/eopkg.py3 (useful to have around for performance regression testing)
  • /usr/bin/lseopkg.py3
  • /usr/bin/lseopkg symlink pointing to /usr/bin/lseopkg.py3
  • (/usr/bin/lspisi symlink pointing to /usr/bin/lseopkg.py3)
  • /usr/bin/uneopkg.py3
  • /usr/bin/uneopkg symlink pointing to /usr/bin/uneopkg.py3
  • (/usr/bin/unpisi symlink pointing to /usr/bin/uneopkg.py3)

Implementation

Status: Phase 2 complete, Phase 3 in testing, Phase 4 being researched

Test Plan

• eopkg.bin ar, dr, er, info, hi, hi -t, it, lr, rm, rmf, search, up, ur
• eopkg.py2 ar, dr, er, info, hi, hi -t, it, lr, rm, rmf, search, up, ur
• eopkg.py3 ar, dr, er, info, hi, hi -t, it, lr, rm, rmf, search, up, ur
• Use Solus SC to install a package from 3rd party
• Use KDE Discover and GNOME Software to install, remove and upgrade packages
• Use solbuild to build packages with a bumped+rebuilt ypkg that is built against the updated phase2 python-eopkg package

Feedback and discussion

Feedback and discussion of implementation gotchas etc. are most welcome.

Let's make absolutely sure we hit this one out of the park!