eopkg: Handle the py2-based pisi to py3-based eopkg migration as robustly as possible #2193
Labels
Blocked
This is blocked on something else
Feature
Something can be enhanced.
Help Wanted
Plumbing
Core components
Priority: High
High priority
Tooling
Tools and scripts
Milestone
Motivation
Py2 was designated EOL in 2020. We need to move off py2 for our core tooling.
Blocked-by
Currently, the actual py2 to py3 migration is blocked by the following tasks:
Note that the PackageKit eopkg4 branch we use for our development is intended to work on both py2 and py3, so finishing this also unblocks py3 PK integration.
Design Considerations
The transition off of depending on python2 for Solus tooling will be split up into four phases (1+2 before the /usr merge epoch bump, 3+4 after the /usr merge epoch bump).
The transition is currently planned to look like this:
1. Land unified eopkg4 py3-based package
Status: DONE.
Get the unified eopkg4 (py3) package landed in the Solus repo, which includes a nuitka-compiled eopkg.bin executable that's written in python3, but due to the magic of the nuitka compiler, only depends on glibc.
2. Make some clever adjustments to the eopkg4 and pisi solus packages
Status: DONE. PR here.
In the 2nd phase, we update the eopkg4 (py3), pisi (py2) and solbuild packages to:
This makes it possible to have the eopkg4 package (contains eopkg.bin), the python-eopkg package (contains eopkg.py3, lseopkg->lseopkg.py3, and uneopkg->uneopkg.py3), and the pisi package (contains eopkg.py2 and a bunch of other scripts with the .py2 extension) all be co-installable while being able to compare functionality for them.
3. Interlude -- /usr-merge and 4.6 ISO release
Status: DONE (/usr-merge script in prod, 4.6 ISO released)
Tracker issues/PRs:
The short version is that the 4.6 ISO release will need to contain co-installed eopkg4 (py3) and pisi (py2) packages. The ISO script was recently updated to do a /usr-merge run during the ISO generation itself, thus ensuring that all ISO created from now on are /usr-merged.
We also got rid of the requirement for AppArmor patches to be present, which means that we can begin working on ISO generation automation in earnest.
solbuild
will be updated to calleopkg.py3 build
for 3rd party packaging wrapper operations (upstream solbuild repo has a draft PR up, which will be landed at the requisite time)Prerequisites for phase 4 readiness
eopkg.py2
toeopkg.py3
(pisi/eopkg: Add versioned FilesDB and LazyDB instances #4063)4. Epoch bump, remove py2 (including pisi and Solus SC), switch to utf-8 package strings
Status: TBD
eopkg
andeopkg-cli
symlinks, which will both point toeopkg.bin
eopkg.bin
This will be deployed in the same way as the usr-merge as a script, but this script will only run after a successful /usr-merge run. Because we set things up so that /usr-merged systems are guaranteed to have eopkg4 installed as part of system.base, this phase will use
eopkg.bin
to remove the pisi and solus-sc packages (and their dependencies) from the system during early boot via a systemd oneshot script.The reason we can do it this way, is that we were recently lucky enough to figure out how to fix eopkg4 to use the protocol=2 py2 compatible pickle cache version. This means that people who did their last update to their systems with Solus SC (py2), will still be left with a readable pickle cache when first the /usr-merge script and then the epoch bump script run using
eopkg.bin
.Testing the process
Status: TBD
It is worth mentioning that ^ can all be tested in a VM by using e.g. a Solus 4.5 ISO as the starting point, then making a snapshot of it before any updates are applied, at which point the testing process is trivially repeatable by rolling back to this initial snapshot.
Here, the "process" is "update through phase 1 and 2, do the phase 3 /usr-merge system and the phase 4 epoch jump, then watch the system still work".
What then?
At this point (post the /usr-merge and phase 4 epoch bump), Solus is using eopkg.bin (py3) and utf-8 string in packages, and solbuild is using nuitka-compiled py3 eopkg.bin and ypkg binaries. Python2 packages are no longer present in the repo.
Any ISO released after phase 4 is completed will contain no python2 packages.
Originally, I had proposed to do a second epoch jump after phase 4, but sheepman pointed out that this might not be necessary, which led us to the process described above.
Solus Packaging Context
The pisi package (before phase 1+2) contains the following binaries in
/usr/bin
:All (or at least most) of these binaries would need to be replaced by a combination of pure py3 executables (with the extension .py3) and nuitka-compiled py3 executables (with the extension .bin), which would in turn be symlinked to the correct executable artefact names per above.
Proposed Solution for making eopkg4 the "main" eopkg package (phase 4)
In packaging terms, this implies that the pisi package will be the "backup" package that is co-installable with (and depends on) eopkg4 (and not the other way around):
/usr/bin/eopkg.bin
/usr/share/PackageKit/helpers/eopkg/eopkgBackend.bin
/usr/share/defaults/eopkg
/usr/share/man/man1/eopkg.1
/usr/bin/eopkg
symlink pointing to/usr/bin/eopkg.bin
/usr/bin/eopkg-cli
symlink pointing to/usr/bin/eopkg.bin
/usr/lib/python3.11/site-packages/eopkg*
/usr/lib/python3.11/site-packages/pisi/
/usr/bin/check-newconfigs.py3
(TBD)/usr/bin/check-newconfigs.py
symlink pointing to/usr/bin/check-newconfigs.py3
(TBD)/usr/bin/eopkg.py3
(useful to have around for performance regression testing)/usr/bin/lseopkg.py3
/usr/bin/lseopkg
symlink pointing to/usr/bin/lseopkg.py3
/usr/bin/lspisi
symlink pointing to/usr/bin/lseopkg.py3
)/usr/bin/uneopkg.py3
/usr/bin/uneopkg
symlink pointing to/usr/bin/uneopkg.py3
/usr/bin/unpisi
symlink pointing to/usr/bin/uneopkg.py3
)Implementation
Status: Phase 2 complete, Phase 3 in testing, Phase 4 being researched
Test Plan
eopkg.bin ar, dr, er, info, hi, hi -t, it, lr, rm, rmf, search, up, ur
eopkg.py2 ar, dr, er, info, hi, hi -t, it, lr, rm, rmf, search, up, ur
eopkg.py3 ar, dr, er, info, hi, hi -t, it, lr, rm, rmf, search, up, ur
solbuild
to build packages with a bumped+rebuiltypkg
that is built against the updated phase2python-eopkg
packageFeedback and discussion
Feedback and discussion of implementation gotchas etc. are most welcome.
Let's make absolutely sure we hit this one out of the park!
The text was updated successfully, but these errors were encountered: