From 4586f7b24d463782559210434aaadc379cca8557 Mon Sep 17 00:00:00 2001 From: Isuru Fernando Date: Tue, 15 Oct 2024 10:52:58 -0400 Subject: [PATCH 1/9] Add a blog post about python noarch variants --- blog/2024-10-15-python-noarch-variants.md | 64 +++++++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 blog/2024-10-15-python-noarch-variants.md diff --git a/blog/2024-10-15-python-noarch-variants.md b/blog/2024-10-15-python-noarch-variants.md new file mode 100644 index 0000000000..ad92ccc8c7 --- /dev/null +++ b/blog/2024-10-15-python-noarch-variants.md @@ -0,0 +1,64 @@ +--- +authors: + - isuruf +tags: [infrastructure] +--- + +# Noarch variant packages for python packages on conda-forge + +conda-forge packages have always been batteries included. When +a package has some build options by default to reduce dependencies +we have enabled these options to give the most functionality and +and performance to our users. + +In the python world, some packages are written in C/C++/Cython +to get the most performance out of a package. However these packages +have reference implementation written in python. The python +reference implementation is a good way to check the C/C++/Cython +code against a much simpler python implementation and is also +useful for platforms like PyPy where the C/C++/Cython implementation +can be slower than the python reference implementation due to the +emulation of the Python C/C++ API by PyPy. For eg: for the Cython +package, setting ``CYTHON_NO_COMPILE`` environment variable +when building the cython wheel itself, it will use the python reference +implementation. + +To support platforms like PyPy, Some packages build wheels with +compiled extensions for the platforms that are +known to be more performant with the compiled extension, but also +provide a universal pure python wheel for the other platforms. +This also provides a way for new python versions and variants +like the freethreading python build to use these packages by the +early adpoters of these python versions. + +On conda-forge we usually have compiled python packages, but provide +no reference implementation. This makes early adpoters of new python +versions to wait for the conda-forge bot managed by @conda-forge/bot +team to start the migration and rebuild the packages. For eg: the +freethreading python 3.13 build is still not under way because +conda-forge has decided to wait until the default (GIL enabled) +python 3.13 build has migrated enough and upstream packages have added +support for freethreading builds. +Another issue is that some packages have cyclic dependencies at build +or test time and this requires manual handling to reduce dependencies +before the migration and add the dependencies later on. + +We have been adding ``noarch: python`` variants for some feedstocks +so that the compiled extension has higher priority and the pure +python extension has lower priority which makes the conda solver +use the ``noarch: python`` variant. One issue is that the linter +might not like selectors on noarch recipes. We added an option + +```yaml +linter: + skip: + - lint_noarch_selectors +``` +to ``conda-forge.yml`` that will make the linter skip this warning/error. + +We list some PRs here as a reference for conda-forge maintainers who +want to help out with this effort. + +- [coverage](https://github.com/conda-forge/coverage-feedstock/pull/123) +- [cython](https://github.com/conda-forge/cython-feedstock/pull/147) +- [aiohttp](https://github.com/conda-forge/aiohttp-feedstock/pull/99) From bd3b47806e4304d9ed6d3fef81255fd7da5ce7a0 Mon Sep 17 00:00:00 2001 From: Isuru Fernando Date: Tue, 15 Oct 2024 11:59:46 -0400 Subject: [PATCH 2/9] fix typos --- blog/2024-10-15-python-noarch-variants.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/blog/2024-10-15-python-noarch-variants.md b/blog/2024-10-15-python-noarch-variants.md index ad92ccc8c7..79621c3146 100644 --- a/blog/2024-10-15-python-noarch-variants.md +++ b/blog/2024-10-15-python-noarch-variants.md @@ -7,9 +7,9 @@ tags: [infrastructure] # Noarch variant packages for python packages on conda-forge conda-forge packages have always been batteries included. When -a package has some build options by default to reduce dependencies -we have enabled these options to give the most functionality and -and performance to our users. +a package has some build options turned off by default to reduce +dependencies we have enabled these options to give the most +functionality and performance to our users. In the python world, some packages are written in C/C++/Cython to get the most performance out of a package. However these packages From f9a331a4e43cf9b939f96aaf4881d492fddf3c57 Mon Sep 17 00:00:00 2001 From: Isuru Fernando Date: Tue, 15 Oct 2024 12:07:10 -0400 Subject: [PATCH 3/9] pre-commit --- blog/2024-10-15-python-noarch-variants.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/blog/2024-10-15-python-noarch-variants.md b/blog/2024-10-15-python-noarch-variants.md index 79621c3146..4358c908ab 100644 --- a/blog/2024-10-15-python-noarch-variants.md +++ b/blog/2024-10-15-python-noarch-variants.md @@ -19,7 +19,7 @@ code against a much simpler python implementation and is also useful for platforms like PyPy where the C/C++/Cython implementation can be slower than the python reference implementation due to the emulation of the Python C/C++ API by PyPy. For eg: for the Cython -package, setting ``CYTHON_NO_COMPILE`` environment variable +package, setting `CYTHON_NO_COMPILE` environment variable when building the cython wheel itself, it will use the python reference implementation. @@ -43,10 +43,10 @@ Another issue is that some packages have cyclic dependencies at build or test time and this requires manual handling to reduce dependencies before the migration and add the dependencies later on. -We have been adding ``noarch: python`` variants for some feedstocks +We have been adding `noarch: python` variants for some feedstocks so that the compiled extension has higher priority and the pure python extension has lower priority which makes the conda solver -use the ``noarch: python`` variant. One issue is that the linter +use the `noarch: python` variant. One issue is that the linter might not like selectors on noarch recipes. We added an option ```yaml @@ -54,7 +54,8 @@ linter: skip: - lint_noarch_selectors ``` -to ``conda-forge.yml`` that will make the linter skip this warning/error. + +to `conda-forge.yml` that will make the linter skip this warning/error. We list some PRs here as a reference for conda-forge maintainers who want to help out with this effort. From 65389c16122bacfac9bd18180321e561982629b6 Mon Sep 17 00:00:00 2001 From: Isuru Fernando Date: Tue, 15 Oct 2024 12:24:08 -0400 Subject: [PATCH 4/9] Apply suggestions from code review Co-authored-by: jaimergp --- blog/2024-10-15-python-noarch-variants.md | 32 +++++++++++------------ 1 file changed, 16 insertions(+), 16 deletions(-) diff --git a/blog/2024-10-15-python-noarch-variants.md b/blog/2024-10-15-python-noarch-variants.md index 4358c908ab..fb297be8d8 100644 --- a/blog/2024-10-15-python-noarch-variants.md +++ b/blog/2024-10-15-python-noarch-variants.md @@ -4,48 +4,48 @@ authors: tags: [infrastructure] --- -# Noarch variant packages for python packages on conda-forge +# Noarch variant packages for Python packages on conda-forge conda-forge packages have always been batteries included. When a package has some build options turned off by default to reduce -dependencies we have enabled these options to give the most +dependencies, we have enabled these options to give the most functionality and performance to our users. -In the python world, some packages are written in C/C++/Cython +In the Python world, some packages are written in C/C++/Cython to get the most performance out of a package. However these packages -have reference implementation written in python. The python +have a reference implementation written in Python. The Python reference implementation is a good way to check the C/C++/Cython code against a much simpler python implementation and is also useful for platforms like PyPy where the C/C++/Cython implementation -can be slower than the python reference implementation due to the +can be slower than the Python reference implementation due to the emulation of the Python C/C++ API by PyPy. For eg: for the Cython package, setting `CYTHON_NO_COMPILE` environment variable -when building the cython wheel itself, it will use the python reference +when building the Cython wheel itself, it will use the Python reference implementation. To support platforms like PyPy, Some packages build wheels with compiled extensions for the platforms that are known to be more performant with the compiled extension, but also -provide a universal pure python wheel for the other platforms. -This also provides a way for new python versions and variants -like the freethreading python build to use these packages by the -early adpoters of these python versions. +provide a universal pure Python wheel for the other platforms. +This also provides a way for new Python versions and variants +like the free-threading Python build to use these packages by the +early adopters of these Python versions. -On conda-forge we usually have compiled python packages, but provide -no reference implementation. This makes early adpoters of new python +On conda-forge we usually have compiled Python packages, but provide +no reference implementation. This makes early adopters of new Python versions to wait for the conda-forge bot managed by @conda-forge/bot team to start the migration and rebuild the packages. For eg: the -freethreading python 3.13 build is still not under way because +free-threading Python 3.13 build is still not under way because conda-forge has decided to wait until the default (GIL enabled) -python 3.13 build has migrated enough and upstream packages have added -support for freethreading builds. +Python 3.13 build has migrated enough and upstream packages have added +support for free-threading builds. Another issue is that some packages have cyclic dependencies at build or test time and this requires manual handling to reduce dependencies before the migration and add the dependencies later on. We have been adding `noarch: python` variants for some feedstocks so that the compiled extension has higher priority and the pure -python extension has lower priority which makes the conda solver +Python extension has lower priority which makes the conda solver use the `noarch: python` variant. One issue is that the linter might not like selectors on noarch recipes. We added an option From 630ee4f81088fa2c49be2a454563266eb6c9a32e Mon Sep 17 00:00:00 2001 From: Isuru Fernando Date: Tue, 15 Oct 2024 12:26:40 -0400 Subject: [PATCH 5/9] Add a short paragraph --- blog/2024-10-15-python-noarch-variants.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/blog/2024-10-15-python-noarch-variants.md b/blog/2024-10-15-python-noarch-variants.md index fb297be8d8..add4a64e58 100644 --- a/blog/2024-10-15-python-noarch-variants.md +++ b/blog/2024-10-15-python-noarch-variants.md @@ -6,6 +6,13 @@ tags: [infrastructure] # Noarch variant packages for Python packages on conda-forge +We introduce noarch variants for python packages on conda-forge +that have compiled extensions but with pure python reference +implementations to make life easier for early adoopters of +new python variants. + + + conda-forge packages have always been batteries included. When a package has some build options turned off by default to reduce dependencies, we have enabled these options to give the most From fa71ef70ac423b0a3cae0b9ea07499f44ccd64eb Mon Sep 17 00:00:00 2001 From: Isuru Fernando Date: Tue, 15 Oct 2024 14:01:59 -0400 Subject: [PATCH 6/9] Fix typo Co-authored-by: Filipe --- blog/2024-10-15-python-noarch-variants.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/2024-10-15-python-noarch-variants.md b/blog/2024-10-15-python-noarch-variants.md index add4a64e58..1d8fdef380 100644 --- a/blog/2024-10-15-python-noarch-variants.md +++ b/blog/2024-10-15-python-noarch-variants.md @@ -8,7 +8,7 @@ tags: [infrastructure] We introduce noarch variants for python packages on conda-forge that have compiled extensions but with pure python reference -implementations to make life easier for early adoopters of +implementations to make life easier for early adopters of new python variants. From 4bdd4ad78882efd80d65d4796ea24e3f55b7a197 Mon Sep 17 00:00:00 2001 From: Isuru Fernando Date: Wed, 16 Oct 2024 10:33:09 -0400 Subject: [PATCH 7/9] Add more info --- blog/2024-10-15-python-noarch-variants.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/blog/2024-10-15-python-noarch-variants.md b/blog/2024-10-15-python-noarch-variants.md index 1d8fdef380..72aec6fd08 100644 --- a/blog/2024-10-15-python-noarch-variants.md +++ b/blog/2024-10-15-python-noarch-variants.md @@ -20,7 +20,7 @@ functionality and performance to our users. In the Python world, some packages are written in C/C++/Cython to get the most performance out of a package. However these packages -have a reference implementation written in Python. The Python +sometimes have a reference implementation written in Python. The Python reference implementation is a good way to check the C/C++/Cython code against a much simpler python implementation and is also useful for platforms like PyPy where the C/C++/Cython implementation @@ -28,7 +28,10 @@ can be slower than the Python reference implementation due to the emulation of the Python C/C++ API by PyPy. For eg: for the Cython package, setting `CYTHON_NO_COMPILE` environment variable when building the Cython wheel itself, it will use the Python reference -implementation. +implementation. The only way to figure out if a package has a Python +reference implementation is to look at `setup.py` on packages that +provide such a file to see if `extensions` are built always or +with a switch. To support platforms like PyPy, Some packages build wheels with compiled extensions for the platforms that are From 1c634804f1c6a95dda70811157cefc078f402da0 Mon Sep 17 00:00:00 2001 From: Isuru Fernando Date: Wed, 16 Oct 2024 14:37:36 -0400 Subject: [PATCH 8/9] empty commit From d1aa79a2208bf89cd93d214e4ff76b572a077088 Mon Sep 17 00:00:00 2001 From: Isuru Fernando Date: Mon, 21 Oct 2024 08:44:21 -0400 Subject: [PATCH 9/9] Apply suggestions from code review Co-authored-by: jakirkham --- blog/2024-10-15-python-noarch-variants.md | 24 +++++++++++------------ 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/blog/2024-10-15-python-noarch-variants.md b/blog/2024-10-15-python-noarch-variants.md index 72aec6fd08..c26fa1bd02 100644 --- a/blog/2024-10-15-python-noarch-variants.md +++ b/blog/2024-10-15-python-noarch-variants.md @@ -25,13 +25,12 @@ reference implementation is a good way to check the C/C++/Cython code against a much simpler python implementation and is also useful for platforms like PyPy where the C/C++/Cython implementation can be slower than the Python reference implementation due to the -emulation of the Python C/C++ API by PyPy. For eg: for the Cython +emulation of the Python C/C++ API by PyPy. For example for the Cython package, setting `CYTHON_NO_COMPILE` environment variable when building the Cython wheel itself, it will use the Python reference implementation. The only way to figure out if a package has a Python -reference implementation is to look at `setup.py` on packages that -provide such a file to see if `extensions` are built always or -with a switch. +reference implementation is to look at the library's source code +to see if `extensions` are optional. To support platforms like PyPy, Some packages build wheels with compiled extensions for the platforms that are @@ -42,16 +41,15 @@ like the free-threading Python build to use these packages by the early adopters of these Python versions. On conda-forge we usually have compiled Python packages, but provide -no reference implementation. This makes early adopters of new Python -versions to wait for the conda-forge bot managed by @conda-forge/bot -team to start the migration and rebuild the packages. For eg: the -free-threading Python 3.13 build is still not under way because -conda-forge has decided to wait until the default (GIL enabled) -Python 3.13 build has migrated enough and upstream packages have added -support for free-threading builds. +no reference implementation. This means early adopters of new Python +versions need to wait for the conda-forge bot managed by @conda-forge/bot +team to start the migration and rebuild the packages. For example the +free-threading Python 3.13 build is still paused as +conda-forge has decided to focus on the default (GIL enabled) +Python 3.13 build first while upstream packages work on +supporting free-threading. Another issue is that some packages have cyclic dependencies at build -or test time and this requires manual handling to reduce dependencies -before the migration and add the dependencies later on. +or test time and this requires some manual handling. We have been adding `noarch: python` variants for some feedstocks so that the compiled extension has higher priority and the pure