From 639da12398bea4781af5dc0a34ec228391b2ef3c Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 24 Jul 2023 11:03:18 +0200 Subject: [PATCH 01/14] mention other archs in intro (#1534) --- sphinx/src/user/introduction.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/sphinx/src/user/introduction.rst b/sphinx/src/user/introduction.rst index 8772816c32..83492b8c98 100644 --- a/sphinx/src/user/introduction.rst +++ b/sphinx/src/user/introduction.rst @@ -34,6 +34,7 @@ conda-forge is a community effort that tackles these issues: - Care is taken that all packages are up-to-date. - Common standards ensure that all packages have compatible versions. - By default, we build packages for macOS, Linux AMD64 and Windows AMD64. + Other architectures are also available on request (e.g. Apple Silicon, PowerPC, Linux ARM). - Many packages are updated by multiple maintainers with an easy option to become a maintainer. - An active core developer team is trying to also maintain abandoned packages. From df299ca56c5ef5d9a6ab65280121b1fef34a3513 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 24 Jul 2023 12:30:20 +0200 Subject: [PATCH 02/14] document cross-compilation a bit more --- sphinx/src/maintainer/knowledge_base.rst | 101 ++++++++++++++++++++++- 1 file changed, 97 insertions(+), 4 deletions(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index c475dd6c9a..47682aec3a 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -303,14 +303,68 @@ A package that needs all five compilers would define there's no need to specify ``libgcc`` or ``libgfortran``. There are additional informations about how conda-build 3 treats compilers in the `conda docs `__. +.. cross-compilation: + Cross-compilation ----------------- For some other architectures (like ARM), packages can be built natively on that architecture or they can be cross-compiled. -In other words built on a different common architecture (like x86_64) while still targeting the original architecture (ARM). +In other words, built on a different common architecture (like x86_64) while still targeting the original architecture (ARM). This helps one leverage more abundant CI resources in the build architecture (x86_64). -A package needs to make a few changes in their recipe to be compatible with cross-compilation. Here are a few examples. +Cross-compilation settings depend on the ``build_platform`` and ``target_platform`` conda-build +variables: + +- ``build_platform``: The platform on which ``conda-build`` is running, which defines the ``build`` + environment in ``$BUILD_PREFIX``. +- ``target_platform``: The platform on which the package will be installed. Defines the platform of + the ``host`` environment in ``$PREFIX``. Defaults to the value of ``build_platform``. + +To change the value of ``target_platform`` and enable cross-compilation, you must use +the :ref:`build_platform` mapping in ``conda-forge.yml`` and then :ref:`rerender +` the feedstock. This will generate the appropriate CI workflows and +conda-build input metadata. See also :ref:`test` for how to skip the test phase when +cross-compiling. Provided the requirements metadata and build scripts are written correctly, the +package should just work. However, in some cases, it'll need some adjustments; see examples below +for some common cases. + +.. note:: + + The ``build_platform`` and ``target_platform`` variables are exposed as environment variables in + the build scripts (e.g. ``$build_platform``), and also as Jinja variables in the ``meta.yaml`` + selectors (e.g. ``# [build_platform != target_platform]``). + +In addition to these two variables, there are some more environment variables that are set by +conda-forge's automation (e.g. ``conda-forge-ci-setup``, compiler activation packages, etc) that +can aid in cross-compilation setups: + +- ``CONDA_BUILD_CROSS_COMPILATION``: set to ``1`` when ``build_platform`` and ``target_platform`` + differ. +- ``BUILD_PLATFORM``: the platform on which ``conda-build`` is running. +- ``BUILD``: the autoconf triplet expected for build platform. +- ``HOST_PLATFORM``: the platform on which the package will be installed. +- ``HOST``: the autoconf triplet expected for host platform. +- ``CMAKE_ARGS``: arguments needed to cross-compile with CMake. Pass it to ``cmake`` in your build + script. +- ``MESON_ARGS``: arguments needed to cross-compile with Meson. Pass it to ``meson`` in your build + script. Note a `cross build definition file `__ is + automatically created for you too. +- ``CC_FOR_BUILD``: C compilers targeting the build platform. +- ``CXX_FOR_BUILD``: C++ compilers targeting the build platform. + +This is all supported by two main conda-build features introduced in version 3: + +- How `requirements metadata + `__ + is expressed in ``meta.yaml``, which distinguishes between ``build`` and ``host`` platforms. +- The ``compiler()`` Jinja function and underlying `conventions for the compiler packages + `__. + +Cross-compilation examples +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A package needs to make a few changes in their recipe to be compatible with cross-compilation. Here +are a few examples. A simple C library using autotools for cross-compilation might look like this: @@ -323,7 +377,8 @@ A simple C library using autotools for cross-compilation might look like this: - pkg-config - gnuconfig -In the build script, it would need to update the config files and guard any tests when cross-compiling: +In the build script, it would need to update the config files and guard any tests when +cross-compiling: .. code-block:: sh @@ -335,6 +390,44 @@ In the build script, it would need to update the config files and guard any test make check fi +A simple C++ library using CMake for cross-compilation might look like this: + +.. code-block:: yaml + + requirements: + build: + - {{ compiler("cxx") }} + - cmake + - make + +In the build script, it would need to update ``cmake`` call and guard any tests when cross-compiling: + +.. code-block:: sh + + # Pass ``CMAKE_ARGS`` to ``cmake`` + cmake ${CMAKE_ARGS} .. + + # Skip ``ctest`` when cross-compiling + if [[ "${CONDA_BUILD_CROSS_COMPILATION}" != "1" ]]; then + ctest + fi + +Similarly, with Meson: + +.. code-block:: yaml + + requirements: + build: + - {{ compiler("c") }} + - {{ compiler("cxx") }} + - meson + - make + +.. code-block:: sh + + # Pass ``MESON_ARGS`` to ``meson`` + meson ${MESON_ARGS} builddir/ + A simple Python extension using Cython and NumPy's C API would look like so: .. code-block:: yaml @@ -1705,7 +1798,7 @@ Apple Silicon builds The new Apple M1 processor is the first Apple Silicon supported by conda-forge `osx-arm64 `__ builds. -For new builds to be available, via cross-compilation, a migration is required for +For new builds to be available, via `cross-compilation `_, a migration is required for the package and its dependencies. These builds are experimental as many of them are untested. To request a migration for a particular package and all its dependencies: From cb3af4e2b615ad17004e26a256688832c46b926d Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 24 Jul 2023 13:16:33 +0200 Subject: [PATCH 03/14] add details about Python particularities --- sphinx/src/maintainer/knowledge_base.rst | 37 ++++++++++++++++++++++++ 1 file changed, 37 insertions(+) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index 47682aec3a..c652fc6f26 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -453,6 +453,43 @@ but merely to provide a starting point with some guidelines. Please look at `oth .. _other recipes for more examples: https://github.com/search?q=org%3Aconda-forge+path%3Arecipe%2Fmeta.yaml+%22%5Bbuild_platform+%21%3D+target_platform%5D%22&type=code +Details about cross-compiled Python packages +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Cross-compiling Python packages is a bit more involved than other packages. This is because Python +doesn't have official support for cross-compilation and a series of workarounds need to be applied +to make it work. See `PEP720 `__ for more details. + +On conda-forge, there are two extra metadata bits that are needed in ``meta.yaml``: + +- Adding ``cross-python_{{ target_platform }}`` in ``build`` requirements, provided by the + `cross-python-feedstock `__. This is a + wrapper for the ``crossenv`` Python interpreters with `some activation logic that adjust some of + the crossenv workarounds + `__ + so they work better with the conda-build setup. +- Copying some Python-related packages from ``host`` to ``build`` with a ``[build_platform != + target_platform]`` selector: + - ``python`` itself, to support ``crossenv``. + - Non-pure Python packages (i.e. they ship compiled libraries) that need to be present while the + package is being built, like ``cython`` and ``numpy``. + +In the terms of the `PEP720 `__, the conda-forge setup +implements the "faking the target environment" approach. More specifically: + +This will result in the following changes before the builds scripts run: + +- A modified ``crossenv`` installation in ``$BUILD_PREFIX/venv``, mimicking the architecture of + ``$PREFIX``. +- Forwarder binaries in ``$BUILD_PREFIX/bin`` that point to the ``crossenv`` installation. +- Symlinks that expose the ``$BUILD_PREFIX`` site-packages in the ``crossenv`` installation, which + is also included in ``$PYTHONPATH``. +- A copy of all ``$PREFIX`` site-packages to ``$BUILD_PREFIX`` (except the compiled libraries). + +All in all, this results in a setup where ``conda-build`` can run a ``$BUILD_PREFIX``-architecture +``python`` interpreter that can see the packages in ``$PREFIX`` (with the compiled bits provided by +their corresponding counterparts in ``$BUILD_PREFIX``) and mimic that target architecture. + Rust Nightly ------------ From 3404626bf556d95f2f504ab14a3d6c0e0b20f039 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 24 Jul 2023 13:26:41 +0200 Subject: [PATCH 04/14] fix errors --- sphinx/src/maintainer/knowledge_base.rst | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index c652fc6f26..08ac6bd743 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -412,7 +412,7 @@ In the build script, it would need to update ``cmake`` call and guard any tests ctest fi -Similarly, with Meson: +Similarly, with Meson, the ``meta.yaml`` needs: .. code-block:: yaml @@ -423,6 +423,8 @@ Similarly, with Meson: - meson - make +And this in ``build.sh``: + .. code-block:: sh # Pass ``MESON_ARGS`` to ``meson`` @@ -470,14 +472,14 @@ On conda-forge, there are two extra metadata bits that are needed in ``meta.yaml so they work better with the conda-build setup. - Copying some Python-related packages from ``host`` to ``build`` with a ``[build_platform != target_platform]`` selector: + - ``python`` itself, to support ``crossenv``. - Non-pure Python packages (i.e. they ship compiled libraries) that need to be present while the package is being built, like ``cython`` and ``numpy``. In the terms of the `PEP720 `__, the conda-forge setup -implements the "faking the target environment" approach. More specifically: - -This will result in the following changes before the builds scripts run: +implements the "faking the target environment" approach. More specifically, this will result in the +following changes before the builds scripts run: - A modified ``crossenv`` installation in ``$BUILD_PREFIX/venv``, mimicking the architecture of ``$PREFIX``. From eafa7efd694e80d5d47afd06e8449b041b28a02e Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 24 Jul 2023 13:30:56 +0200 Subject: [PATCH 05/14] use ref --- sphinx/src/maintainer/knowledge_base.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index 08ac6bd743..5935975708 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -1837,7 +1837,7 @@ Apple Silicon builds The new Apple M1 processor is the first Apple Silicon supported by conda-forge `osx-arm64 `__ builds. -For new builds to be available, via `cross-compilation `_, a migration is required for +For new builds to be available, via :ref:`cross-compilation `, a migration is required for the package and its dependencies. These builds are experimental as many of them are untested. To request a migration for a particular package and all its dependencies: From 190546ba07c492a083cfdb1ca64ff3fc24890f37 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Mon, 24 Jul 2023 13:34:15 +0200 Subject: [PATCH 06/14] missing underscore in ref --- sphinx/src/maintainer/knowledge_base.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index 5935975708..780333ca70 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -303,7 +303,7 @@ A package that needs all five compilers would define there's no need to specify ``libgcc`` or ``libgfortran``. There are additional informations about how conda-build 3 treats compilers in the `conda docs `__. -.. cross-compilation: +.. _cross-compilation: Cross-compilation ----------------- From 652db2c8a776dc6e853da8696e42310548a67a8c Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 25 Jul 2023 10:27:27 +0200 Subject: [PATCH 07/14] Apply suggestions from code review Co-authored-by: Isuru Fernando --- sphinx/src/maintainer/knowledge_base.rst | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index 780333ca70..7a6a39f224 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -340,10 +340,8 @@ can aid in cross-compilation setups: - ``CONDA_BUILD_CROSS_COMPILATION``: set to ``1`` when ``build_platform`` and ``target_platform`` differ. -- ``BUILD_PLATFORM``: the platform on which ``conda-build`` is running. -- ``BUILD``: the autoconf triplet expected for build platform. -- ``HOST_PLATFORM``: the platform on which the package will be installed. -- ``HOST``: the autoconf triplet expected for host platform. +- ``CONDA_TOOLCHAIN_BUILD``: the autoconf triplet expected for build platform. +- ``CONDA_TOOLCHAIN_HOST``: the autoconf triplet expected for host platform. - ``CMAKE_ARGS``: arguments needed to cross-compile with CMake. Pass it to ``cmake`` in your build script. - ``MESON_ARGS``: arguments needed to cross-compile with Meson. Pass it to ``meson`` in your build From 864798217f5b04de773b51a46d22431336ff22e4 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 25 Jul 2023 18:11:17 +0200 Subject: [PATCH 08/14] Update src/maintainer/knowledge_base.rst Co-authored-by: Isuru Fernando --- sphinx/src/maintainer/knowledge_base.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index 7a6a39f224..908315587c 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -406,7 +406,7 @@ In the build script, it would need to update ``cmake`` call and guard any tests cmake ${CMAKE_ARGS} .. # Skip ``ctest`` when cross-compiling - if [[ "${CONDA_BUILD_CROSS_COMPILATION}" != "1" ]]; then + if [[ "${CONDA_BUILD_CROSS_COMPILATION:-}" != "1" || "${CROSSCOMPILING_EMULATOR:-}" != "" ]]; then ctest fi From a07252473cf507752c452246c1f0b9d37ef51956 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 25 Jul 2023 18:39:17 +0200 Subject: [PATCH 09/14] add some more notes from https://github.com/conda-forge/conda-forge.github.io/issues/1841 Co-authored-by: ahesford --- sphinx/src/maintainer/knowledge_base.rst | 54 ++++++++++++++++++++---- 1 file changed, 45 insertions(+), 9 deletions(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index 908315587c..72225e768c 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -308,9 +308,29 @@ A package that needs all five compilers would define Cross-compilation ----------------- -For some other architectures (like ARM), packages can be built natively on that architecture or they can be cross-compiled. -In other words, built on a different common architecture (like x86_64) while still targeting the original architecture (ARM). -This helps one leverage more abundant CI resources in the build architecture (x86_64). +conda-forge defaults to native builds of packages for x86_64 on Linux, macOS and Windows, because +that's the architecture powering the default CI runners. Other architectures are supported too, +but they are not guaranteed to have native builds. In those platforms where we can't provide native +CI runners, we can still resort to either cross-compilation or emulation. + +Cross-compiling means building a package for a different architecture than the one the build process +is running on. Given how abundant x86_64 runners are, most common cross-compilation setups will target +non-x86_64 architectures from x86_64 runners. + +Cross-compilation terminology usually distinguishes between two types of machine: + +- Build: The machine running the building process. +- Host: The machine we are building packages for. + +.. note:: + + Some cross-compilation documentation might also distinguish between a third type of machine, the + target machine. You can read more about it in `this Stack Overflow question + `__. + For the purposes of conda-forge, we'll consider the target machine to be the same as the host. + +How to enable cross-compilation +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Cross-compilation settings depend on the ``build_platform`` and ``target_platform`` conda-build variables: @@ -358,6 +378,17 @@ This is all supported by two main conda-build features introduced in version 3: - The ``compiler()`` Jinja function and underlying `conventions for the compiler packages `__. +Placing requirements in build or host +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The rule of the thumb is: + +- If it needs to run during the build, it goes in ``build``. +- If it needs to be available on the target host, it goes in ``host``. +- If both conditions are true, it belongs in both. + +However, there are some exceptions to this rule; most notably Python cross-compilation (see below). + Cross-compilation examples ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -384,7 +415,7 @@ cross-compiling: cp $BUILD_PREFIX/share/gnuconfig/config.* . # Skip ``make check`` when cross-compiling - if [[ "${CONDA_BUILD_CROSS_COMPILATION}" != "1" ]]; then + if [[ "${CONDA_BUILD_CROSS_COMPILATION:-}" != "1" || "${CROSSCOMPILING_EMULATOR:-}" != "" ]]; then make check fi @@ -456,11 +487,15 @@ but merely to provide a starting point with some guidelines. Please look at `oth Details about cross-compiled Python packages ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Cross-compiling Python packages is a bit more involved than other packages. This is because Python -doesn't have official support for cross-compilation and a series of workarounds need to be applied -to make it work. See `PEP720 `__ for more details. +Cross-compiling Python packages is a bit more involved than other packages. The main pain point is +that we need an executable Python interpreter (i.e. ``python`` in ``build``) that knows how to +provide accurate information about the target platform. Since this is not officially supported, a +series of workarounds are required to make it work. Refer to `PEP720 +`__ or `the discussion in this issue +`__ for more information. -On conda-forge, there are two extra metadata bits that are needed in ``meta.yaml``: +In practical terms, for conda-forge, this results into two extra metadata bits that are needed in +``meta.yaml``: - Adding ``cross-python_{{ target_platform }}`` in ``build`` requirements, provided by the `cross-python-feedstock `__. This is a @@ -488,7 +523,8 @@ following changes before the builds scripts run: All in all, this results in a setup where ``conda-build`` can run a ``$BUILD_PREFIX``-architecture ``python`` interpreter that can see the packages in ``$PREFIX`` (with the compiled bits provided by -their corresponding counterparts in ``$BUILD_PREFIX``) and mimic that target architecture. +their corresponding counterparts in ``$BUILD_PREFIX``) and sufficiently mimic that target +architecture. Rust Nightly ------------ From 73766bbe298a88daa98b732380fc07dc319bdba3 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 25 Jul 2023 18:54:45 +0200 Subject: [PATCH 10/14] add emulation docs --- sphinx/src/maintainer/knowledge_base.rst | 42 +++++++++++++++++++++++- 1 file changed, 41 insertions(+), 1 deletion(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index 72225e768c..58ffe9b252 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -387,7 +387,8 @@ The rule of the thumb is: - If it needs to be available on the target host, it goes in ``host``. - If both conditions are true, it belongs in both. -However, there are some exceptions to this rule; most notably Python cross-compilation (see below). +However, there are some exceptions to this rule; most notably Python cross-compilation +(:ref:`see below `). Cross-compilation examples ^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -484,6 +485,8 @@ but merely to provide a starting point with some guidelines. Please look at `oth .. _other recipes for more examples: https://github.com/search?q=org%3Aconda-forge+path%3Arecipe%2Fmeta.yaml+%22%5Bbuild_platform+%21%3D+target_platform%5D%22&type=code +.. _python_cross_compilation: + Details about cross-compiled Python packages ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -526,6 +529,43 @@ All in all, this results in a setup where ``conda-build`` can run a ``$BUILD_PRE their corresponding counterparts in ``$BUILD_PREFIX``) and sufficiently mimic that target architecture. +.. _emulation: + +Emulated builds +--------------- + +When cross-compilation is not possible, one can resort to emulation. This is a technique that uses +a virtual machine (`QEMU `__) to emulate the target platform, which has a +significant overhead. However, ``conda-build`` will see the target platform as native, so very +little changes are usually needed in the recipe. + +To enable emulated builds, you must use the :ref:`provider` mapping in ``conda-forge.yml``. +This key maps a ``build_platform`` to a ``provider`` that will be used to emulate the platform. +``conda-smithy`` will know how to detect whether the provider supports that platform natively or +requires emulation, and will adjust the appropriate CI steps to ensure that QEMU runs the process. +Ensure changes are applied by :ref:`rerendering ` the feedstock. + +.. warning:: + + Emulated builds are very slow and incur an additional strain on conda-forge CI resources. + Whenever possible, please consider cross-compilation instead. Only use emulated builds as a last + resort. + +Emulation examples +^^^^^^^^^^^^^^^^^^ + +Configure ``conda-forge.yml`` to emulate ``linux-ppc64le``, but use native runners for ``linux-64`` +and ``linux-aarch64``. This works because ``linux-ppc64le`` is not natively supported by Azure, so +``conda-smithy`` will add QEMU steps to emulate it. However, ``linux-64`` and ``linux-aarch64`` are +natively supported by Azure and Travis CI, respectively, so no emulation is needed. + +.. code-block:: yaml + + provider: + linux_aarch64: travis + linux_ppc64le: azure + linux_64: azure + Rust Nightly ------------ From 8d03df87f760102669339cc387167da28aa45f54 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 25 Jul 2023 19:19:31 +0200 Subject: [PATCH 11/14] document CROSSCOMPILING_EMULATOR --- sphinx/src/maintainer/knowledge_base.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index 58ffe9b252..b9f1866c39 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -369,6 +369,8 @@ can aid in cross-compilation setups: automatically created for you too. - ``CC_FOR_BUILD``: C compilers targeting the build platform. - ``CXX_FOR_BUILD``: C++ compilers targeting the build platform. +- ``CROSSCOMPILING_EMULATOR``: the emulator to use when emulating the target platform. This is + usually set to the ``qemu`` binary for the host platform. This is all supported by two main conda-build features introduced in version 3: From 3e69da7c85c6920212a9a931f50bfd483da6e168 Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 25 Jul 2023 19:38:49 +0200 Subject: [PATCH 12/14] backlink to env vars in cross-compilation --- sphinx/src/maintainer/knowledge_base.rst | 31 ++++++++++++++++-------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index b9f1866c39..ab32d8111c 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -325,10 +325,12 @@ Cross-compilation terminology usually distinguishes between two types of machine .. note:: Some cross-compilation documentation might also distinguish between a third type of machine, the - target machine. You can read more about it in `this Stack Overflow question + target machine. You can read more about it in `this Stack Overflow question `__. For the purposes of conda-forge, we'll consider the target machine to be the same as the host. +.. _cross_compilation_howto: + How to enable cross-compilation ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -369,8 +371,8 @@ can aid in cross-compilation setups: automatically created for you too. - ``CC_FOR_BUILD``: C compilers targeting the build platform. - ``CXX_FOR_BUILD``: C++ compilers targeting the build platform. -- ``CROSSCOMPILING_EMULATOR``: the emulator to use when emulating the target platform. This is - usually set to the ``qemu`` binary for the host platform. +- ``CROSSCOMPILING_EMULATOR``: Path to the ``qemu`` binary for the host platform (see + :ref:`emulation`). This is all supported by two main conda-build features introduced in version 3: @@ -438,7 +440,7 @@ In the build script, it would need to update ``cmake`` call and guard any tests # Pass ``CMAKE_ARGS`` to ``cmake`` cmake ${CMAKE_ARGS} .. - + # Skip ``ctest`` when cross-compiling if [[ "${CONDA_BUILD_CROSS_COMPILATION:-}" != "1" || "${CROSSCOMPILING_EMULATOR:-}" != "" ]]; then ctest @@ -507,11 +509,11 @@ In practical terms, for conda-forge, this results into two extra metadata bits t wrapper for the ``crossenv`` Python interpreters with `some activation logic that adjust some of the crossenv workarounds `__ - so they work better with the conda-build setup. + so they work better with the conda-build setup. - Copying some Python-related packages from ``host`` to ``build`` with a ``[build_platform != target_platform]`` selector: - - ``python`` itself, to support ``crossenv``. + - ``python`` itself, to support ``crossenv``. - Non-pure Python packages (i.e. they ship compiled libraries) that need to be present while the package is being built, like ``cython`` and ``numpy``. @@ -526,7 +528,7 @@ following changes before the builds scripts run: is also included in ``$PYTHONPATH``. - A copy of all ``$PREFIX`` site-packages to ``$BUILD_PREFIX`` (except the compiled libraries). -All in all, this results in a setup where ``conda-build`` can run a ``$BUILD_PREFIX``-architecture +All in all, this results in a setup where ``conda-build`` can run a ``$BUILD_PREFIX``-architecture ``python`` interpreter that can see the packages in ``$PREFIX`` (with the compiled bits provided by their corresponding counterparts in ``$BUILD_PREFIX``) and sufficiently mimic that target architecture. @@ -556,7 +558,7 @@ Ensure changes are applied by :ref:`rerendering ` the feeds Emulation examples ^^^^^^^^^^^^^^^^^^ -Configure ``conda-forge.yml`` to emulate ``linux-ppc64le``, but use native runners for ``linux-64`` +Configure ``conda-forge.yml`` to emulate ``linux-ppc64le``, but use native runners for ``linux-64`` and ``linux-aarch64``. This works because ``linux-ppc64le`` is not natively supported by Azure, so ``conda-smithy`` will add QEMU steps to emulate it. However, ``linux-64`` and ``linux-aarch64`` are natively supported by Azure and Travis CI, respectively, so no emulation is needed. @@ -568,6 +570,15 @@ natively supported by Azure and Travis CI, respectively, so no emulation is need linux_ppc64le: azure linux_64: azure +Use this variables in your builds scripts if needed: + +- ``CONDA_BUILD_CROSS_COMPILATION``: Set to ``1`` when build and host platforms differ. This can + mean your are cross-compiling or emulating. +- ``CROSSCOMPILING_EMULATOR``: Path to the ``qemu`` binary for the host platform. + +See also :ref:`cross_compilation_howto` for other variables you might find useful in your build +scripts. + Rust Nightly ------------ @@ -638,13 +649,13 @@ When should CDTs be used? 2. When a conda packaged library will not work properly. For example: a new ``glibc`` package means we would have to edit the elf interpreter of all the conda package binaries. - + What's are some good examples? ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1. The OpenCL loader (``ocl-icd`` together with ``ocl-icd-system``) provides an OpenCL loading library. The loader will look at OpenCL implementations given in - ``$CONDA_PREFIX/etc/OpenCL/vendors``. + ``$CONDA_PREFIX/etc/OpenCL/vendors``. For example: Pocl is a conda packaged implementation that runs OpenCL on the CPU. Vendor specific implementations like the NVIDIA OpenCL or ROCm OpenCL are not conda packaged, so we have to rely on the system. By installing ``ocl-icd-system`` we enable the loader to look at From 3c71428a738428cde0443937540f32d2daf3013e Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 25 Jul 2023 19:39:03 +0200 Subject: [PATCH 13/14] only on linux --- sphinx/src/maintainer/knowledge_base.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index ab32d8111c..d6cdb53d80 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -549,6 +549,8 @@ This key maps a ``build_platform`` to a ``provider`` that will be used to emulat requires emulation, and will adjust the appropriate CI steps to ensure that QEMU runs the process. Ensure changes are applied by :ref:`rerendering ` the feedstock. +Note that only Linux architectures are currently supported via emulation. + .. warning:: Emulated builds are very slow and incur an additional strain on conda-forge CI resources. From efbb19d665dedcbff46b0690d34401fb62cd912d Mon Sep 17 00:00:00 2001 From: jaimergp Date: Tue, 25 Jul 2023 20:06:15 +0200 Subject: [PATCH 14/14] clarify CROSSCOMPILING_EMULATOR --- sphinx/src/maintainer/knowledge_base.rst | 12 ++---------- 1 file changed, 2 insertions(+), 10 deletions(-) diff --git a/sphinx/src/maintainer/knowledge_base.rst b/sphinx/src/maintainer/knowledge_base.rst index d6cdb53d80..94d6fef2f7 100644 --- a/sphinx/src/maintainer/knowledge_base.rst +++ b/sphinx/src/maintainer/knowledge_base.rst @@ -371,8 +371,8 @@ can aid in cross-compilation setups: automatically created for you too. - ``CC_FOR_BUILD``: C compilers targeting the build platform. - ``CXX_FOR_BUILD``: C++ compilers targeting the build platform. -- ``CROSSCOMPILING_EMULATOR``: Path to the ``qemu`` binary for the host platform (see - :ref:`emulation`). +- ``CROSSCOMPILING_EMULATOR``: Path to the ``qemu`` binary for the host platform. Useful for running + tests when cross-compiling. This is all supported by two main conda-build features introduced in version 3: @@ -572,14 +572,6 @@ natively supported by Azure and Travis CI, respectively, so no emulation is need linux_ppc64le: azure linux_64: azure -Use this variables in your builds scripts if needed: - -- ``CONDA_BUILD_CROSS_COMPILATION``: Set to ``1`` when build and host platforms differ. This can - mean your are cross-compiling or emulating. -- ``CROSSCOMPILING_EMULATOR``: Path to the ``qemu`` binary for the host platform. - -See also :ref:`cross_compilation_howto` for other variables you might find useful in your build -scripts. Rust Nightly ------------