Publish and install build dependencies to/from alternative package index

[ryanking13]

Proposal

I propose hosting multiple versions of packages used as a build dependency (== numpy) in an alternative package index, and let pyodide-build use it when installing build dependencies.

Why?

When building packages out-of-tree, each package may require a different version of Numpy. For now, if a package requires numpy as a build dependency, we just replace it with the numpy that is included in the Pyodide distribution, and hope it work.

For example, after we update the Numpy version in Pyodide distribution >= 2.0, some packages that use Numpy < 2.0 in the build step will fail to build because of the version mismatch.

How?

I've been publishing packages built in pyodide/pyodide-recipes to Anaconda package registry: Anaconda Pyodide org. For now, these packages cannot be installed in the Pyodide runtime (CORS issue), but we can still use it during build time.

pyodide-build will point to this package index when installing build dependencies.

Issues / Concerns

• How should we handle the in-tree cases?
• After unvendoring recipes, what should change?

Overview

As-is:

sequenceDiagram
    Note over pyodide-build: Start building a package
    PyPI->>pyodide-build: Install native build dependency
    xbuildenv->>pyodide-build: Replace cross-build files
    Note over pyodide-build: Continue building package

Loading

To-be:

sequenceDiagram
    Note over pyodide-build: Start building a package
    Pyodide Package Index->>pyodide-build: Install cross-compiled build dependency
    Note over pyodide-build: Continue building package

Loading

[hoodmane]

Nice diagrams. Sounds like a reasonable idea.