Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GH-43519: [Python][CI][Packaging] Use released versions to build and test wheels on Python 3.13 #44193

Merged
merged 8 commits into from
Oct 10, 2024

Conversation

jorisvandenbossche
Copy link
Member

@jorisvandenbossche jorisvandenbossche commented Sep 23, 2024

Rationale for this change

Now that numpy, cython and pandas all have releases that support Python 3.13, we can remove the extra index and ensure that we are building our wheels with the released version instead of nightly dev version.

@jorisvandenbossche
Copy link
Member Author

@github-actions crossbow submit -g wheel

Copy link

Revision: 7d19a5b

Submitted crossbow builds: ursacomputing/crossbow @ actions-91d56dc958

Task Status
python-sdist GitHub Actions
wheel-macos-monterey-cp310-cp310-amd64 GitHub Actions
wheel-macos-monterey-cp310-cp310-arm64 GitHub Actions
wheel-macos-monterey-cp311-cp311-amd64 GitHub Actions
wheel-macos-monterey-cp311-cp311-arm64 GitHub Actions
wheel-macos-monterey-cp312-cp312-amd64 GitHub Actions
wheel-macos-monterey-cp312-cp312-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-arm64 GitHub Actions
wheel-macos-monterey-cp39-cp39-amd64 GitHub Actions
wheel-macos-monterey-cp39-cp39-arm64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-arm64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-arm64 GitHub Actions
wheel-windows-cp310-amd64 GitHub Actions
wheel-windows-cp311-amd64 GitHub Actions
wheel-windows-cp312-amd64 GitHub Actions
wheel-windows-cp313-amd64 GitHub Actions
wheel-windows-cp39-amd64 GitHub Actions

@jorisvandenbossche
Copy link
Member Author

I forgot that the free-threading build is of course using the same requirements files, and for those builds we still need cython main (there is not yet a cython release with freethreaded support/wheels).

I do think it is important to build are main released wheels with a released version of numpy/cython, so we will have to add the pip flags depending on using a free-threaded python build or not.

@jorisvandenbossche
Copy link
Member Author

@github-actions crossbow submit wheelcp313t

Copy link

Revision: bbe9812

Submitted crossbow builds: ursacomputing/crossbow @ actions-31ac4e075e

Task Status
wheel-macos-monterey-cp313-cp313t-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-arm64 GitHub Actions

@jorisvandenbossche
Copy link
Member Author

@github-actions crossbow submit wheel-macoscp313t

Copy link

Revision: 7a5f484

Submitted crossbow builds: ursacomputing/crossbow @ actions-7e55a83117

Task Status
wheel-macos-monterey-cp313-cp313t-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-arm64 GitHub Actions

@jorisvandenbossche
Copy link
Member Author

(the macos arm64 is passing now after restarting it, but so the first time it was failing with a segfault in /test_flight_async.py, that might be something to later stress test more under the free-threaded build)

@pitrou
Copy link
Member

pitrou commented Sep 24, 2024

(the macos arm64 is passing now after restarting it, but so the first time it was failing with a segfault in /test_flight_async.py, that might be something to later stress test more under the free-threaded build)

The C++ Async Flight tests also see intermittent crashes on some configurations, I wouldn't necessarily fault PyArrow here. Perhaps the free-threaded build just helps stress threading more in the the Flight tests.

@jorisvandenbossche
Copy link
Member Author

@github-actions crossbow submit -g wheel

Copy link

Revision: ae5fc8f

Submitted crossbow builds: ursacomputing/crossbow @ actions-8d08b45b5c

Task Status
python-sdist GitHub Actions
wheel-macos-monterey-cp310-cp310-amd64 GitHub Actions
wheel-macos-monterey-cp310-cp310-arm64 GitHub Actions
wheel-macos-monterey-cp311-cp311-amd64 GitHub Actions
wheel-macos-monterey-cp311-cp311-arm64 GitHub Actions
wheel-macos-monterey-cp312-cp312-amd64 GitHub Actions
wheel-macos-monterey-cp312-cp312-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-arm64 GitHub Actions
wheel-macos-monterey-cp39-cp39-amd64 GitHub Actions
wheel-macos-monterey-cp39-cp39-arm64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-arm64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-arm64 GitHub Actions
wheel-windows-cp310-amd64 GitHub Actions
wheel-windows-cp311-amd64 GitHub Actions
wheel-windows-cp312-amd64 GitHub Actions
wheel-windows-cp313-amd64 GitHub Actions
wheel-windows-cp39-amd64 GitHub Actions

@jorisvandenbossche
Copy link
Member Author

jorisvandenbossche commented Sep 24, 2024

The mac Python 3.9 build shows those warnings, which might explain the failures (ModuleNotFoundError: No module named 'setuptools.command.build'):

WARNING: Target directory /Users/runner/work/crossbow/crossbow/build-env/lib/python3.9/site-packages/wheel-0.44.0.dist-info already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/runner/work/crossbow/crossbow/build-env/lib/python3.9/site-packages/wheel already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/runner/work/crossbow/crossbow/build-env/lib/python3.9/site-packages/distutils-precedence.pth already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/runner/work/crossbow/crossbow/build-env/lib/python3.9/site-packages/setuptools already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/runner/work/crossbow/crossbow/build-env/lib/python3.9/site-packages/pkg_resources already exists. Specify --upgrade to force replacement.
WARNING: Target directory /Users/runner/work/crossbow/crossbow/build-env/lib/python3.9/site-packages/_distutils_hack already exists. Specify --upgrade to force replacement.

Now, those warnings also appear in the builds for python >= 3.10, so not entirely sure how to explain the 3.9 failure

@jorisvandenbossche
Copy link
Member Author

@github-actions crossbow submit wheel-macos-*

Copy link

Revision: fa65a63

Submitted crossbow builds: ursacomputing/crossbow @ actions-b3d3ae8e9c

Task Status
wheel-macos-monterey-cp310-cp310-amd64 GitHub Actions
wheel-macos-monterey-cp310-cp310-arm64 GitHub Actions
wheel-macos-monterey-cp311-cp311-amd64 GitHub Actions
wheel-macos-monterey-cp311-cp311-arm64 GitHub Actions
wheel-macos-monterey-cp312-cp312-amd64 GitHub Actions
wheel-macos-monterey-cp312-cp312-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-amd64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-arm64 GitHub Actions
wheel-macos-monterey-cp39-cp39-amd64 GitHub Actions
wheel-macos-monterey-cp39-cp39-arm64 GitHub Actions

@jorisvandenbossche
Copy link
Member Author

@github-actions crossbow submit -g wheel

Copy link

github-actions bot commented Oct 2, 2024

Revision: 6eea81c

Submitted crossbow builds: ursacomputing/crossbow @ actions-35278aadcf

Task Status
python-sdist GitHub Actions
wheel-macos-monterey-cp310-cp310-arm64 GitHub Actions
wheel-macos-monterey-cp311-cp311-arm64 GitHub Actions
wheel-macos-monterey-cp312-cp312-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-arm64 GitHub Actions
wheel-macos-monterey-cp39-cp39-arm64 GitHub Actions
wheel-macos-ventura-cp310-cp310-amd64 GitHub Actions
wheel-macos-ventura-cp311-cp311-amd64 GitHub Actions
wheel-macos-ventura-cp312-cp312-amd64 GitHub Actions
wheel-macos-ventura-cp313-cp313-amd64 GitHub Actions
wheel-macos-ventura-cp313-cp313t-amd64 GitHub Actions
wheel-macos-ventura-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-arm64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-arm64 GitHub Actions
wheel-windows-cp310-amd64 GitHub Actions
wheel-windows-cp311-amd64 GitHub Actions
wheel-windows-cp312-amd64 GitHub Actions
wheel-windows-cp313-amd64 GitHub Actions
wheel-windows-cp39-amd64 GitHub Actions

@jorisvandenbossche
Copy link
Member Author

cc @raulcd

@jorisvandenbossche jorisvandenbossche added this to the 18.0.0 milestone Oct 9, 2024
@jorisvandenbossche jorisvandenbossche added the Priority: Blocker Marks a blocker for the release label Oct 9, 2024
Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me. Could you rebase to fix the CI failures?

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Oct 10, 2024
@jorisvandenbossche
Copy link
Member Author

@github-actions crossbow submit -g wheel

@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 10, 2024
Copy link

Revision: 63963fe

Submitted crossbow builds: ursacomputing/crossbow @ actions-98e88ff273

Task Status
python-sdist GitHub Actions
wheel-macos-monterey-cp310-cp310-arm64 GitHub Actions
wheel-macos-monterey-cp311-cp311-arm64 GitHub Actions
wheel-macos-monterey-cp312-cp312-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313-arm64 GitHub Actions
wheel-macos-monterey-cp313-cp313t-arm64 GitHub Actions
wheel-macos-monterey-cp39-cp39-arm64 GitHub Actions
wheel-macos-ventura-cp310-cp310-amd64 GitHub Actions
wheel-macos-ventura-cp311-cp311-amd64 GitHub Actions
wheel-macos-ventura-cp312-cp312-amd64 GitHub Actions
wheel-macos-ventura-cp313-cp313-amd64 GitHub Actions
wheel-macos-ventura-cp313-cp313t-amd64 GitHub Actions
wheel-macos-ventura-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2-28-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2-28-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2-28-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2-28-cp39-cp39-arm64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-amd64 GitHub Actions
wheel-manylinux-2014-cp310-cp310-arm64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-amd64 GitHub Actions
wheel-manylinux-2014-cp311-cp311-arm64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313-arm64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-amd64 GitHub Actions
wheel-manylinux-2014-cp313-cp313t-arm64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-amd64 GitHub Actions
wheel-manylinux-2014-cp39-cp39-arm64 GitHub Actions
wheel-windows-cp310-amd64 GitHub Actions
wheel-windows-cp311-amd64 GitHub Actions
wheel-windows-cp312-amd64 GitHub Actions
wheel-windows-cp313-amd64 GitHub Actions
wheel-windows-cp39-amd64 GitHub Actions

Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@github-actions github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Oct 10, 2024
@raulcd raulcd merged commit 6fe0ff0 into apache:main Oct 10, 2024
36 checks passed
@raulcd raulcd removed the awaiting merge Awaiting merge label Oct 10, 2024
raulcd pushed a commit that referenced this pull request Oct 10, 2024
…test wheels on Python 3.13 (#44193)

### Rationale for this change

Now that numpy, cython and pandas all have releases that support Python 3.13, we can remove the extra index and ensure that we are building our wheels with the released version instead of nightly dev version.

* GitHub Issue: #43519

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Raúl Cumplido <[email protected]>
@jorisvandenbossche jorisvandenbossche deleted the gh-43519-cleanup-requirements branch October 10, 2024 18:21
Copy link

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6fe0ff0.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 7 possible false positives for unstable benchmarks that are known to sometimes produce them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Python Priority: Blocker Marks a blocker for the release
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants