Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft: Add initial build wheels CI/CD pipeline #763

Open
wants to merge 43 commits into
base: main
Choose a base branch
from

Conversation

dudoslav
Copy link

@dudoslav dudoslav commented Aug 28, 2024

This PR aims to modify the build system for the VCF project to comply with our packaging and distribution guidelines:
https://www.notion.so/Packaging-and-Distribution-96af10190b4f455f9a31180899f47a83?pvs=4

Specifically we want to enable these scenarios:

  1. Simply just build locally without any package. With CMake options you can specify if TileDB should be local or external as well as if Python bindings should be compiled as well.
  2. Correct PyPI wheel, meaning that all shared libraries MUST be packaged inside of the wheel (tiledb, tiledbvcf)
  3. Correct Conda package, meaning that no shared libraries that are provided by Conda should be packaged (included in the install group)

How to achieve this:

  1. Change the CMake logic to have two callable CMakeLists.txt (./libtiledbvcf and ./apis/python).
  2. When calling ./libtiledbvcf users can specify if they wish to also build python bindings by using the -DTILEDBVCF_ENABLE_PYTHON option. When this is set libtiledbvcf AND bindings are build otherwise just libtiledbvcf is build. This is used for PyPI package (python bindings enabled) and for libtiledbvcf conda package (python bindings disabled)
  3. When calling ./apis/python only python bindings are build. This requires pre-installed libtiledbvcf and libtiledb. This scenario is useful for packaging conda tiledbvcf-py bindings.
  4. In order to make sdist work we need to move pyproject.toml out of apis/python to top level of the project.

NOTE: This PR is being tested in my personal fork.

@dudoslav dudoslav self-assigned this Aug 28, 2024
@jdblischak
Copy link
Collaborator

Correct Conda package, meaning that no shared libraries that are provided by Conda should be packaged (included in the install group)

When calling ./apis/python only python bindings are build. This requires pre-installed libtiledbvcf and libtiledb. This scenario is useful for packaging conda tiledbvcf-py bindings.

Once this is implemented, this will fix Issue #717

@dudoslav
Copy link
Author

Right now I was able to produce wheels in this workflow:
https://github.com/dudoslav/TileDB-VCF/actions/runs/10827370316/job/30040372000

Next steps:

  1. Clean up changes
  2. Make sure that feedstocks are not negatively affected (a new PR needs to be created)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants