Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python bindings for index writer (draft) #8

Open
wants to merge 70 commits into
base: main
Choose a base branch
from
Open

Python bindings for index writer (draft) #8

wants to merge 70 commits into from

Conversation

atemerev
Copy link

@atemerev atemerev commented Sep 17, 2024 •
edited
Loading

Added Python interface to index_writer (with nanobind) to work with mpi4py, ported the index_writer test, added to CI

@atemerev
Add index_writer.py
30bd7fe
@atemerev
Add nanobind bindings for index writer C++ code and update CMake build
cc72148
@atemerev
Nanobind CMake fixes
4950834
@atemerev
feat: add sample code to index_writer.py
1fe3e1a
@atemerev
CMake/nanobind fix for PIC build error
b4754b5
@atemerev
Index writer python bindings as index_writer_py
9e9a1e1
@atemerev
Remove include directories redundancy
e088495
@atemerev
feat: Integrate neuron_parquet and index_writer projects into a singl…
e882b38 
…e CMake project
@atemerev
fix: Add HighFive library to CMakeLists.txt
b11054f
@atemerev
fix: Remove unnecessary installation of Python module
6ac9203
@atemerev
fix: Add HighFive include directories and link HighFive library to in…
15ddf73 
…dex_writer_py target
@atemerev
fix: Link MPI library to touch2parquet target
dd65b72
@atemerev
feat: Add MPI support to touch2parquet target
469a132
@atemerev
build: Update MPI linking for tests
dbfc2a1
@atemerev
feat: Add MPI setup and teardown to test_indexing.cpp
5ccaaae
@atemerev
test indexing revert
094fef0
@atemerev
feat: Add MPI initialization to index_writer.py
f231482
@atemerev
feat: Add MPI support to index bindings and writer
61e3cba
@atemerev
fix: Use correct method to extract MPI communicator from nanobind object
49e7edc
@atemerev
fix: Use MPI4Py to extract MPI_Comm from Python object
5f22072
@atemerev
fix: Remove direct inclusion of mpi4py header and use PyCapsule API t…
c225eca 
…o extract MPI_Comm
@atemerev
fix: Replace deprecated HighFive::MPIOFileDriver with HighFive::MPIOF…
2f0b112 
…ileAccess
@atemerev
fix: add MPI_INFO_NULL parameter to MPIOFileAccess constructor
a64b026
@atemerev
refactor: Create datasets without parallel access in index_writer.py
d9e24d0
@atemerev
refactor: Remove parallel I/O for file creation
430bb0a
@atemerev
fix: Remove rank participation message in file creation
de41a36
@atemerev
fix: Correct MPI communicator handling between Python and C++
acfc865
@atemerev
fix: Remove direct inclusion of mpi4py header and use PyCapsule API t…
db0a89c 
…o extract MPI communicator
@atemerev
fix: Correctly handle MPI communicator in C++ bindings
162e877
@atemerev
fix: Use correct MPI communicator in index_writer.py
bf27457
@atemerev
chore: Add tracing and logging to understand code behavior
e4dc308
@atemerev
fix: Ensure errors from C++ bindings are properly propagated and disp…
d377355 
…layed on the console
@atemerev
fix: Add verification step to ensure datasets are written correctly
b215579
@atemerev
fix: add GROUP constant definition to index_bindings.cpp
a4a5843
@atemerev
refactor: Improve error handling in index_bindings.cpp
311cca7
@atemerev
build: Add more logging and synchronization to diagnose MPI issue
986b98e
@atemerev
feat: Add more detailed logging to test_indexing and generate_data
40fbae5
@atemerev
fix: Add timeout mechanism and detailed logging to index_writer.py
329aa85
@atemerev
fix: Add timeout mechanism for index_writer_py.write
b35811f
@atemerev
fix: Remove threading and ensure write_index completion before verifi…
3306d95 
…cation
@atemerev
fix: Add file existence check and improve error handling in write_index
e414888
@atemerev
fix: resolve compilation errors in index_bindings.cpp
918536f
@atemerev
fix: Ensure file existence before calling index_write
f8a70c9
@atemerev
fix: Ensure file visibility on all MPI ranks before proceeding with test
132b3d1
@atemerev
fix: Ensure file visibility on all MPI ranks
268574c
@atemerev
fix: Use MPI-IO for file creation and verification in generate_data f…
1b9b5ab 
…unction
@atemerev
fix: Create test file in current directory
642cac1
@atemerev
fix: Update SEARCH/REPLACE block to match existing content in tests/t…
62ebace 
…est_indexing.py
@atemerev
fix: Use MPI-IO to open HDF5 file for parallel access
076f499
@atemerev
fix: enable parallel HDF5 support for HighFive
0e79024
@atemerev
fix: replace deprecated use_mpi_comm with `add(HighFive::MPIOFileAc…
9ce55eb 
…cess)` in index_bindings.cpp
@atemerev
fix: Remove MPIO driver from Python test file
bff14e8
@atemerev
chore: remove std::cout debug logging from index_bindings
bca9a43
@atemerev
chore: remove debug logging from python test_indexing
64a3f60
@atemerev
feat: run verification from all nodes
5ec7f92
@atemerev
build: Exclude MPI test from regular test run and add separate job to…
32c80a9 
… run it with mpirun
@atemerev
chore: install h5py and add build directory to PYTHONPATH in CI script
2af1d9c
@atemerev
build: Combine environment variable settings and mpirun command into …
4f40a3b 
…a single line
@atemerev
build: Update MPI run options to -n 2 --oversubscribe
71f11c5
@atemerev atemerev marked this pull request as ready for review October 10, 2024 13:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@atemerev @pramodk