Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimise hdf5 #36

Open
wants to merge 58 commits into
base: main
Choose a base branch
from
Open

Optimise hdf5 #36

wants to merge 58 commits into from

Conversation

hllelli2
Copy link
Collaborator

@hllelli2 hllelli2 commented Oct 1, 2024

  • MapDataset, ArrayDataset and MapDataloader to handle inputs of .mrc or .map data.
  • Transformations and Augmentations build upon CCPEM_Utils.
  • Hdf5 intermediate data store
  • Added accompanying tests
  • Added basic pip install

@hllelli2
Updates to gitignore
c64fbff
@hllelli2
basic setup config for pip install
2f4d58f
@hllelli2
Added MapDataset and MapDataLoader
6afd99e
@hllelli2
Added custom augments and transfroms as well as tiling
ef73dd2
@hllelli2
Added tests for new code
2ab4260
@hllelli2
feat: Add HDF5DataStore class for handling hdf5 files
8b9d06d
@hllelli2
Add pytest configuration file and conftest.py for test fixtures
d8729a1
@hllelli2
Added a dataset config class to have all the defaults in one place
1a6ed24
@hllelli2
Create a class for lazy-loading a hdf5 file
c8486b8
@hllelli2
Moved multi-processing to utils, Added an array dataset which can loa…
3779ee4 
…d arrays into the hdf5 without a file.
@hllelli2
Moved Multi-processing code here, code here to duplciate arrays from …
b9a91cb 
…the dataloader, augment them and saves them to array. For a specific usecase in ml-toolkit
@hllelli2
Changed augment to take arrays instead of map-objects to make it comp…
ab2ce2f 
…atible with ArrayDataset
@hllelli2
Removed mapObject transform base
d6eda3a
@hllelli2
chore: Refactor transforms module and update transform classes
a86c910
@hllelli2
first attempt at reducing if label/weight... is not none repeated code
41d40b3
@hllelli2
chore: Add test fixtures and update conftest.py for test setup
6f73aae
@hllelli2
set_gpu is not in the current ccpem-utils so check added JIC
07e2318
@hllelli2
chore: Refactor HDF5DataStore class and add support for temporary dir…
a6cc9b1 
…ectory storage
@hllelli2
Removed "_" id logic in MapDataLoader, might reimplement later but no…
05f792b 
…t sure if needed for multi-class porblems
@hllelli2
Refactor process_datasets function and remove unnecessary code
5a770f7
@hllelli2
Refactor transforms module and update transform classes
0a99fd7
@hllelli2
re-added __del__ method to HDF5Store
8772fa6
@hllelli2
Refactor MapDataset and ArrayDataset classes to handle weight tensors
93c8424
@hllelli2
Refactor
9128e9a
@hllelli2
Refactor DecomposeToSlices, MapObjectMaskCrop, and MapObjectPadding c…
b2a49c1 
…lasses
@hllelli2
Refactor test_map_io.py to handle weight tensors and update dataset l…
ce47030 
…engths
@hllelli2
Refactor HDF5DataStore class to handle weight tensors and implement c…
5d34da0 
…aching mechanism
@hllelli2
pre-commit fixes
d049499
@hllelli2
Refactor MapDataLoader to add caching mechanism for HDF5DataStore
1e65dd8
@hllelli2
Refactor HDF5DataStore class to handle weight tensors and update cach…
f31870b 
…ing mechanism
@hllelli2 hllelli2 requested a review from aj26git October 1, 2024 13:13
@hllelli2 hllelli2 self-assigned this Oct 1, 2024
aj26git
aj26git reviewed Oct 2, 2024
View reviewed changes
src/caked/Transforms/augments.py
:return: (np.ndarray) rotated volume or (np.ndarray, int, int) rotated volume and rotation parameters
"""

def __init__(self, random_seed: int = 42):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need the random seed argument here. Does seem to be used?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh its not implemented yet!

aj26git
aj26git reviewed Oct 3, 2024
View reviewed changes
tests/test_map_io.py Outdated Show resolved Hide resolved
aj26git
aj26git reviewed Oct 3, 2024
View reviewed changes
tests/conftest.py Outdated Show resolved Hide resolved
@aj26git
Copy link
Collaborator

aj26git commented Oct 3, 2024

@hllelli2 : Overall it looks good. I have added a few comments/suggestions. It is good to go once these are addressed.

@hllelli2
Add iterator method to HDF5DataStore and simplify key checks in tests
0696806
@hllelli2
Refactor test fixtures to use default scope and improve readability
7fbefdf
@hllelli2
Refactor: Rename tiles to slice_indicies and update related logic
36a5883
@hllelli2
Refactor: Simplify iterator method in HDF5DataStore
b24d299
@hllelli2
Refactor: update default voxel parameter type and improve kwargs hand…
7d12ece 
…ling
@hllelli2
Merge branch 'OptimiseHdf5' of https://github.com/alan-turing-institu…
f4555ae 
…te/CAKED into OptimiseHdf5
@hllelli2
Updated MapDataLoader to accepted the Mapdataset Kwargs in the load f…
2f32536 
…unction. This allows any keywords to be applied to all MapDatasets.
@hllelli2
Fix: Change default value of 'vox' to a float and update related logi…
1844ec7 
…c in MapObjectVoxelNormalisation
@hllelli2
Merge branch 'OptimiseHdf5' of https://github.com/alan-turing-institu…
96772ae 
…te/caked into OptimiseHdf5
@hllelli2
Cleanup: remove debug print statements from MapDataset class
b8d9c65
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aj26git aj26git aj26git left review comments

Assignees

@hllelli2 hllelli2

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@hllelli2 @aj26git