-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
multiprocessing consistency across cooltools #431
Comments
Can we make this into a decorator to ensure the argument is consistent across all tools and stays like that in the future? |
note 10/23/2023: some of the cootlools functions using see @nvictus comment #464 (comment) |
in the process of addressing this issue, @Yaoyx ran into a design choice: any recommendations @golobor @nvictus @agalitsyna @sergpolly ?
cooltools/cooltools/api/dotfinder.py Line 1307 in 81b9045
|
imho multiprocess/dill is much more preferable, because multiprocessing/pickle prevents one more thing: open2c/open2c_examples#29 also I've been trying to hint at that here #477 |
Multiprocessing will eventually let you register custom serializers, but until that happens I would stick with multiprocess+dill or another 3rd party pool manager like loky or mpire. Also relevant: python/cpython#89467 |
We might also want to explore changing the default context (which is from multiprocess import get_context
with get_context("spawn").Pool() as pool:
.... See this for details. |
closed in #489 |
many user-facing functions take nproc as an argument.
e.g.
cooltools/cooltools/api/snipping.py
Line 824 in c85d647
cooltools.sample()
does notcooltools/cooltools/api/sample.py
Line 54 in c85d647
(however, multiprocessing is offered as an nproc argument in=cooltools.cli.random-sample does)
Any suggestion on preferred behavior?
The text was updated successfully, but these errors were encountered: