Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Menv query radius and memory usage #152

Open
lgarrison opened this issue Sep 21, 2024 · 1 comment
Open

Menv query radius and memory usage #152

lgarrison opened this issue Sep 21, 2024 · 1 comment
Assignees
Labels
HOD for abacusnbody.hod

Comments

@lgarrison
Copy link
Member

lgarrison commented Sep 21, 2024

prepare_sim is using a lot of memory right now. In particular, do_Menv_from_tree() uses 100+ GB per process with default settings on an AbacusSummit sim. I think the reason is the r_outer tree query:

allinds_outer = querypos_tree.query_ball_point(pos_cut, r=r_outer, workers=nthread)

I think the problem started in #78, when the default r_outer when from 5 to 10. But I can't tell why that value was changed from the PR.

@SandyYuan, @boryanah: do you know if we need 10 Mpc/h as the outer radius?

For context, we're trying to compute the sum of masses of neighbor halos in two radius apertures and then taking the difference. The inner aperture seems okay, just the outer is using a ton of memory to save all the indices. It finds over 2 billion matches (which should only be ~16 GB, but maybe cause these are Python lists, we're hitting over 100 GB. Still not 100% clear on why...).

Really, the ideal algorithm wouldn't even store the indices. We would just add up the mass on the fly as we encounter a valid pair in the tree query. But scipy.KDTree doesn't seem to support that. Maybe another library does? Or maybe we could use a different Menv metric?

Probably related to #143.

@lgarrison lgarrison added the HOD for abacusnbody.hod label Sep 21, 2024
@lgarrison
Copy link
Member Author

@epaillas, feel free to jump in here too, since this is probably related to #143 (although it isn't explicitly related to whether one is using halo light cones).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
HOD for abacusnbody.hod
Projects
None yet
Development

No branches or pull requests

3 participants