Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improves runtime of apply_hdrfix #146

Merged
merged 2 commits into from
Sep 9, 2024
Merged

improves runtime of apply_hdrfix #146

merged 2 commits into from
Sep 9, 2024

Conversation

havok2063
Copy link
Collaborator

This PR closes #123. It improves the runtime of the apply_hdrfix function. Adds some missing tests as well.

I profiled the extract_metadata function at Utah using MJD 60255, which has 1105 raw frame to process. Without the apply_hdrfix, extract_metadata takes 217 seconds to run, with the bulk of the time spent in fits.getheader (215 seconds). With the original method, it took ~31000 seconds to run, with the bulk of the time spent in the pathlib.Path.rglob of apply_hdrfix. This PR removes the pathlib rglob search and replaces it with an fnmatch check. This reduces the time significantly. With the new apply_hdrfix, extract_method took 229 seconds to run, with the bulk time still spent in fits.getheader, 214 seconds with 13 seconds spent in apply_hdrfix.

@havok2063 havok2063 added bug Something isn't working enhancement New feature or request labels Sep 8, 2024
@havok2063 havok2063 self-assigned this Sep 8, 2024
@ajmejia
Copy link
Contributor

ajmejia commented Sep 9, 2024

Brian, this all looks great to me! I test it at Utah the extraction and is better now. I'll probably look into using faster methods to get the header from the raw data. We can merge now.

@ajmejia ajmejia closed this Sep 9, 2024
@ajmejia ajmejia reopened this Sep 9, 2024
@ajmejia ajmejia merged commit 471de55 into master Sep 9, 2024
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Metadata caching runs slow (specially at Utah)
2 participants