Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cache: better locking #503

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Conversation

matthiasdiener
Copy link
Contributor

@matthiasdiener matthiasdiener commented Jul 8, 2021

  • Use an actual file lock instead of file presence
  • Do not wait for 1 second in case of lock conflict (reduced to 0.05s)
  • actual locking code is from https://github.com/benediktschmitt/py-filelock (public domain code); we can also add the package as a dependency or just extract the relevant code if you prefer.

Background: this code is part of the reason for the CI slowdowns in mirgecom (especially with profiling).

- Use an actual file lock instead of file presence
- Do not wait for 1 second in case of lock conflict
Copy link
Owner

@inducer inducer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I think it's probably better to add this as a dependency instead. (https://github.com/conda-forge/filelock-feedstock also already exists.)

Out of curiosity: What prompted this change? Were their any specific failure modes?

@matthiasdiener
Copy link
Contributor Author

matthiasdiener commented Jul 13, 2021

Thanks! I think it's probably better to add this as a dependency instead. (https://github.com/conda-forge/filelock-feedstock also already exists.)

The main reason I didn't use the package is that https://github.com/benediktschmitt/py-filelock/blob/master/filelock.py logs a lot of debug messages.

Out of curiosity: What prompted this change? Were their any specific failure modes?

illinois-ceesd/mirgecom#418 fixed the immediate issue we had seen in mirgecom, so this PR isn't a priority at the moment. However, I still think a 1 sec timeout is a bit much, maybe we should reduce it to 0.05s (like in the filelock package)?

@inducer
Copy link
Owner

inducer commented Jul 14, 2021

I'm open to reducing the timeout.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants