Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download artifacts in consistent way #525

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

Koubek
Copy link
Contributor

@Koubek Koubek commented Jun 20, 2023

ARTIFACT CONSISTENCY PROBLEM:

We have realized that containers using shared folders to store artifacts were suffering from time to time from inconsistently downloaded artifacts.

When running on orchestrator, but basically whenever downloading artifacts from containers (and not from BcContainerHelper from the host machine) and using a shared folder, it's not possible to use mutext (which is the difference between in-container download vs. download on the host using BCCH) and guarantee only one process accessing the shared resources.

The provided functionality uses file lock that controls access and guarantees only one process can access and other containers will be waiting until done (with success or not).

@Koubek
Copy link
Contributor Author

Koubek commented Jun 20, 2023

I have created this as a draft because I think I will wait several next days before confirming everything works as expected.

@Koubek
Copy link
Contributor Author

Koubek commented Oct 16, 2023

We have tested it during the last months and it works great. We haven't been experiencing any inconsistency since putting this into our container-based environments.

I will switch from draft to the regular PR. You can decide if it's worth to implement into master or not.

@Koubek Koubek marked this pull request as ready for review October 16, 2023 09:44
@Koubek Koubek requested a review from a team as a code owner October 16, 2023 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant