Replies: 2 comments 9 replies
-
That's a good catch! I think we already use a different depot for each build, but this might be needed too... I think we should try it. Deleting stuff after the build sometimes fails (so we have to go in and manually delete stuff from time to time) but that's ok, we already have to do that anyways. I think it will actually be easier to manage all the files if we create a directory for every buildkite build that contains both the julia installation and the depot, and then delete that directory when the build is done. Let me know if that makes sense... |
Beta Was this translation helpful? Give feedback.
-
Update to this, I tried using the buildkite plugin for OceanBioME and it seems like it is more reliable, but it takes >5 times longer to run the test suite. I have these two PRs OceanBioME/OceanBioME.jl#190 and testing it with OceanBioME/OceanBioME.jl#196 Another thought I had was that it might be over all more efficient to prevent any test concurrency which I think you could do by making each machine only have one buildkite agent. Given that one when there are concurrent jobs at least one if not both normally fail this would probably be a good intermediate step until we can work out why the plugin is so much slower? |
Beta Was this translation helpful? Give feedback.
-
I gather that the CI failures that sometimes happen because all builds put julia in the same place:
Oceananigans.jl/.buildkite/pipeline.yml
Lines 16 to 17 in eb7c178
So if there are multiple PRs happening at the same time they try and install it in the same place, so one build might try and run julia as another is unzipping and then its missing.
Since it is downloaded everytime anyway couldn't each build have its own version e.g.
...
- "$SVERDRUP_HOME/julia-$BUILDKITE_BUILD_NUMBER/julia-$JULIA_VERSION/bin/julia -O0 --color=yes --project -e 'using Pkg; Pkg.instantiate(; verbose=true)'"
and then delete them after?
Obviously its more overhead but it might stop the CI issues?
Beta Was this translation helpful? Give feedback.
All reactions