MG CI is too slow

[GitGab19]

As we discussed during last dev call, the current MG CI execution is too slow. We need to improve it, by using caches, multi-threading, or docker containers to run things in parallel.
I just found an interesting article regarding this stuff: https://www.uffizzi.com/blog/optimizing-rust-builds-for-faster-github-actions-pipelines

[pavlenex]

Perhaps slightly related to #928 but maybe not, certainly connected so cross-linking them to help us track similar isuses.

[plebhash]

one potential mitigation strategy that @Fi3 pointed out during the call is to restructure coverage.yaml

currently, all MG tests are lumped together into the same job: message-generator-test

as we get false alarms, our only recourse is to click this button on the PR Actions UI while hoping the next run won't have any false alarms and we can finally merge the PR:

this strategy forces all MG tests to re-run (because they're all lumped together into the same job), as opposed to only the false alarms... this is suboptimal for multiple reasons:

• it gives room for new false alarms on tests that were already successfully executed in the previous run
• it makes the process much slower, because we need to wait for all MG tests again, as opposed to only the ones that had a false alarm

if we could break the different MG tests into separate jobs under coverage.yaml, this process would be a bit better

of course, this is not the ultimate solution for the issue here, it is only a mitigation strategy while we don't find a way to bring deterministic execution to the MG tests under Github CI

[plebhash]

to expand a bit on my previous comment, I see two potential courses of action for this issue:

• a paliative solution, which makes MG CI slightly less worse by breaking tests into multiple jobs and allowing us to re-run false alarms in a more agile way
• a definitive solution, which makes MG CI execution deterministic

[johnnyasantoss]

I want to work on this

[johnnyasantoss]

Related discussion: #944

[jbesraa]

just a note: i think it would be better to speedup the GH action check initially if its a quick win before working on improving the local setup

[Fi3]

just a note: i think it would be better to speedup the GH action check initially if its a quick win before working on improving the local setup

100% agree

[johnnyasantoss]

I'm splitting the work in multiple steps. The first part is to add cache to the build and avoid recompiling unchanged deps and running tests in parallel. The other steps we will still need to discuss it later on.

[GitGab19]

Partially solved by #1056

[plebhash]

Partially solved by #1056

actually this ended up being #1056 + #1060 + #1064

now we are able to re-run MG CI jobs independently, which helps bypass false alarms

however, optimizations on MG CI are still desirable, so @johnnyasantoss feel free to continue working on this if you want

[pavlenex]

@GitGab19 @plebhash Is there anything concrete that @johnnyasantoss should work on? Since we had #1056 #1060 #1064 my question is, is there enough context on what needs to be done, or should we specify that for @johnnyasantoss? I guess a question for @johnnyasantoss as well 😄

[GitGab19]

I think that we could just try to optimize what done by @plebhash. For example adding cache where possible, since it's also a thing that @johnnyasantoss mentioned here: #1028 (comment)

[johnnyasantoss]

@pavlenex I have enough context on this and can get this done. Just need to figure out a better time organization.

@GitGab19 Yep. I have a local branch with optimizations on the CI side. I'll push the changes soon