Find optimal number of runs for fuzz/invariant tests

[zugdev]

Resolves #943

[ubiquity-os-deployer]

53f4e45

11a9b78

53f4e45

ef11463

333896d

e233d0d

9cf5eb1

[zugdev]

Deep Fuzz

Deep Fuzz job most recent configuration with 10^5 runs takes on average (193+194+194+194+188) / 5 seconds to complete. You can check these times here. We should consider a slightly worse scenario, to add a little breathing room. 250 seconds is arbitrarily good.

That's with 10^5 runs, if we divide 10^5 by 250 we get, rounded down, 400 runs per second.

In 6 hours we have 21600 seconds, multiplying by 400 runs per second we get 8640000 runs. We should round this down, 8 x 10^6 are enough runs, as we should guarantee that the workflow doesn't go over the 6 hours limit.

8000000 runs / 400 runs per second = 20000 seconds
20000 seconds / 3600 seconds per hour = 5.555555556 hours

That's 5 hours 33 minutes and 20 seconds. Considering the other steps in the job take an average of 30 seconds combined, that's a very good number of runs.

EXPECTED Optimal Fuzz Runs = 8000000 (8 x 10^6) runs

Invariant Tests

Invariant Tests job most recent configuration with 2.5 x 10^3 runs takes on average (79+79+79+79+81) / 5 seconds to complete. You can check these times here We should consider a slightly worse scenario, to add a little breathing room. 125 seconds is arbitrarily good.

That's with 2.5 x 10^3 runs, if we divide that by 125 seconds we get, rounded down, 20 runs per second.

In 6 hours we have 21600 seconds, multiplying by 20 runs per second we get 432000 runs. We should round this down, 4 x 10^5 are enough runs, as we should guarantee that the workflow doesn't go over the 6 hours limit.

400000 runs / 20 runs per second = 20000 seconds
20000 seconds / 3600 = 5,555555556 hours

That's 5 hours 33 minutes and 20 seconds. Considering the other steps in the job take an average of 30 seconds combined, that's a very good number of runs.

EXPECTED Optimal Invariant Runs = 400000 (4 x 10^5) runs

[zugdev]

This is only my thought process, I will be closing this since the proper PR with no unnecessary commits is available here.