Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix flaky test downstream_sv1::diff_management::test::test_diff_management #988

Open
jbesraa opened this issue Jun 20, 2024 · 6 comments · Fixed by #1074 or #1075
Open

Fix flaky test downstream_sv1::diff_management::test::test_diff_management #988

jbesraa opened this issue Jun 20, 2024 · 6 comments · Fixed by #1074 or #1075

Comments

@jbesraa
Copy link
Contributor

jbesraa commented Jun 20, 2024

Noisy test. Produces a lot of false alarms

https://github.com/stratum-mining/stratum/actions/runs/9601943467/job/26481865946?pr=968#step:7:422

 running 4 tests
test downstream_sv1::downstream::tests::gets_difficulty_from_target ... ok
test proxy::bridge::test::test_version_bits_insert ... ok
test downstream_sv1::diff_management::test::test_converge_to_spm_from_low ... ok
test downstream_sv1::diff_management::test::test_diff_management ... FAILED
failures:
---- downstream_sv1::diff_management::test::test_diff_management stdout ----
thread 'downstream_sv1::diff_management::test::test_diff_management' panicked at translator/src/lib/downstream_sv1/diff_management.rs:349:9:
Calculated shares per minute are outside the 99% confidence interval. Error: 4776.132, Error margin: 189.73665961010275, 5776.132
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
failures:
    downstream_sv1::diff_management::test::test_diff_management
test result: FAILED. 3 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 16.02s
error: test failed, to rerun pass `-p translator_sv2 --lib`
@jbesraa jbesraa changed the title Fix flaky test downstream_sv1::diff_management::test::test_diff_management' Fix flaky test `downstream_sv1::diff_management::test::test_diff_management' Jun 20, 2024
@jbesraa jbesraa changed the title Fix flaky test `downstream_sv1::diff_management::test::test_diff_management' Fix flaky test downstream_sv1::diff_management::test::test_diff_management Jun 20, 2024
@plebhash
Copy link
Collaborator

plebhash commented Jul 8, 2024

closed via #995

@plebhash plebhash closed this as completed Jul 8, 2024
@jbesraa
Copy link
Contributor Author

jbesraa commented Jul 25, 2024

@plebhash I think we should reopen this issue as the test is not fixed. I would suggest to #[ignore] the test for now.

@Fi3
Copy link
Collaborator

Fi3 commented Jul 25, 2024

about this test I use a z_score of 2.576 total runtime of 40 seconds and measure hash rate of 10 seconds.
With this values on my machine it pass almost all the time, I saw it fail just one time.
z_score I use for 99%, from what I saw more then 10 second do not improve the estimation, I always get the same error range also if I double or quadruple it on average. 40 seconds empirically are the shortest amount of time that almost never fail, ofc that on my machine dunno but maybe is relevant.

@pavlenex
Copy link
Collaborator

Do you guys want me to re-open this issue?

@plebhash
Copy link
Collaborator

#1074 is disabling the test as a temporary solution

we should eventually try to come up with a more sustainable fix to this problem

@GitGab19
Copy link
Collaborator

GitGab19 commented Sep 5, 2024

Reopening it, as agreed here

@GitGab19 GitGab19 reopened this Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
5 participants