Handling flaky tests

[glatterf42]

Closes #731 by marking tests as flaky. Through discussion with @khaeru, we agreed that this is the best option for now; should the tests make more issues in the future, we can still identify their root cause.

How to review

• Read the diff and note that the CI checks all pass.

PR checklist

• Continuous integration checks all ✅
• Adapt CI tests; coverage checks both ✅
• [ ] Add, expand, or update documentation. Nothing new added.
• [ ] Update release notes. Nothing public-facing added

[codecov]

Codecov Report

Merging #732 (d65e36b) into main (b5e93b7) will increase coverage by 0.0%.
The diff coverage is 100.0%.

@@          Coverage Diff          @@
##            main    #732   +/-   ##
=====================================
  Coverage   94.4%   94.5%           
=====================================
  Files         43      43           
  Lines       3434    3463   +29     
=====================================
+ Hits        3245    3273   +28     
- Misses       189     190    +1     

Files Changed	Coverage Δ
...age_ix/tests/test_feature_bound_activity_shares.py	100.0% <100.0%> (ø)
message_ix/tests/test_feature_price_emission.py	100.0% <100.0%> (ø)
message_ix/tests/test_integration.py	100.0% <100.0%> (ø)
message_ix/tests/test_legacy_version.py	100.0% <100.0%> (ø)
message_ix/tests/test_macro.py	100.0% <100.0%> (ø)
message_ix/tests/test_tutorials.py	97.5% <100.0%> (-2.5%)	⬇️

[khaeru]

Looks effective, thanks.

Per the changes to the files, I think it would be better to do something like:

# Define a mark as a module global
FLAKY = pytest.mark.flaky(
    reruns=5
    rerun_delay=2,
    condition="GITHUB_ACTIONS" in os.environ and sys.platform == "darwin",
    reason="Flaky; See iiasa/message_ix#731"
)

…

# Use the mark to decorate multiple tests
@FLAKY
def test_add_bound_activity_up_all_modes(message_test_mp):
    …

Two reasons:

• A general heuristic (not sure where or from which example I picked it up) for pytest usage is to organize tests (modules, files, classes, and functions) by functional meaning or in a way that mirrors the organization of the code being tested. For instance, we might have a class TestPlatform that collects tests of class Platform. This makes it easy to navigate.
  If these tests cease to be flaky (for instance, with a new ixmp4 backend) then the class grouping would no longer be meaningful and we'd have to move the tests out again.
• It would create minimally-invasive, small diffs.
• It would be clear when multiple tests are marked in the same meaning, without having to read an compare the function arguments.

[glatterf42]

Thank you for the comment, I agree that your suggestion is more elegant. I will re-work this PR and the other ones.

[khaeru]

Good to go, thanks for the thorough and clean work.