-
Notifications
You must be signed in to change notification settings - Fork 25
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ci: add codspeed to benchmark ANTA #826
Conversation
This pull request has conflicts, please resolve those before we can evaluate the pull request. |
Conflicts have been resolved. A maintainer will review the pull request shortly. |
Quality Gate passedIssues Measures |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Description
Use Codspeed to benchmark ANTA.
Use RESPX to mock eAPI responses.
The code generates an ANTA catalog from the unit test data at tests/units/anta_tests.
We should consider refactoring these data to use a dictionary with a tuple (Type[AntaTest], [unit test name]: str) as key to avoid duplicates.
It relies on a specific eAPI request ID format:
ANTA-{test name}:{unit test name}:{command index}-{command UUID}
. We currently patch ANTA commands to implement this, but we could consider implementing this directly in ANTA (the eAPI request ID is used for debugging purpose and its format has been arbitrary set toANTA-{test name}-{command UUID}
.Benchmark performances
As of this writing, there are 556 unit tests for ANTA tests. The code will run 2 benchmarks with 1 device on which those 556 tests are run and another one with 2 devices for a total of 1112 tests run (556 on each device).
This takes 4 minutes to run on the current GitHub Actions runners,
pytest-xdist
(see https://docs.codspeed.io/benchmarks/python#running-benchmarks-in-parallel) has not improved the performances. This could be considered on a bigger runner, but we will loose stdout.Requires
Checklist:
pre-commit run
)tox -e testenv
)