Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ci: add codspeed to benchmark ANTA #826

Merged
merged 6 commits into from
Sep 26, 2024
Merged

Conversation

mtache
Copy link
Collaborator

@mtache mtache commented Sep 12, 2024

Description

Use Codspeed to benchmark ANTA.
Use RESPX to mock eAPI responses.

The code generates an ANTA catalog from the unit test data at tests/units/anta_tests.
We should consider refactoring these data to use a dictionary with a tuple (Type[AntaTest], [unit test name]: str) as key to avoid duplicates.

It relies on a specific eAPI request ID format: ANTA-{test name}:{unit test name}:{command index}-{command UUID}. We currently patch ANTA commands to implement this, but we could consider implementing this directly in ANTA (the eAPI request ID is used for debugging purpose and its format has been arbitrary set to ANTA-{test name}-{command UUID}.

Benchmark performances

As of this writing, there are 556 unit tests for ANTA tests. The code will run 2 benchmarks with 1 device on which those 556 tests are run and another one with 2 devices for a total of 1112 tests run (556 on each device).
This takes 4 minutes to run on the current GitHub Actions runners, pytest-xdist (see https://docs.codspeed.io/benchmarks/python#running-benchmarks-in-parallel) has not improved the performances. This could be considered on a bigger runner, but we will loose stdout.

Requires

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have run pre-commit for code linting and typing (pre-commit run)
  • I have made corresponding changes to the documentation
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes (tox -e testenv)

Copy link
Contributor

This pull request has conflicts, please resolve those before we can evaluate the pull request.

Copy link
Contributor

Conflicts have been resolved. A maintainer will review the pull request shortly.

@mtache mtache requested review from a team and removed request for gmuloc and carl-baillargeon September 25, 2024 10:44
Copy link

sonarcloud bot commented Sep 26, 2024

Copy link
Contributor

@carl-baillargeon carl-baillargeon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mtache mtache merged commit 2214ff0 into aristanetworks:main Sep 26, 2024
10 checks passed
@mtache mtache deleted the ci/codspeed branch September 26, 2024 16:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants