Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Richer validation output #79

Open
halfflat opened this issue Aug 26, 2019 · 0 comments
Open

Richer validation output #79

halfflat opened this issue Aug 26, 2019 · 0 comments
Labels
enhancement New feature or request validation Relates to nsuite validation component

Comments

@halfflat
Copy link
Contributor

The pass/fail output from a validation run is too simple, really, to be very useful. The validation runs should:

  • Catch definitely wrong behaviour.
  • Allow comparisons of fidelity between builds and simulators.

Proposal:

  • Make the pass criterion for the tests quite lax, so that FAIL really means that something went wrong.
  • Produce a 'standard' report output from the validation tests. This would follow a well defined schema and contain a list of derived values of interest, their pass thresholds (and less than, greater than, etc.), and metadata for sim and nsuite version.
  • Manually running the validation tests in a new 'verbose' mode would output a human- and script- readable version of this data.
  • Reports from multiple tests in the same validation run should be able to be combined and presented in a form that is easy for programmatic consumption, e.g. for presentation in Jenkins or other tools.
@halfflat halfflat added the enhancement New feature or request label Aug 26, 2019
@halfflat halfflat added the validation Relates to nsuite validation component label Feb 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request validation Relates to nsuite validation component
Projects
None yet
Development

No branches or pull requests

1 participant