-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
More flexible --test option #359
base: develop
Are you sure you want to change the base?
Conversation
That is a bad change. When working with test zones you want to be able to run just the specific test case and nothing else. Secondly, with that change, if Basic fails completely no other test case is run, and that is not what we want when testing and creating test zones. I am against a change that makes it impossible to run just the selected test case. |
All I'm hearing is "no" and "bad". Is there anything about this proposal that you do like? Also, I'm trying to understand your objection here. AFAICT you're saying that when a user requests that a single test case is run on a domain that doesn't fulfill the requirements posed by Basic, the requested test case still needs to run. Is that correct? And if so, could you explain why this is important, because it's not obvious to me. |
It would be good to be able to exclude test cases without creating a new profile. It would be good to be able to select two or more test cases and they are run without overlap. Sometimes it is probably good that Basic is run before a selected test.
I want to see what happens in that test case for the test zone even when there are errors. And I want minimal output without the extra output from Basic when testing test zones. Sometimes I have to run with Give the possibility to run a test without adding Basic. If Basic is not in the profile, why should Basic still be run? The magic could rather be to put the Basic test cases in the profile first, if there are any. |
Ok. Yeah, I see how this could be very valuable in development contexts. And I also don't particularly like that Basic is forced. I have an idea of how we could simplify things in Engine in a way that supports this. I think I'll make a draft PR there too. I won't make any more updates to the code here before Engine is in a better shape to support it. But if anyone has more feedback on the direction of this PR, please let me know. Maybe it'll affect the updates in Engine. |
Purpose
This PR implements flexible command line overriding of which test cases to run from the command line.
Context
This is an extension to the improvements in Extend --test option to allow passing only a testcase #333.
Replaces Better performance when running specified test. Function to exclude tests #130. The point of that PR is to allow users to exclude test cases from the command line. This PR includes the same functionality using a different command line syntax.
Leverages Ability to run single testcase via test_zone() zonemaster-engine#1312 to properly handle the Basic test module (c.f. Better performance when running specified test. Function to exclude tests #130 (comment)).
Handles the interaction of
--test
and--profile
in accordance with Better performance when running specified test. Function to exclude tests #130 (comment).For what it's worth, the idea for this design came to me in a discussion about Refactoring of function Zonemaster::Engine::Test::run_one() zonemaster-engine#1294.
Changes
The set of values accepted by
--test
is extended.The semantics are changed of repeated
--test
options. They used to be cumulative but now the last one takes precedence.Which test cases to include in the run is now controlled by setting the
test_cases
profile property and running Zonemaster::Engine->test_zone().How to test this PR
TBD