Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move Cross-field Validation out of Pipeline et al #31

Open
ghickman opened this issue Apr 11, 2022 · 1 comment
Open

Move Cross-field Validation out of Pipeline et al #31

ghickman opened this issue Apr 11, 2022 · 1 comment

Comments

@ghickman
Copy link
Contributor

After moving job-runner's project.py into this library we'll have all validation done by pydantic. This is mostly very good, but where it falls down is where we have to mutate various values when reading them. Two good examples of this are:

  • changes to the config schema based on version, for instance outputs being unique
  • the run command being mutated based on other options and version

This has mostly cropped up when trying to type check various validator functions on the pydantic models, particularly those with pre=True set which are dealing with the raw data.

I think a better approach here is going to be:

  • use Pydantic to validate the raw data with no mutations
  • create a dataclass/attrs class as the final config instance we'll return
  • add functions/methods to translate from the pydantic model to the final class doing all the mutation and extra validation we want along the way
  • wrap pydantic and the translation functions so our error output is consistent across the whole process (likely just sticking with ProjectValidationError).
@ghickman
Copy link
Contributor Author

Some of the pain here, specificially mutating the run key, has gone away thanks to pulling ActionSpecification back into job-runner (since it's job-runner specific).

This might be useful to #42, but we need to do more investigation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant