Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add integration tests #26

Open
pederhan opened this issue Jan 26, 2023 · 0 comments
Open

Add integration tests #26

pederhan opened this issue Jan 26, 2023 · 0 comments
Labels
help wanted Extra attention is needed tests Automated tests

Comments

@pederhan
Copy link
Member

pederhan commented Jan 26, 2023

The various commands that interact with the Harbor API need some sort of integration tests to ensure they work as expected. To facilitate this, we either need to set up a Harbor environment we can test on, or we need to somehow be able to re-use the tests from harborapi that set up mock endpoints, on which we can test our commands.

Straight-up copying all the mock endpoints from harborapi tests and re-using them in our tests is not an option. It doubles the work required to maintain two almost identical test sets, and introduces the possibility of the two test sets drifting apart over time.

Re-using harborapi mock endpoints

As of now, the tests in harborapi define the mock endpoints inside the test functions, and therefore it's not possible to re-use them without heavily re-writing the tests. This brings in another problem, having to re-write tests in harborapi to accommodate Harbor CLI probably brings in a degree of coupling I am not comfortable with. As such, maybe it's better if Harbor CLI actually tests against a real Harbor test instance somehow?

Test against real Harbor instance

The best test regime would be to actually run against a live, deterministic Harbor environment. I am not sure how we can do this comfortably with GitHub actions, and it might require either a dedicated test runner or some sort of local testing regime (bad). If Harbor could be run as a single container image, it wouldn't be much of an issue, but it is unfortunately composed of several different microservices running together (managed by Docker-compose, k8s, etc.) We could potentially run Docker-in-docker or something similar to orchestrate this in GitHub actions, but that sounds less than ideal and maybe not even possible (?)

Use an API integration test service (Wiremock, etc.)

There are both hosted and self-hostable API integration test tools out there, such as Wiremock, which allows us to use a Swagger definition (which we have from VMWare here) to set up mock endpoints. This, however, does not mock all the endpoints correctly, as a lot of the endpoint paths are quite dynamic, such as the ones for accessing projects, repositories, artifacts, and tags.

@pederhan pederhan added help wanted Extra attention is needed tests Automated tests labels Jan 26, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted Extra attention is needed tests Automated tests
Projects
None yet
Development

No branches or pull requests

1 participant