You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The various commands that interact with the Harbor API need some sort of integration tests to ensure they work as expected. To facilitate this, we either need to set up a Harbor environment we can test on, or we need to somehow be able to re-use the tests from harborapi that set up mock endpoints, on which we can test our commands.
Straight-up copying all the mock endpoints from harborapi tests and re-using them in our tests is not an option. It doubles the work required to maintain two almost identical test sets, and introduces the possibility of the two test sets drifting apart over time.
Re-using harborapi mock endpoints
As of now, the tests in harborapi define the mock endpoints inside the test functions, and therefore it's not possible to re-use them without heavily re-writing the tests. This brings in another problem, having to re-write tests in harborapi to accommodate Harbor CLI probably brings in a degree of coupling I am not comfortable with. As such, maybe it's better if Harbor CLI actually tests against a real Harbor test instance somehow?
Test against real Harbor instance
The best test regime would be to actually run against a live, deterministic Harbor environment. I am not sure how we can do this comfortably with GitHub actions, and it might require either a dedicated test runner or some sort of local testing regime (bad). If Harbor could be run as a single container image, it wouldn't be much of an issue, but it is unfortunately composed of several different microservices running together (managed by Docker-compose, k8s, etc.) We could potentially run Docker-in-docker or something similar to orchestrate this in GitHub actions, but that sounds less than ideal and maybe not even possible (?)
Use an API integration test service (Wiremock, etc.)
There are both hosted and self-hostable API integration test tools out there, such as Wiremock, which allows us to use a Swagger definition (which we have from VMWare here) to set up mock endpoints. This, however, does not mock all the endpoints correctly, as a lot of the endpoint paths are quite dynamic, such as the ones for accessing projects, repositories, artifacts, and tags.
The text was updated successfully, but these errors were encountered:
The various commands that interact with the Harbor API need some sort of integration tests to ensure they work as expected. To facilitate this, we either need to set up a Harbor environment we can test on, or we need to somehow be able to re-use the tests from
harborapi
that set up mock endpoints, on which we can test our commands.Straight-up copying all the mock endpoints from
harborapi
tests and re-using them in our tests is not an option. It doubles the work required to maintain two almost identical test sets, and introduces the possibility of the two test sets drifting apart over time.Re-using
harborapi
mock endpointsAs of now, the tests in
harborapi
define the mock endpoints inside the test functions, and therefore it's not possible to re-use them without heavily re-writing the tests. This brings in another problem, having to re-write tests inharborapi
to accommodate Harbor CLI probably brings in a degree of coupling I am not comfortable with. As such, maybe it's better if Harbor CLI actually tests against a real Harbor test instance somehow?Test against real Harbor instance
The best test regime would be to actually run against a live, deterministic Harbor environment. I am not sure how we can do this comfortably with GitHub actions, and it might require either a dedicated test runner or some sort of local testing regime (bad). If Harbor could be run as a single container image, it wouldn't be much of an issue, but it is unfortunately composed of several different microservices running together (managed by Docker-compose, k8s, etc.) We could potentially run Docker-in-docker or something similar to orchestrate this in GitHub actions, but that sounds less than ideal and maybe not even possible (?)
Use an API integration test service (Wiremock, etc.)
There are both hosted and self-hostable API integration test tools out there, such as Wiremock, which allows us to use a Swagger definition (which we have from VMWare here) to set up mock endpoints. This, however, does not mock all the endpoints correctly, as a lot of the endpoint paths are quite dynamic, such as the ones for accessing projects, repositories, artifacts, and tags.
The text was updated successfully, but these errors were encountered: