rerun integration test suites from some components

[cgwalters]

There's a natural tension between a project like this and the per-project test suites we have in atomic, rpm-ostree etc. Particularly because we aren't running this on upstream PRs.

Anyways, I think a relatively simple thing we could do to avoid duplication would be to run some of the upstream integration tests here again. We'll need to be sure that the upstream suites don't actually rebuild the code, we just want to use them as a source of tests.

[miabbott]

I worry about increasing the amount of time that would be added to the sanity/smoketest runs by including additional integration tests.

I do see value in re-running the tests, but maybe we should run them after the smoketests were successful?

I guess it would depend on which integration tests we would choose.

[cgwalters]

The way I'd think of this is, to start - let's define the total amount of time we should take for a smoketest. Is it 5 minutes? 30? 2 hours? Then, which tests can we fit into that, and which run after? (This crosses a bit with resources; we could obviously parallellize more of the testing)

[cgwalters]

Though really the main thing I was trying to achieve with this is that as soon as we do it, it becomes a lot more natural to share code/methodology between this project and the different upstreams.

[jlebon]

Rerunning upstream tests is nice, but being able to run a-h-ts in upstream PRs is even nicer I think. I mentioned this in CentOS/sig-atomic-buildscripts#229 -- if we have a known working image + tests, then we can use that to validate PRs against those same tests using those same images.

[miabbott]

The way I'd think of this is, to start - let's define the total amount of time we should take for a smoketest. Is it 5 minutes? 30? 2 hours?

TBH, I don't believe we've done any baselining to figure out how long the tests take right now. I'll work on getting that data.

[jlebon]

Rerunning upstream tests is nice, but being able to run a-h-ts in upstream PRs is even nicer I think.

I was thinking about this today, and I think I was subtly confusing two distinct things when I made this comment.

Today, a-h-t is mostly used as a gating test for branch promotion (in both Fedora and CAHC's smoketested branch). So it plays a critical role in ensuring that the final releases are sane. In that capacity, it definitely makes sense to rerun integration tests of upstream projects, which are readily available and well-maintained.

This would not be a duplication of efforts, since in upstream projects, PRs are tested against the latest stable releases, not candidate builds that a-h-t is targeted for (e.g. in PAPR today, none of the repos use the fedora-atomic/25/x86_64/testing/docker-host branch).

But another place where a-h-t would be really useful (and what my original comment alluded to) would be flipping this around and running a-h-t in upstream PRs using the latest stable base image + the built binaries. You then instantly get a comprehensive integration test on every change. (Though this is probably something that we want to run on merge only, i.e. pulls: false in PAPR).

[miabbott]

(5) Run integration tests for ostree as part of [a-h-t] sanity suite

[jlebon]

One thing that might be worth looking at for this is the dist-git test for rpm-ostree: https://src.fedoraproject.org/rpms/rpm-ostree/blob/master/f/tests/tests.yml. We run the vmcheck testsuite there on the artifact. It uses fedpkg to retrieve the srpm and extract the testsuite from that.

Of course, the simplification we can make there is that the dist-git repo is already checked out at the right revision corresponding to the rpm-ostree that's in the artifact. But here, you'd have to figure out a way to get to the right revision yourself. You can probably just directly query rpmdb for the %{VERSION} and git checkout that tag from the GitHub repo.

One thing to note though is that if we end up running a-h-t as part of the CI pipeline, we'd essentially be rerunning the vmcheck suite twice: once as part of the rpm-ostree gate, and once more as part of a-h-t. Maybe we should run only a subset of the tests when running as part of the CI pipeline? (But otherwise, run it all?)