chore: split kuttl test steps

[realshuting]

This PR attempts to fix the flaky kuttl test refresh-volumes-in-pods, ref kyverno/kyverno#8334.

I'm able to consistently reproduce the issue by removing the sleep 5 command, the tests ran in Kyverno CIs could take longer than 5s. I'm splitting the check step into 2 and adding more delay to see if it helps.

        ]
    logger.go:42: 16:43:29 | refresh-volumes-in-pods/4-check | running command: [sh -c val=$(kubectl exec pod01 -n refresh-vols-ns -- cat /mnt/foo/foo)
        if [ "$val" = "bar" ]; then exit 0; else exit 1; fi
        ]
    logger.go:42: 16:43:29 | refresh-volumes-in-pods/4-check | command failure, skipping 3 additional commands
    case.go:367: failed in step 4-check
    case.go:369: exit status 1

[chipzoller]

Wait, why did CI not run here?

[chipzoller]

Kuttl tests are supposed to run for these PRs.

[realshuting]

Well I don't think this PR fixes the root cause, the other test failed with timeout during cleaup:

    logger.go:42: 09:40:06 | require-base-image/99-delete | test step completed 99-delete
    logger.go:42: 09:40:06 | require-base-image | require-base-image events from ns kuttl-test-saved-frog:
    logger.go:42: 09:40:06 | require-base-image/2-manifests | CronJob:require-base-image-ns/goodcronjob01 deleted
    logger.go:42: 09:40:06 | require-base-image/2-manifests | Deployment:require-base-image-ns/gooddeployment01 deleted
    logger.go:42: 09:40:06 | require-base-image/2-manifests | Pod:require-base-image-ns/goodpod03 deleted
    logger.go:42: 09:40:06 | require-base-image/2-manifests | Pod:require-base-image-ns/goodpod02 deleted
    logger.go:42: 09:40:06 | require-base-image/2-manifests | Pod:require-base-image-ns/goodpod01 deleted
    step.go:215: timed out waiting for the condition
    logger.go:42: 09:41:36 | require-base-image | Deleting namespace: kuttl-test-saved-frog

[realshuting]

If cleanups are required, can we delete test resources in separate tasks instead of deleting everything at one time kubectl delete all --all --force --grace-period=0 -n require-base-image-ns?

[chipzoller]

Sure, can try that.