Should we setup e2e tests separately from integ tests?

[jmazanec15]

Description

For this comment from @heemin32 (#2200 (review)), I was thinking I could add recall tests for this as well, but this can come with cost of adding minutes to the CI.

That being said, should we have e2e tests that we know take longer to run and could be triggered selectively by a maintainer and/or on a nightly or weekly basis? Thus, for most changes we wouldnt need to run these. What does everyone think?

I think AD has setup their repo to do this. See https://github.com/opensearch-project/anomaly-detection/blob/main/build.gradle#L321-L369.

[jmazanec15]

Discussed step: As first step, we should split up e2e tests and integ tests so that they can run in parallel for CI. In future, we can revisit selectively running.