Store e2e test artifacts between test runs to catch regressions

[klochowicz]

I've been thinking how we could solve the following problems that we sometimes encounter:

• failing database migrations
• breaking changes in Rust-DLC

I realised that it might be a quite simple solution for this, by storing the databases from e2e test runs.
This way we would know whether we're about to introduce a regression in maker, app, or coordinator.

unsolved problem is how to deal with the fact when we decide that it's OK to break things - perhaps manually clearing the action cache via the CLI or web interface would be enough in such situation.

[bonomat]

I like the idea. @holzeis and me brainstormed about this as well.

A few thoughts we had are:

• a cache is not a reliable storage, meaning, Github could randomly delete the cache, maybe a separate repository would be a better place?
• if we store the coordinator's data we will also need to store the blockchain data + the database
• our apps currently use random data directories and generate seeds/etc for every run. This needs to be changed to use deterministic data. However, having fixed data to start from might lead to flaky tests because one test run could change the stored data

[klochowicz]

a cache is not a reliable storage, meaning, Github could randomly delete the cache, maybe a separate repository would be a better place?

it's not random at all, the eviction policy is clearly defined: a cache is not a reliable storage, meaning, Github could randomly delete the cache, maybe a separate repository would be a better place?

https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy

as long as we're continuously updating it (we are) and we're within our quota storage, we should be good.
If this won't satisfy our needs, then of course we can change the strategy, but IMHO it's enough for the PoC of this feature.

if we store the coordinator's data we will also need to store the blockchain data + the database

It would be nice, but I don't think we need to (at least in the beginning).
This would become more important in case we want to expand our tests more. e.g. being able to have an open position between re-runs.

our apps currently use random data directories and generate seeds/etc for every run. This needs to be changed to use deterministic data. However, having fixed data to start from might lead to flaky tests because one test run could change the stored data

I don't think it needs to be changed in any way - if we restore the data from cache, we are not generating new seed.
I would probably nuke the cache manually every now and then (to keep the size small, and ensure no flakiness - also, in a situation when we know we're breaking backwards compatibility and we're OK with it)

However, If we decided we want to have them more deterministic, we could have a separate cache just for the seeds, and different for the data (2-stage).

[holzeis]

This would become more important in case we want to expand our tests more. e.g. being able to have an open position between re-runs.

The main benefit to me of using cached state in between versions is so that we are able to find upgrade issues.

If we wouldn't keep the blockchain, the channels would break and would get force closed on every run. Meaning that we'd have to create a new channel every time. I don't see much difference to running tests without the cache.

[bonomat]

it's not random at all, the eviction policy is clearly defined
docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy

Fair, it's not random and well defined :) I wouldn't trust a cache to store data which we want to base assumptions on. I think something more reliable should be used. Also, a different place than github cache would allow us to run these tests locally as well.

if we store the coordinator's data we will also need to store the blockchain data + the database
It would be nice, but I don't think we need to (at least in the beginning). This would become more important in case we want to expand our tests more. e.g. being able to have an open position between re-runs.

As Richie said: It depends on what you want to store. As soon as you want to store any position-related data you will need the blockchain data as well as otherwise the position will be closed and the channel force-closed.

---

I think we will need to define more clearly what we want to achieve with the cached/stored test data.
I'm not sure if we are thinking of the same thing currently :)

[Restioson]

I guess there is a difference semantically between a cache and storing test data to prevent regressions.

Caching imo means that it's purely done to speed something up. I.e, it is transparent, and wouldn't affect the outcome of the tests at all. Storing test data has its own application in regression testing (almost like guideline testing or snapshot testing with known-difficult snapshots). Hence, it tests something distinct.

[klochowicz]

I agree. I'll rename the discussion to "Store test artifacts..."

[klochowicz]

a few questions / comments that rose from discussion, hopefully we can get more aligned with this @bonomat :) please feel free to respond / clarify / add more requirements!

1. do we even want to support upgrading the app with a position open?
   I believe that this is not essential, and we could remove this self-imposed requirement. If anyone updates with a position open, then they're doing something that's not supported, risky etc. Note: we want to support keeping the lightning channel open during the updates, and I guess the subchannel state regressions are the ones we want to catch?

2. do we want to run these tests locally by pulling the data from remote setup?

We currently can test these kinds of regressions locally, if you run just e2e and then switch a branch and run it again without wiping. It should be trivial to reproduce the same regression if we know the pre and post-update commit hashes.

3. Cache vs having it in a repo? starting with db vs storing the blockchain too?

• I'm not opposed to either of these, just wanted to limit the scope for initial PoC to see whether it brings tangible improvements into our workflow before we go far with it

4. Can we get any useful results without storing the blockchain?

• we can spot regressions / failed migrations etc in trading DB
• we should be able to spot regressions in sled DB as well - of course, it will be upset about not having the blockchain data, but it might be handled gracefully? (I haven't tested this, and I concur, perhaps adding the few directories to the "cache" might be faster and more error-proof).