Integration tests for deploy targets on all deployment platforms (docker, linux/ubuntu, windows, macos)

[tegefaulkes]

Specification

after creating the packaged executable using pkg we need to run a variety of tests on it. We need to do this in all of the main environments such as Linux, windows and MAC. We will also need to test the docker image but I'm nut sure the same tests can apply in that case.

All of the main tests we need to do are already done by the bin tests. So we just need to repurpose all or a subset of these tests and run them on the target platform. Assuming the tests themselves don't need some kind of modification to work on the target platforms. Then the only changes to the tests that need to be made are to exec the packaged executable instead of the ts/js code.

Keep in mind that most of the tests don't actually exec pk to run them but are run in thread using mocks to capture output. With that in mind we will need to review the tests to see if any of them require access to the thread to apply mocks for the tests. We may also apply a subset of the tests to run instead of running all of them.

I think we can prioritise the tests based on complexity. starting with just running Polykey to full integration. I think the stages go as follows.

1. Running polykey
   1. bin/agent
   2. bin/bootstrap.test.ts
   3. bin/polykey.test.ts
2. local local
   1. bin/keys
   2. bin/sessions.test.ts
   3. bin/vaults
   4. bin/secrets
3. network level
   1. bin/nodes
   2. bin/notifications
   3. bin/identities

We will need a way to specify that we are running the tests targeting the packaged executable. This will be specified as a global however it will be set via and env variable. So the global check the env and provides the parameter to the tests.

The tests will need to check the target and change it's behaviour based on that. First by changing modifying how the commands are run. The utility functions pkStdio, pkExec and pkSpawn will need to be modified to run the packaged target if that global parameter is set. The functions will need to function with the same API so the the new behaviour will be a drop in replacement.

Most tests utilise okStdio which is running the command 'in thread' with mocking. This could be a problem because if any tests is doing mocking on the command itself then we can't run it in a separate thread using exec. This means the test itself could be incompatible with testing the packaged executable. Either a change to the test needs to be made or the test needs to be filtered out when testing the packaged executable. This is unlikely to be a problem since there seems to be little mocking and any mocking will be done against the polykey instance running 'in thread' anyway.

Additional context

• Related ci: merge staging to master Polykey#379
• Fix Failing Build Tests on Windows & MacOS #14

Tasks

1. Allow setting a global variable specifiying the path to the packaged executable under test
2. Allow bin tests to target the specified path when running the tests
3. Fix up or filter out any tests that are incompatable with using the packaged executable for testing.
4. test manually for each target platform to ensure tests are working.
5. configure CI/CD to use integration tests for each target platform and ensure that it works.

[CMCDragonkai]

The spec for this needs to be fleshed out with respect to:

1. Docker integration tests - all the caveats that are related to running with dind, such as not needing a pid namespace and possibility the inability to mount host network namespace... and all other docker-related bugs (like the lack of $HOME), and perhaps the change to using Entrypoint`
2. Windows integration tests - different signal behaviour, need to integrate Fix Failing Build Tests on Windows & MacOS #14
3. Mac integration tests - also integrate Fix Failing Build Tests on Windows & MacOS #14

Make this issue an epic, rename it, and create 3 subissues for docker, windows and mac.

Then in your PR, target them one at a time, or target all 3. Probably all 3 targets and then @emmacasolin can join you in the same PR to work out all the bug fixes.

[CMCDragonkai]

We also need to spec out the changes to the utility functions of pkStdio, pkExec, pkSpawn and more to be more generic. @tegefaulkes please add a listed out spec for each behaviour.

[CMCDragonkai]

The docker integration tests will have different namespace behaviour:

1. Rather than --network host, you'll need an IP address and Port mapping, make sure this works on the CI/CD too
2. Rather than --pid host you'll need handling of the signals by the PK agent
3. With --userns host and --user "$(id -u)" this may still be necessary locally, in order to allow the files written to be readable by the test code which is running as part of our local user. But on the CI/CD this both may be the root user. But experiment with this.

Need to see if the CI/CD job can actually do mount binding with dind.

[CMCDragonkai]

Because this will also depend on test fixes for mac and windows at the build stage in CI/CD, we will need to do this in a pipeline style. Additional tests should being written for integration into testnet.polykey.io.

[CMCDragonkai]

Ok so let's focus on docker as a priority. So @tegefaulkes and @emmacasolin will work together in a pipeline on docker and testnet integration.

One to work any bug/test failures with doing docker testing (and associated source code changes).

The other to work on integration to testnet.polykey.io by writing integration tests that contact the testnet.polykey.io.

Because testnet ultimately requires the docker integration to be working, this is good way of focusing on fixing up one area of the application of PK.

[CMCDragonkai]

I've added MatrixAI/Polykey#432 to track standardisation of the tests/utils/exec.ts functions.

[CMCDragonkai]

Moving to Polykey-CLI.

[tegefaulkes]

The mac and windows bulding in CI are working again. But to address this we need to add integration tests for mac, win and linux. They don't need to be complex. We can use the docker integration tests as a template and they're only testing start and connecting to a network.