Switch from using the global agent to using global keypairs for tests #404

emmacasolin · 2022-07-11T02:15:02Z

Specification

#264 introduced our global agent which is used in our tests as a way to reduce setup time (since we no longer need to create an agent for every test). However, this solution complicates our tests by requiring all side effects to be reversed (some side effects cannot be undone, e.g. sigchain changes and some identities changes) and preventing tests that use the global agent from running in parallel.

An alternative solution is to have a pre-generated root private key that we can pass into the PolykeyAgent creation. This should sidestep the key generation process and speed up agent creation significantly.

Additional context

Fix Concurrent Testing Scalability - Find Blocking Code, Make them Non-Blocking #264 - Original issue for improving test scalability (where the global agent was introduced)
Test load balancing (migrating changes from TypeScript-Demo-Lib) #394 (comment) - Comment discussing the need to switch from using the global agent to global keys

Tasks

1. Add an override to KeyManager generation to provide a root private key to skip root key-pair generation.
2. Propagate this override up to the PolykeyAgent parameters
3. create tests confirming this behaviour.
4. Propagate this change up to pk agent start and pk bootstrap. Including new parameter + ENV variable.
5. Replace all usages of global agent with root key override.
6. Move global keypair fixtures to a common directory as tests/fixtures

The text was updated successfully, but these errors were encountered:

tegefaulkes · 2022-07-15T04:37:28Z

I'm going to change this up slightly and include it in MatrixAI/Polykey-CLI#10, #391. Instead of using mocking for this I'll add a parameter/ENV for setting the root private key to skip the generation step. We can extend this change to key mocking across all tests. That should be a separate issue however.

The changes here should be

Allow override to KeyManager generation to provide a root private key to skip keypair generation.
Propigate this override up to the PolykeyAgent parameters
create tests confirming this behaviour.
Propagate this change up to pk agent start and pk bootstrap. Including new parameter + env variable.
Replace all usages of global agent with root key override.

I also need to create a new issue to replace key mocking for all tests.

…erride` parameter for `createKeyManager` This will skip key generation and use the provided `PrivateKey` instead. This should speed up testing by skipping the key generation. Related #404

tegefaulkes · 2022-07-15T09:18:41Z

The neat thing is, with the new private-Key override we gain the feature of having a node signed by another node. This can be done by generating the node with a private key of another node and then renewing the keys to generate a new root key-pair signed by the previous one. The affect of which is having the node essentially signed by the other node.

CMCDragonkai · 2022-07-15T09:20:04Z

Loading a key that is created by another node is not exactly the same as cross-signing (PKI). See #154 for what I mean.

…ootstrap.ts` This should allow us to override the keypair generation with the provided private key. this will speed up agent starting. Still need to test this for `agent start` and `bootstrap`. Related #404

…mandBootstrap.ts` This should allow us to override the keypair generation with the provided private key. this will speed up agent starting. Note that the key is provided as a Pem. The `PrivateKey` type contains functions that get destroyed somewhere between `commandStart.ts` and `keyManager.createKeyManager`. So I'm using the Pem string to keep the type primitive Related #404

tegefaulkes · 2022-07-18T04:20:32Z

pk agent start and pk bootstrap take the --private-key-file parameter now. This will be a path to a file containing the private key in Pem format. SInce the Pem format includes newlines, we can't pass it directly as an ENV or parameter.

The Pem is passed through createPolykeyAgent to createKeyManager. It is converted to a keyPair inside the key manager during the key generation step. I originally tried using the PrivateKey format but that contained functions inside it that were lost at certain stages of passing the key around.

I'm starting on removing the global agent in the tests now.

CMCDragonkai · 2022-07-18T04:32:24Z

I'm thinking you should call this parameter --root-key. Do we use -file suffix in other parameters?

CMCDragonkai · 2022-07-18T04:33:01Z

And yes you can pass newlines as environment variable. It's not a problem. Variables in shell Env can have newlines.

CMCDragonkai · 2022-07-18T04:34:09Z

Also if this parameter is specified, you have to ignore any usage of recovery code. It needs to be ignored.

If the file is password protected you'll need both a password to decrypt it, and a password to encrypt it.

So --password would decrypt it if necessary and be the same password to encrypt.

tegefaulkes · 2022-07-18T04:36:52Z

I'll double check the Env.

As for the -file suffix. we have --recovery-code-file, --password-new-file, --password-file et al. I can change it to --root-key-file.

…-root-key-file` Related #404

This is a quick and easy way to create an agent with a pre-generated key. Related #404

Related #404

tegefaulkes · 2022-07-19T01:06:04Z

While this issue deals specifically with removing the global agent. The root key override can be used to optimise any test using a polykey agent, not just the bin tests. Such as the nodes, discovery, vaults? etc.

I think these changes would be slightly out of the scope of this issue. So I should create a new issue for it.

tegefaulkes · 2022-07-19T02:04:59Z

Looks like theres a bug with allows/disallows/gets gestalt permissions by node in tests/bin/identities/allowDisallowPermissions.test.ts:101. It throws an error when shutting down the agents right after the test. This started happening because I changed the afterAll to afterEach.

The error is

ErrorConnectionNotRunning: 
    at constructor_.getConnectionInfoByProxy (/home/faulkes/matrixcode/polykey/js-polykey/src/network/Proxy.ts:273:40)
    at constructor_.getConnectionInfoByProxy (/home/faulkes/matrixcode/polykey/js-polykey/node_modules/@matrixai/async-init/src/StartStop.ts:176:18)
    at constructor_.getRootCertChain (/home/faulkes/matrixcode/polykey/js-polykey/src/nodes/NodeConnection.ts:202:33)
    at constructor_.getRootCertChain (/home/faulkes/matrixcode/polykey/js-polykey/node_modules/@matrixai/async-init/src/CreateDestroy.ts:155:18)
    at constructor_.getExpectedPublicKey (/home/faulkes/matrixcode/polykey/js-polykey/src/nodes/NodeConnection.ts:218:31)
    at constructor_.getExpectedPublicKey (/home/faulkes/matrixcode/polykey/js-polykey/node_modules/@matrixai/async-init/src/CreateDestroy.ts:155:18)
    at /home/faulkes/matrixcode/polykey/js-polykey/src/nodes/NodeManager.ts:176:40
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at /home/faulkes/matrixcode/polykey/js-polykey/src/nodes/NodeConnectionManager.ts:204:16
    at withF (/home/faulkes/matrixcode/polykey/js-polykey/node_modules/@matrixai/resources/src/utils.ts:24:12)
    at constructor_.withConnF (/home/faulkes/matrixcode/polykey/js-polykey/src/nodes/NodeConnectionManager.ts:196:12)
    at constructor_.requestChainData (/home/faulkes/matrixcode/polykey/js-polykey/src/nodes/NodeManager.ts:152:7)
    at constructor_.setupDiscoveryQueue (/home/faulkes/matrixcode/polykey/js-polykey/src/discovery/Discovery.ts:370:21)

I think @emmacasolin wrote the tests for this. Do you have any idea why this is happening? It's coming from the discovery queue. Is it not ending properly during agent shutdown?

I've also used the key override where needed for `createPolykeyAgent`. Related #404

tegefaulkes · 2022-07-19T03:39:40Z

It's an edge case, the discovery queue is handling ErrorNodeConnectionDestroyed and ErrorNodeConnectionTimeout when doing requestChainData. But it's not handling ErrorConnectionNotRunning. It's a quick fix.

tegefaulkes · 2022-07-19T03:55:27Z

This is done now. I'll create a new issue for the key generation override for all the other tests.

tegefaulkes · 2022-07-25T01:39:04Z

This likely fixed #347, I'll include it in the PR.

#404 #420

…erride` parameter for `createKeyManager` This will skip key generation and use the provided `PrivateKey` instead. This should speed up testing by skipping the key generation. Related #404

…mandBootstrap.ts` This should allow us to override the keypair generation with the provided private key. this will speed up agent starting. Note that the key is provided as a Pem. The `PrivateKey` type contains functions that get destroyed somewhere between `commandStart.ts` and `keyManager.createKeyManager`. So I'm using the Pem string to keep the type primitive Related #404

…-root-key-file` Related #404

This is a quick and easy way to create an agent with a pre-generated key. Related #404

Related #404

I've also used the key override where needed for `createPolykeyAgent`. Related #404

#404 #420

…erride` parameter for `createKeyManager` This will skip key generation and use the provided `PrivateKey` instead. This should speed up testing by skipping the key generation. Related #404

…mandBootstrap.ts` This should allow us to override the keypair generation with the provided private key. this will speed up agent starting. Note that the key is provided as a Pem. The `PrivateKey` type contains functions that get destroyed somewhere between `commandStart.ts` and `keyManager.createKeyManager`. So I'm using the Pem string to keep the type primitive Added `PK_ROOT_KEY` ENV and changed `--private-key-file` to `--root-key-file` Related #404

This is a quick and easy way to create an agent with a pre-generated key. Related #404

I've also used the key override where needed for `createPolykeyAgent`. Related #404

#404 #420

emmacasolin added the development Standard development label Jul 11, 2022

tegefaulkes self-assigned this Jul 15, 2022

tegefaulkes mentioned this issue Jul 15, 2022

Packaged executable integration tests - initial integration tests with docker #391

Merged

10 tasks

tegefaulkes added a commit that referenced this issue Jul 18, 2022

fixin: added PK_ROOT_KEY ENV and changed --private-key-file to `-…

3a77fc6

…-root-key-file` Related #404

tegefaulkes added a commit that referenced this issue Jul 18, 2022

test: added setupTestAgent for creating test agents

84a671a

This is a quick and easy way to create an agent with a pre-generated key. Related #404

tegefaulkes added a commit that referenced this issue Jul 18, 2022

wip: removing global agent usuage

c5b1752

Related #404

tegefaulkes added a commit that referenced this issue Jul 19, 2022

test: removing global agent usage

41ba9cc

I've also used the key override where needed for `createPolykeyAgent`. Related #404

tegefaulkes mentioned this issue Jul 19, 2022

Override key generation for all tests #420

Closed

2 tasks

CMCDragonkai mentioned this issue Jul 22, 2022

Problem with tests/bin/keys/reset.test.ts and tests/bin/keys/renew.test.ts stopping the PolykeyAgent #347

Closed

tegefaulkes added a commit that referenced this issue Jul 26, 2022

fix: moved globalRootKeyPems.ts to fixtures

8c545a5

#404 #420

tegefaulkes added a commit that referenced this issue Jul 26, 2022

fixin: added PK_ROOT_KEY ENV and changed --private-key-file to `-…

4f6a2b5

…-root-key-file` Related #404

tegefaulkes added a commit that referenced this issue Jul 26, 2022

test: added setupTestAgent for creating test agents

1d84ee8

This is a quick and easy way to create an agent with a pre-generated key. Related #404

tegefaulkes added a commit that referenced this issue Jul 26, 2022

wip: removing global agent usuage

c0c68b4

Related #404

tegefaulkes added a commit that referenced this issue Jul 26, 2022

test: removing global agent usage

8612f3c

I've also used the key override where needed for `createPolykeyAgent`. Related #404

tegefaulkes added a commit that referenced this issue Jul 26, 2022

fix: moved globalRootKeyPems.ts to fixtures

36f377e

#404 #420

tegefaulkes added a commit that referenced this issue Jul 27, 2022

test: added setupTestAgent for creating test agents

7a4cdde

This is a quick and easy way to create an agent with a pre-generated key. Related #404

tegefaulkes added a commit that referenced this issue Jul 27, 2022

test: removing global agent usage

cb17957

I've also used the key override where needed for `createPolykeyAgent`. Related #404

tegefaulkes added a commit that referenced this issue Jul 27, 2022

fix: moved globalRootKeyPems.ts to fixtures

29e8f4e

#404 #420

tegefaulkes added a commit that referenced this issue Jul 27, 2022

fix: moved globalRootKeyPems.ts to fixtures

6e3594d

#404 #420

CMCDragonkai closed this as completed in #391 Jul 27, 2022

CMCDragonkai added the r&d:polykey:supporting activity Supporting core activity label Jul 10, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Switch from using the global agent to using global keypairs for tests #404

Switch from using the global agent to using global keypairs for tests #404

emmacasolin commented Jul 11, 2022 •

edited by CMCDragonkai

Loading

tegefaulkes commented Jul 15, 2022

tegefaulkes commented Jul 15, 2022

CMCDragonkai commented Jul 15, 2022

tegefaulkes commented Jul 18, 2022

CMCDragonkai commented Jul 18, 2022

CMCDragonkai commented Jul 18, 2022

CMCDragonkai commented Jul 18, 2022

tegefaulkes commented Jul 18, 2022

tegefaulkes commented Jul 19, 2022

tegefaulkes commented Jul 19, 2022 •

edited

Loading

tegefaulkes commented Jul 19, 2022

tegefaulkes commented Jul 19, 2022

tegefaulkes commented Jul 25, 2022

Switch from using the global agent to using global keypairs for tests #404

Switch from using the global agent to using global keypairs for tests #404

Comments

emmacasolin commented Jul 11, 2022 • edited by CMCDragonkai Loading

Specification

Additional context

Tasks

tegefaulkes commented Jul 15, 2022

tegefaulkes commented Jul 15, 2022

CMCDragonkai commented Jul 15, 2022

tegefaulkes commented Jul 18, 2022

CMCDragonkai commented Jul 18, 2022

CMCDragonkai commented Jul 18, 2022

CMCDragonkai commented Jul 18, 2022

tegefaulkes commented Jul 18, 2022

tegefaulkes commented Jul 19, 2022

tegefaulkes commented Jul 19, 2022 • edited Loading

tegefaulkes commented Jul 19, 2022

tegefaulkes commented Jul 19, 2022

tegefaulkes commented Jul 25, 2022

emmacasolin commented Jul 11, 2022 •

edited by CMCDragonkai

Loading

tegefaulkes commented Jul 19, 2022 •

edited

Loading