Seed node not adding details of connecting node to its NodeGraph

[joshuakarp]

Specification

Authentication a node on the other side of a connection is done buy the Proxy class when a connection is made. This includes obtaining the NodeId, certificates and connection info. I assume that the certificates are authenticated at this stage. This is all the information that we need to authenticate and add a node to the node graph with the added benefit of it being obtained at the time a connection is established.

We need to add a callback to the Proxy that is triggered with the NodeId, remoteHost and remotePort information whenever a connection is made. For simplicity this will be triggered when a ForwardConnection and ReverseConnection is made. So Proxy needs to take a (remoteNodeId, remoteHost, remotePort) => void callback that is triggered whenever a connection is established and authenticated.

A new event needs to be added to the EventBus for this connection event. The handler for this event needs to call NodeManager.setNode() to add the information to the nodeGraph.

Additional context

• Proxies was updated in Refactor proxies to use the same UTP socket for external connections #360 , refactor: combined ForwardProxy and ReverseProxy into Proxy #361. Changes affect this.
• Uses the same events system from Propagate root keypair changes to domains external to KeyManager #312 CLI and Client & Agent Service test splitting #311 ? I'm not sure where it was implemented.

Tasks

• 1. Add a connectionEstablishedCallback to the Proxy that is triggered whenever a connection is authenticated and established.
• 2. Add a new event to the events EventBus in the PolykeyAgent and have this triggered by the callback.
• 3. Add an event handler for the new event that calls the NodeManager.setNode() method thus adding the node to the nodeGraph.
• 4. add tests for;
  • Proxy triggering the callback with the correct information.
  • Integration test for the PolykeyAgent to check that the event properly propagates and updates the NodeGraph.

[joshuakarp]

Going back to the Kademlia protocol, we have the following about new nodes joining the network from the original research paper https://www.scs.stanford.edu/~dm/home/papers/kpos.pdf:

To join the network, a node u must have a contact to an already participating node w. u inserts w into the appropriate k-bucket. u then performs a node lookup for its own node ID. Finally, u refreshes all k-buckets further away than its closest neighbor. During the refreshes, u both populates its own k-buckets and inserts itself into other nodes’ k-buckets as necessary.

And on "refreshing":

Refreshing means picking a random ID in the bucket’s range and performing a node search for that ID.

We don't currently have this notion of "refreshing", nor do we perform this refresh when a node enters the network. Given that we connect to an initial seed node on bootstrapping, we have a different kind of "refreshing"/synchronisation process to initially populate their database:

1. Keynode is created and started
2. Connections are established to seed node/s
   • NOTE: After this issue is resolved, seed nodes should be adding this new node ID to their own NodeGraph
3. Keynode performs a gRPC request to the seed node/s to get the closest k nodes to its own node ID
4. Keynode adds these k nodes from each seed node to their own NodeGraph

---

However, this discussion's important part: During the refreshes, u both populates its own k-buckets and inserts itself into other nodes’ k-buckets as necessary.

This brings up an interesting point: when a node wants to add some other NodeId -> host + port mapping into their NodeGraph, do we want this other node to also add the original node to their own NodeGraph? I remember having some discussion about this very early on, and we simply made the assumption that we couldn't expect this to occur in symmetry on both sides.

When a node "found" a new node ID pointing to some host and port, our initial implementation of Kademlia required a connection to be established to that node, such that we could verify that this node ID and host/port was valid. This has since been removed though, and we'll be looking at a different means of doing this in #322.

Because this connection establishment was required though, we could have used a similar approach to what we'll be implementing with the seed node: on connection establishment, we should automatically add this connecting node to our NodeGraph.

[joshuakarp]

My concluding point from the above is basically that we should be generalising this for all nodes (and not just seed nodes). When a node connects to any node in the Polykey network, the other node should be adding the connecting node to its own NodeGraph.

Therefore, we'd have the following means of the NodeGraph being updated:

• manual insertion with pk nodes add
• node "finding" with pk nodes find, when it contacts its own local nodes to find the closest k nodes to some target node (these closest found nodes are added to our NodeGraph)
• when some node connects to another target node (and the connecting node is then automatically added to the target node's NodeGraph - it's assumed that the connecting node would already have the target node in its NodeGraph)

[joshuakarp]

For generalising this to all nodes, that's easy enough. Given that we require hole punching for a connection to be established between two arbitrary keynodes, we can push this NodeGraph.setNode call into a couple of places:

1. NodeConnectionManager.holePunchReverse: once we start sending hole-punching packets back to the target, we could also add the node to our NodeGraph
2. nodesHolePunchMessageSend (agent service function): just before the call to holePunchReverse, could also add the node here

The first option makes the most sense to me, given that it's directly in the nodes domain instead of right at the boundary.

---

However, the seed nodes are a special case. The seed nodes are not expected to be behind a NAT, thus require no hole-punching or any intervention on their part to support a node's connection establishment. So, what program logic can we link this into? The lower level networking?

[CMCDragonkai]

This sounds like a server side interceptor problem.

We don't have server side interceptor available unfortunately and @tegefaulkes needs it for acquiring authentic connecting node information in #342.

Yes this could have been done at the lower level involving network domain like optimising the ping command to just use UTP instead of going all the way to GRPC.

Basically node A must connect to node B somehow and trigger node B to add node A to its DHT.

Did you check other implementations if they do this?

[CMCDragonkai]

Referencing kademlia paper and kademlia wikipedia page:

A node that would like to join the net must first go through a bootstrap process. In this phase, the joining node needs to know the IP address and port of another node—a bootstrap node (obtained from the user, or from a stored list)—that is already participating in the Kademlia network. If the joining node has not yet participated in the network it computes a random ID number, which by virtue of being a very large random number is extremely likely not to be already assigned to any other node. It uses this ID until leaving the network.
The joining node inserts the bootstrap node into one of its k-buckets. The joining node then performs a node lookup of its own ID against the bootstrap node (the only other node it knows). The "self-lookup" will populate other nodes' k-buckets with the new node ID, and will populate the joining node's k-buckets with the nodes in the path between it and the bootstrap node. After this, the joining node refreshes all k-buckets further away than the k-bucket the bootstrap node falls in. This refresh is just a lookup of a random key that is within that k-bucket range.

The key sentence is The "self-lookup" will populate other nodes' k-buckets with the new node ID, and will populate the joining node's k-buckets with the nodes in the path between it and the bootstrap node.

The refreshing logic discussed later is a separate problem and would need to be a new issue. That's about keeping the nodegraph up to date and would be triggered on max-TTLs of a bucket and when joining the network.

So the look up procedure right now involves agent to agent communication, Node A will call nodesClosestLocalNodesGet service handler of Node B in order to acquire the closest nodes to itself. The lookup procedure is a dynamic loop:

Node lookups can proceed asynchronously. The quantity of simultaneous lookups is denoted by α and is typically three. A node initiates a FIND_NODE request by querying to the α nodes in its own k-buckets that are the closest ones to the desired key. When these recipient nodes receive the request, they will look in their k-buckets and return the k closest nodes to the desired key that they know. The requester will update a results list with the results (node ID's) it receives, keeping the k best ones (the k nodes that are closer to the searched key) that respond to queries. Then the requester will select these k best results and issue the request to them, and iterate this process again and again. Because every node has a better knowledge of its own surroundings than any other node has, the received results will be other nodes that are every time closer and closer to the searched key. The iterations continue until no nodes are returned that are closer than the best previous results. When the iterations stop, the best k nodes in the results list are the ones in the whole network that are the closest to the desired key.

It is during the handler where the Node B needs to add Node A to its own node graph. But it is not just done on Node B, this is done on every node that Node A calls nodesClosestLocalNodesGet.

For example, with Node A, Seed Node and Node B.

https://excalidraw.com/#json=CupwHTnu8UN6SmhTiUV8C,OtRF9UZIbSd9x1UurmV-bA

The process is actually doable on any node, not just seed nodes.

Adding Node A to any node graph requires Node A's ingress host and ingress port. For example, assume Node A and Node B. Node A's ingress host and ingress port information is not "naturally" part of any request sent from Node A to Node B. This is due to our network design.

The original kademlia design was based on UDP. And in UDP, you can send and receive UDP packets on the same port number. Therefore upon receiving a lookup request in UDP, you could just add the host and port directly into the node graph. However this is not the case with our system.

In our network design, not only do we have proxies that are doing TLS termination, Node A's ingress host and ingress port is not on the ReverseProxy.getConnectionInfoByProxy. Instead it returns the type ConnectionInfo which is actually filled with:

    return {
      nodeId: clientNodeIds[0],
      certificates: clientCertificates,
      egressHost: conn.host,
      egressPort: conn.port,
      ingressHost: this.ingressHost,
      ingressPort: this.ingressPort,
    };

And as you can see the ingressHost and ingressPort is the ingress host and ingress port of Node B, not Node A. And Node B's node graph requires Node A's ingressHost and ingressPort.

And while the Node A's ingressHost would match Node A's egressHost, Node A's ingressPort would not be the same as Node A's egressPort. For example upon starting an agent, we would see this:

egressHost	0.0.0.0
egressPort	45923
ingressHost	0.0.0.0
ingressPort	47415

And you can see that the ports are not the same.

Note that using getConnectionInfoByProxy would be the right idea to solve #342, but not sufficient for this problem.

There may be a possibility that Node A's ingressPort and egressPort could be the same port. This is allowable in-code, and would simplify the problem and allow the usage of getConnectionInfoByProxy to solve this, but there may be a conflict between response packets and new request packets in UTP. So we just need to test if this is possible.

If the 2 ports need to be different, then the solution is to add Node A's ingressHost and ingressPort information into every agent to agent communication, and have a server-side interceptor (or alternative to) to always update the node graph with the client node's ID and ingressHost and ingressPort.

[joshuakarp]

From discussions today, two main things that need to be addressed:

1. Adding a node to another node's NodeGraph

Previously we were just thinking about how we can add a newly created node onto the seed node's NodeGraph. I was thinking that we'd need to piggyback off some kind of connection logic (like hole-punching between agents), but we can instead approach this from the gRPC level. Instead of just targeting the seed node too, this will be generalised to all keynodes. That is, when a node A connects to some other node B and asks them for their closest known nodes to some other node, node B will automatically add node A to its NodeGraph. This will allow us to propagate knowledge of A throughout the Polykey network.

In order to support this process:

1. Embed a client node's ingress host and ingress port (i.e. outward facing host/port of the ReverseProxy) into the metadata for all gRPC calls
2. At the agent service handler's nodesClosestLocalNodesGet function, we:
   1. Retrieve the node ID of the caller from the call's metadata (through the certificate)
   2. Retrieve the caller's ingress host and ingress port from the call's metadata
3. The receiving server node adds the node ID -> host + port mapping into its own NodeGraph in nodesClosestLocalNodesGet

Typically you'd just be able to get the host and port from any network request. However, because the request originates from the client's ForwardProxy, we'd instead be getting its host and port (and not the ReverseProxy ingress host and port - the outward facing host and port for receiving requests, which is what we store in the NodeGraph)

By embedding the host and port in the metadata of all calls, this also provides opportunity for additional service functions to utilise this information as required (e.g. performing NodeGraph insertions/updates too).

2. "Refreshing" of buckets

Discussed in #345 now.

[CMCDragonkai]

If the ports can be the same, it would be simplest solution, so to try this. I need to go into PolykeyAgent and reverse the start up of ForwardProxy and ReverseProxy:

      await this.fwdProxy.start({
        proxyHost: networkConfig_.proxyHost,
        proxyPort: networkConfig_.proxyPort,
        egressHost: networkConfig_.egressHost,
        egressPort: networkConfig_.egressPort,
        tlsConfig,
      });
      await this.revProxy.start({
        serverHost: this.grpcServerAgent.host,
        serverPort: this.grpcServerAgent.port,
        ingressHost: networkConfig_.ingressHost,
        ingressPort: networkConfig_.ingressPort,
        tlsConfig,
      });

Into this:

      await this.revProxy.start({
        serverHost: this.grpcServerAgent.host,
        serverPort: this.grpcServerAgent.port,
        ingressHost: networkConfig_.ingressHost,
        ingressPort: networkConfig_.ingressPort,
        tlsConfig,
      });
      await this.fwdProxy.start({
        proxyHost: networkConfig_.proxyHost,
        proxyPort: networkConfig_.proxyPort,
        egressHost: this.revProxy.getIngressHost(),
        egressPort: this.revProxy.getIngressPort(),
        tlsConfig,
      });

---

Just tried, it doesn't work. If you try to use the same port for both ingress and egress, you get an ELIFECYCLE error, in that the address is already in use.

Reminder: Add this possible exception to the possible network/errors to make it pretty.

[CMCDragonkai]

Ok so this means that we need to have ingressHost and ingressPort in GRPC metadata for agent to agent calls.

This wouldn't even be solved with having the reverse proxy be capable of inspecting GRPC frames. So I guess GRPC metadata is the only foolproof solution here.

We have to deal with when the metadata is not there, and then fail the request. We can add a client side interceptor on our GRPCClientAgent that always adds this information when sending requests.

Furthermore, the data received on the server side may be incorrect or spoofed. However this is the same problem as DHT poisoning #226 and #150, since any node can store invalid DHT records and present them. Possibly the same problem as relayed requests. Basically no guarantee that the node graph entries are correct. I suppose that's where the node bucket refreshing system can help.

[joshuakarp]

Also another reasonable resource for understanding Kademlia process https://stackoverflow.com/questions/19329682/adding-new-nodes-to-kademlia-building-kademlia-routing-tables

[CMCDragonkai]

This will need ETAed, as it is the main blocker for #326.

[CMCDragonkai]

Refactoring the NodeGraph right now, I noticed that there's no setting of the node id during the creation or starting. This means NodeGraph should work with any size of node id. However the expectation is that upon creating the NodeGraph, and subsequent usage of node id would always be the same size. I think this constraint can either be made more explicit by having a key size constraint set up during creation OR it can be made flexible.

Any flexibility could only happen by auto-expanding the node id bit size when the node ids get larger. But if the node ids get smaller than the buckets would need to be re-laid out.

Therefore it makes more sense to have the key size specified ahead of time in the node graph creation, and later have a migration process if the key size were to change.

[CMCDragonkai]

This issue can be expanded to generally deal with any policies to add new nodes into the NodeGraph compared to #266 which is about dealing generally with removing nodes from the NodeGraph.

[CMCDragonkai]

Suppose N1 connects to N2. How does N2 add N1 to it's NodeGraph?

Currently we do not have access to server side interceptors. This is why we created all those utility functions in src/agent/utils.ts and src/client/utils.ts.

And even if we were to create the utility function, it would only work when a GRPC call is made, not just when a connection is established.

So if we want to do this as soon as a connection has been established, then the only 2 places that executes code here would be:

1. ReverseProxy
2. GRPCServer

If you are going through the proxy, the verification of the client certificate is on the ReverseProxy's ConnectionReverse, if you are not going through the proxy, the verification of the client certificate is on GRPCServer.

So there are 2 ways we can implement this:

1. Use a callback like we do in KeyManager and directly call NodeManager to set the node.
2. Use a callback like we do in KeyManager and use the events system to publish an event about this new connection, that PolykeyAgent can register and handle by calling the NodeManager

Since we already have the events system available, we might as well use it.

Also NodeConnection should be considered an abstraction that composes GRPCClient and ForwardProxy together.

[CMCDragonkai]

N1 would already N2's information by the time it contacts through the ForwardProxy. But if the connection establishment fails, then N1 would need to remove N2's record.

However the failure could occur in different ways:

1. NodeConnection failed to connect - see Expunge any added "invalid" nodes from k-buckets #226 (comment)
2. If you directly ping using the ForwardProxy using only just openConnection

If ForwardProxy could establish connections by itself, it may require having a callback there as well. So I wonder if this needs to be done on both, or just NodeConnection is sufficient. I think this depends on how the ping method would work as part of NodeConnectionManager and NodeManager.

But because all of this would happen in the nodes domain, it seems like any callback to the ForwardProxy would be unnecessary.

[tegefaulkes]

I'm trying to spec out the solution for this. I've run into some problems I'm still trying to solve.

So the core of the problem is that we need to authenticate an incoming connection to check if it is a valid node and obtain the NodeId and NodeAddress information. At first blush it seems simple since the ReverseProxy already checks the certificate of the MTLS connection so we can obtain the address and nodeId from that right? Well the NodeId will be correct but the ReverseProxy only gets the senders ForwardProxy's address information. This is problematic since we need the sender's ReverseProxy's address information if we want to open a connection back to it. The IP information of the ReverseProxy and ForwardProxy should be the same so there is no problem there. so we only care about obtaining the sender's ReverseProxy's port information.

The problem is. we can't know this until a connection is started. a port for the reverse proxy doesn't exist until a connection is started by the reverse proxy. Only then is the port for it created on the NAT. The Reverse proxy doesn't know what this port is either, only something receiving this connection can know what that port is.

So it seems getting the port information and storing it in the NodeGraph is next to useless. UNLESS we have the case of doing a direct connection. However we will be doing NAT traversal in most cases.

SO it seems that for most cases we only need to store the NodeId and IP of the incoming connection. Both can be gotten from the ForwardProxy information when a connection is established. as for trying to start a new connection we will need to make use of the NAT traversal system to coordinate the new connection and port information.

In the case where we can establish a direct connection without using the NAT busting. I think that's the case where the receiving node has a public facing IP and port. We will need the sender's ReverseProxy port information to start that connection. So were at the original problem, how do we get that information? It would be much simpler if the A->B and A<-B connection shared the port between the proxies. We could ask a 3rd party like a seed node to tell us what our connection info is and then include that in the message metadata.

There is something bothering me with how the NodeConnection tries to initiate the Nat busting. we need to have a discussion about that.

[tegefaulkes]

with #360 being merged into master we should be able to continue this now.

[CMCDragonkai]

I'm going to assume this is done. @tegefaulkes please check the last comment #344 (comment) and reopen if not.