NAT-Traversal Testing with testnet.polykey.io #159
Comments
|
This should be incorporated into our automated tests when we run jest. But that would also mean using |
|
We need to implement a test for relaying the hole punching message. This is not meant to be using the notifications domain because it's part of automated connection establishment. We need to test several situations:
|
CMCDragonkai
changed the title
|
@joshuakarp I'm curious how exactly are we going to implement a optimised routing system for routing the hole punching relay messages? The same algorithm can later be used for picking the optimal relay for mesh proxying to defeat symmetric NAT. I remember we mentioned some usage of kademlia or represenation of "closest" node. If we just assume that we always use our seed cluster/bootstrap cluster, then this is just centralised routing. But if we enable any keynode to be a relay, then we need to understand that the the PK network is a loose mesh, with loose connection lifetimes as well. Is kademlia actually useful for routing here? This feels like a routing problem, and it seems that existing routers already have algorithms that help solve this problem. Is there any cross over with things like spanning tree algorithms https://en.wikipedia.org/wiki/Minimum_spanning_tree? Given that we all keynodes may be on the public internet. Another matter is whether all live network links are equal in quality. Of course in reality they are not where latency and throughput and reliability matters. But if we are only distinguishing between vertexes where edges can be made vs vertexes where edges cannot be made, then our algorithm should converge very quickly to find the proper relaying route. |
|
@CMCDragonkai Kademlia inherently has a "closeness" mechanism. That is, the XOR value of two node IDs determine closeness (smaller = closer, larger = further away). Remember that with Kademlia, we store more node ID -> node address mappings of the nodes that are "closest" to us: this is the fundamental part of the k-buckets structure. Isn't this inherently a routing solution? See this too, straight from the Kademlia paper https://www.scs.stanford.edu/~dm/home/papers/kpos.pdf:
I found a pretty good animation of this too, to showcase the lookup procedure https://kelseyc18.github.io/kademlia_vis/lookup/ As a side note, I started to read quite an interesting paper about using notions of "trust" to overcome some of the issues with malicious nodes and attack vectors on these kinds of systems: https://ieeexplore.ieee.org/document/6217954 |
|
Kademlia's closeness is used to route to the relevant node that has information on the node ID to IP address. I can see how that might mean that you can trigger a hole punch relay message at that node. Does this mean you would need to send that as a option/flag that means you want to pass on a hole punching message on the call to resolve a node ID? This would mean resolution and relaying a hole punch is done at the same time. Or you would need to know which node returned the resolution and then use that. However there's still a problem with this mechanism. The relaying node must already have an open connection with the receiving node. If the relaying node does not have an open and live connection and that the receiving node is behind a restricted NAT, then the relaying cannot actually relay anything just like the sending node. There is an assumption here that the node that resolves has an open connection to all the IP addresses. But is this actually true? There are several points here:
|
|
Is the kademlia contact database rebalanced/replicated across the network like a DHT? Otherwise how does one store a contact if not by being contacted by it and contacting it in turn? |
In order for Kademlia to function, there are lots of implicit connection establishments taking place. That is, every time you receive So yes, as part of the resolution process, we are already sending hole punch packets to each of these nodes we need to contact.
This could be a worthwhile optimisation.
Yeah, you're right. I remember we had some brief discussion about whether we should consider having "persistent" connections to some of the "closest" nodes in the network. That is, upon coming online, we immediately connect to these nodes. But yeah, in order to even establish these persistent connections, we have the same issue. |
Currently no. There's no rebalancing/replication across the network. There's currently 2 ways that nodes are added to the database:
|
|
I think our plan is for the release, we'll stick with the centralised seed node cluster #194. We can put the problem of decentralised relaying to a post-release issue. This issue is more focused on just creating a test-harness for NAT-traversal, so we should focus this issue on this problem. In the mean time, I'll create a new issue for decentralised relaying. |
|
If you have difficulties working on this, I can ask @nzhang-zh or @Zachaccino to help advise. |
|
Our tests here should probably change to be manual as soon as #194 is done and then figure out how to automate these tests. |
CMCDragonkai
changed the title
|
Start date changed from Nov 15th to Nov 19th (based on delays in #231). |
|
Start date changed from Tuesday Nov 23rd to Monday Dec 6th (delayed from refactoring work in #283). |
|
These tests must be written outside or separately from
It's best to continue using our jest tooling for these tests, but if we need to use OS simulation, then the jest tests may need to be executing shell commands which then encapsulate scripts that run inside a network namespaces. |
|
This issue requires more deeper specifications, that work out all the different cases being tested. It's going to depend on the resolution of #326 as that will finish the testnet deployment. These test cases may use the testnet.polykey.io. |
|
Some ideas for initial cases.. These cases do not have a signalling server. I.e. no seed node involved in coordination.
For the NAT, we need to simulate the 4 types:
I'm not sure if our Linux netns and firewall can simulate all 4, but it should be able to do at the very least port restricted. These cases do have a signalling server:
The signalling server, is enabled by having both node1 and node2 already connected to the seed node. That seed node should then be relaying connection request messages. That should be enough for now. No TURN relay testing yet. Note that some tests are expected to "fail", in that we want to test what the expected exceptional behaviour handling is. Like when the nodes cannot connect, how do we communicate this to the end user. |
|
In order to create these network namespaces, you have to use both This also means when we actually do the tests, the tests will be done with |
|
After doing some prototyping today, I'm now able to setup two nodes bethind two routers (four network namespaces) and have node 1 and node 2 be able to ping each other. The next step will be adding iptables rules to the routers to simulate nat, but for now this is how I'm setting everything up before that point: After running all of these commands, we should be able to ping |
|
To be able to do Nat simulation beyond full cone, you need a stateful firewall. In iptables this is known as conntrack. Have a look at conntrack and stateful iptables. |
|
I believe that with node 1 and router 1 they can all share the same namespace. This is because the network namespace creates its own private network, and both Node 1 and Router 1 are on the same private network. However it may be better for you to test with 4 namespaces first and then see how you can optimise just down to 2. |
|
If your commands will require sudo permissions, then you can run the jest test script as sudo. For example Do note that any files created will be in root ownership so it's important that any temporary files created are deleted. |
|
Also any command using |
From my research I think these iptables rules should replicate a stateful firewall:
I remember having a quick look a conntrack and it didn't seem like the right thing to use, but I can have another look at it. |
Hmm yeah that might work. I'll keep prototyping with four for now but that could be something to look into later. I found this which might be useful for setting up namespaces that contain multiple hosts with a router: https://github.com/mininet/mininet |
|
I'm in the process of testing iptables rules to see if the NAT is working correctly, however I'm finding it hard to test for this. I was wanting to use wireshark but I can't open it from inside a namespace. I tried using |
|
You can change the net namespace of a program using But I'm not sure how well it will work with wireshark. Alternatively you can use tcpdump or netcat for simple testing. |
|
This import { exec } from "child_process";
async function main() {
// Namespaces
const netnsn1 = 'node1';
const netnsn2 = 'node2';
const netnsr1 = 'router1';
const netnsr2 = 'router2';
// Veth cables (ends)
const n1ToR1 = 'veth1-n1';
const r1ToN1 = 'veth1-r1';
const r2ToN2 = 'veth2-r2';
const n2ToR2 = 'veth2-n2';
const r1ToR2 = 'veth3-r1';
const r2ToR1 = 'veth3-r2';
// Subnets
const n1ToR1Subnet = '1.1.1.1';
const r1ToN1Subnet = '1.1.1.2';
const r2ToN2Subnet = '2.2.2.1';
const n2ToR2Subnet = '2.2.2.2';
const r1ToR2Subnet = '3.3.3.1';
const r2ToR1Subnet = '3.3.3.2';
// Subnet mask
const subnetMask = '/24';
// Logger for exec commands
const logger = (error, stdout, stderr) => {
if (error) {
console.log(`error: ${error.message}`);
return;
}
if (stderr) {
console.log(`stderr: ${stderr}`);
return;
}
console.log(`stdout: ${stdout}`);
}
// Create network namespaces for two nodes with NAT routers
exec(`ip netns add ${netnsn1}`, logger);
exec(`ip netns add ${netnsn2}`, logger);
exec(`ip netns add ${netnsr1}`, logger);
exec(`ip netns add ${netnsr2}`, logger);
// Create veth pairs to link the namespaces
exec(`ip link add ${n1ToR1} type veth peer name ${r1ToN1}`, logger);
exec(`ip link add ${r2ToN2} type veth peer name ${n2ToR2}`, logger);
exec(`ip link add ${r1ToR2} type veth peer name ${r2ToR1}`, logger);
// Link up the veth pairs to the correct namespaces
exec(`ip link set ${n1ToR1} netns ${netnsn1}`, logger);
exec(`ip link set ${n2ToR2} netns ${netnsn2}`, logger);
exec(`ip link set ${r1ToN1} netns ${netnsr1}`, logger);
exec(`ip link set ${r1ToR2} netns ${netnsr1}`, logger);
exec(`ip link set ${r2ToN2} netns ${netnsr2}`, logger);
exec(`ip link set ${r2ToR1} netns ${netnsr2}`, logger);
// Loopback and veths are down by default - get them running
exec(`ip netns exec ${netnsn1} ip link set lo up`, logger);
exec(`ip netns exec ${netnsn1} ip link set ${n1ToR1} up`, logger);
exec(`ip netns exec ${netnsn2} ip link set lo up`, logger);
exec(`ip netns exec ${netnsn2} ip link set ${n2ToR2} up`, logger);
exec(`ip netns exec ${netnsr1} ip link set lo up`, logger);
exec(`ip netns exec ${netnsr1} ip link set ${r1ToN1} up`, logger);
exec(`ip netns exec ${netnsr1} ip link set ${r1ToR2} up`, logger);
exec(`ip netns exec ${netnsr2} ip link set lo up`, logger);
exec(`ip netns exec ${netnsr2} ip link set ${r2ToN2} up`, logger);
exec(`ip netns exec ${netnsr2} ip link set ${r2ToR1} up`, logger);
// Create subnets for the veth pairs to communicate over
exec(`ip netns exec ${netnsn1} ip addr add ${n1ToR1Subnet}${subnetMask} dev ${n1ToR1}`, logger);
exec(`ip netns exec ${netnsn2} ip addr add ${n2ToR2Subnet}${subnetMask} dev ${n2ToR2}`, logger);
exec(`ip netns exec ${netnsr1} ip addr add ${r1ToN1Subnet}${subnetMask} dev ${r1ToN1}`, logger);
exec(`ip netns exec ${netnsr1} ip addr add ${r1ToR2Subnet}${subnetMask} dev ${r1ToR2}`, logger);
exec(`ip netns exec ${netnsr2} ip addr add ${r2ToN2Subnet}${subnetMask} dev ${r2ToN2}`, logger);
exec(`ip netns exec ${netnsr2} ip addr add ${r2ToR1Subnet}${subnetMask} dev ${r2ToR1}`, logger);
// Setup the defalt routes for each namespace
exec(`ip netns exec ${netnsn1} ip route add default via ${r1ToN1Subnet}`, logger);
exec(`ip netns exec ${netnsn2} ip route add default via ${r2ToN2Subnet}`, logger);
exec(`ip netns exec ${netnsr1} ip route add default via ${r2ToR1Subnet}`, logger);
exec(`ip netns exec ${netnsr2} ip route add default via ${r1ToR2Subnet}`, logger);
// Check that everything was setup correctly
// Interfaces are up at the correct addresses
exec(`ip netns exec ${netnsn1} ip addr`, logger);
exec(`ip netns exec ${netnsn2} ip addr`, logger);
exec(`ip netns exec ${netnsr1} ip addr`, logger);
exec(`ip netns exec ${netnsr2} ip addr`, logger);
// Routing tables are correct
exec(`ip netns exec ${netnsn1} ip route`, logger);
exec(`ip netns exec ${netnsn2} ip route`, logger);
exec(`ip netns exec ${netnsr1} ip route`, logger);
exec(`ip netns exec ${netnsr2} ip route`, logger);
// Can ping from one node to the other
exec(`ip netns exec ${netnsn1} ping -c 3 ${n2ToR2Subnet}`, logger);
exec(`ip netns exec ${netnsn2} ping -c 3 ${n1ToR1Subnet}`, logger);
// Delete the namespaces
exec(`ip netns del ${netnsn1}`, logger);
exec(`ip netns del ${netnsn2}`, logger);
exec(`ip netns del ${netnsr1}`, logger);
exec(`ip netns del ${netnsr2}`, logger);
}
main(); |
|
I've got the NAT rules working!! For testing this I created this setup of namespaces linked with my real system (since I can only open wireshark from my real system):
I wanted I then added the following iptables rules to the router: This simulates the endpoint-independent NAT mapping used by full-cone, restricted-cone, and port-restricted-cone NAT. Note that specifying the interface that the packet is arriving on for the Even though in our rule we specified the target address to be matched as With all of this setup done, we can now send packets to and from the client and root. From the client's perspective it is communicating directly with root: it's sending packets addressed to
|
|
Some relevant discussions in the PR #357 (comment) about Also for our test cases, I suggest this matrix can help:
It comes from https://dh2i.com/kbs/kbs-2961448-understanding-different-nat-types-and-hole-punching/ The non-routable ones should be routable with a TURN relay. |
|
@emmacasolin Can you change to address ranges instead as that should make it easier when we have more than 1 agent behind a NAT. |
|
With regards to this comment #357 (comment) we only need to simulate port-restricted cone and symmetric NAT for our tests. This is because our NAT busting will work for full cone and address-restricted cone NAT if it works for port-restricted cone and symmetric NAT, since the architectures they use are the same or less sophisticated than port-restricted cone/symmetric NAT. |
CMCDragonkai
changed the title
|
I've changed the issue name here to remove "non-Symmetric NAT" because we are infact testing with symmetric NAT now. This issue is blocked on testnet deployment #378. |
|
@emmacasolin can you tick off the tasks here if they are done. |
|
Earlier tasks are all ticked by the merging of #381 to staging. I've added task 7 to address the testing for Such a test would need to be conditional as well, but this time representing tests that run during integration. @tegefaulkes is currently working on getting our |
|
@emmacasolin you'll start on this now, and since the testnet deployment will occur on each deployment to staging, that means you'll need to trigger testnet deployment locally whenever you're fixing up anything related to the testnet. Please go through your AWS account, and test that you can interact with ECS and ECR. You'll need to use the Some initial related bugs include reviewing #402. Also rename that PR to more specific to what is being solved there. |
|
Get yourself familarized with:
Those will be important to observe as you are redeploying the seed nodes. |
CMCDragonkai
added
the
r&d:polykey:core activity 4
|
The last task is now a separate issue MatrixAI/Polykey-CLI#71, so this issue can be closed. |
Specification
To automatically test for NAT-busting is a bit complex, you need to simulate the existence of multiple machines, and then simulate full-cone nat, restricted cone nat and then symmetric nat.
Since we don't have a relay proxy enabled yet, symmetric NATs is going to be left out for now. So we'll focus on all the NAT architectures except for symmetric NAT.
Additional context
Actually I don't think we should bother with QEMU or NixOS here. It's too complicated. QEMU might be good choice for being able to run the test cross-platform, but lacking expertise on QEMU here (I've already worked on it with respect to netboot work), and more experience with network namespaces should mean we can do this tests on just Linux. NixOS limits our environment even more and requires running in a NixOS environment.
Note that network namespaces with Linux stateful firewalls should be perfectly capable of simulating a port-restricted firewall.
Old context follows...
The best way to do this is with VM system using QEMU.
NixOS has a multi-machine testing system that can be used to do this, however such tests can only run on NixOS: https://nixos.org/manual/nixos/unstable/index.html#sec-nixos-tests We have pre-existing code for this:
NixOS NAT Module Test
Tasks
4. [ ] - The test will have to be run separately fromUsing conditional testing instead Conditional testing for platform-specific tests #380npm testwhich runsjest. This test can be done inside Gitlab CI/CD if the CI/CD on Linux supports creating network namespaces. If not, it's a manual test.ip netnscommands. Remember to check whether the OS is linux before allowing one to run these tests.[ ] - Add in testing involving- Reissued Integration Tests fortestnet.polykey.iowhich should run only during integration testing after theintegration:deploymentjob (because it has to deploy to the testnet in that job).testnet.polykey.comPolykey-CLI#71The text was updated successfully, but these errors were encountered: