Unify /etc/hosts file handling across CL,FRR and Linux VMs and containers

docs/labs/linux.md

@@ -13,6 +13,43 @@ To use any other Linux distribution or container, or to start a home-built Vagra
 
 [^GL]: You can also set the **defaults.devices.linux._provider_.image** attribute to change the Vagrant box or Docker container for all Linux hosts in your lab.
 
+(linux-hosts)=
+## Hosts File
+
+As the typical lab topology does not include DNS, _netlab_ adds entries for all lab devices to the Linux `/etc/hosts` file. The initial configuration templates add entries mapping all non-VRF IPv4 and IPv6 addresses to node names and entries mapping VRF IPv4 and IPv6 addresses to the *vrf*.*node* name.
+
+*netlab* always creates entries for individual device interfaces in the `/etc/hosts` file; otherwise, the name resolution picks a random device IP address instead of the loopback IP address when doing **ping** or **traceroute**. For example, these entries would be generated for a router with two dual-stack interfaces:
+
+```
+10.0.0.1 r
+2001:db8:0:1::1 r
+172.16.0.1 eth1.r
+2001:db8:1::1 eth1.r
+172.16.1.1 eth2.r
+2001:db8:1:1::1 eth2.r
+```
+
+Similar entries are generated for hosts (devices without loopback interfaces):
+
+```
+172.16.0.2 h1 eth1.h1
+2001:db8:1::2 h1 eth1.h1
+```
+
+On a VRF-enabled router, you might get the following entries (the router has only VRF interfaces; 10.0.0.42 and 10.0.0.43 are VRF loopback addresses):
+
+```
+10.0.0.5 dut
+172.16.0.5 eth1.red.dut
+172.16.1.5 eth2.red.dut
+172.16.2.5 eth3.blue.dut
+172.16.3.5 eth4.blue.dut
+10.0.0.42 red.dut
+10.0.0.43 blue.dut
+```
+
+The netlab-generated entries are *appended* to the existing `/etc/hosts` file on virtual machines. The container `/etc/hosts` file is generated from scratch to remove the management IP addresses *containerlab* inserted into the `/etc/hosts` file.
+
 (linux-routes)=
 ## Host Routing
 
@@ -107,7 +144,7 @@ _netlab_ initial configuration script will skip Ubuntu package installation if i
 The initial configuration process (**[netlab initial](../netlab/initial.md)**) does not rely on commands executed within Linux containers:
 
 * The `/etc/hosts` file is generated during the **[netlab create](../netlab/create.md)** process from the ```templates/provider/clab/frr/hosts.j2``` template (see [](clab-config-template)).
-* Interface IP addresses and static routes to default gateway (see [](linux-routes)) are configured with **ip** commands executed on the Linux host but within the container network namespace.
+* Interface IP addresses and static routes to the default gateway (see [](linux-routes)) are configured with **ip** commands executed on the Linux host but within the container network namespace.
 * Static default route points to the management interface.
 
-You can therefore use any container image as a Linux node.
+You can, therefore, use any container image as a Linux node.

docs/platforms.md

@@ -164,20 +164,20 @@ Ansible playbooks included with **netlab** can deploy and collect device configu
 The following system-wide features are configured on supported network operating systems as part of the initial device configuration:
 
 (platform-initial-config)=
-| Operating system      | Hostname | IPv4 hosts |           LLDP            | Loopback<br />IPv4 address | Loopback<br />IPv6 address |
+| Operating system      | Hostname | IPv4/IPv6<br>hosts |           LLDP            | IPv4<br>Loopback | IPv6<br>Loopback |
 | --------------------- | :------: | :--------: | :-----------------------: | :------------------------: | :------------------------: |
 | Arista EOS               | ✅  | ✅  | ✅  | ✅  | ✅  |
 | Aruba AOS-CX             | ✅  |  ❌  | ✅  | ✅  | ✅  |
 | Cisco ASAv               | ✅  | ✅  |  ❌  |  ❌  |  ❌  |
 | Cisco IOS/IOS XE[^18v]   | ✅  | ✅  | ✅  | ✅  | ✅  |
 | Cisco IOS XRv            | ✅  | ✅  | ✅  | ✅  | ✅  |
 | Cisco Nexus OS           | ✅  | ✅  | ✅  | ✅  | ✅  |
-| Cumulus Linux            | ✅  | ✅  | ✅  | ✅  | ✅  |
+| Cumulus Linux            | ✅  | ✅ [^HIF]  | ✅  | ✅  | ✅  |
 | Cumulus Linux 5.0 (NVUE) | ✅  | ✅  | ✅  | ✅  | ✅  |
 | Dell OS10                | ✅  | ✅  | ✅  | ✅  | ✅  |
 | Fortinet FortiOS         | ✅  |  ❌  | ✅  | ✅  | ✅  |
-| FRR                      | ✅  | ✅  |  ❌  | ✅  | ✅  |
-| Generic Linux            | ✅  | ✅  |  ✅[❗](linux-lldp) | ✅  | ✅  |
+| FRR                      | ✅  | ✅ [^HIF]  |  ❌  | ✅  | ✅  |
+| Generic Linux            | ✅  | ✅ [^HIF]  |  ✅[❗](linux-lldp) | ✅  | ✅  |
 | Junos[^Junos]            | ✅  |  ❌  | ✅  | ✅  | ✅  | 
 | Mikrotik RouterOS 6      | ✅  | ✅  | ✅[❗](caveats-routeros6) | ✅ | ✅ |
 | Mikrotik RouterOS 7      | ✅ | ✅ | ✅[❗](caveats-routeros7) | ✅ | ✅ |
@@ -186,6 +186,8 @@ The following system-wide features are configured on supported network operating
 | Sonic                    | ✅  | ✅  |  ❌  | ✅  | ✅  |
 | VyOS                     | ✅  | ✅  | ✅  | ✅  | ✅  |
 
+[^HIF]: Some Linux-based devices can also use interface names in host names. See [/etc/hosts file on Linux](linux-hosts) for more details.
+
 (platform-initial-interfaces)=
 The following interface parameters are configured on supported network operating systems as part of the initial device configuration:
 

netsim/ansible/templates/initial/cumulus.j2

@@ -17,21 +17,10 @@ SCRIPT
 echo "INIT: setting hostname"
 {% set v_hostname = inventory_hostname.replace("_","-") %}
 hostname {{ v_hostname }}
-echo "INIT: creating /etc/hosts"
-awk '/127.0.1.1/,/^$/' /etc/hosts >/tmp/hosts
-echo "127.0.0.1 {{ inventory_hostname }}{% 
-  if "_" in inventory_hostname %} {{ v_hostname }}{% endif %}" >>/tmp/hosts
-{% for hname,hdata in hosts.items() if hname != inventory_hostname %}
-echo "{{ (hdata.ipv4|default([]) + hdata.ipv6|default([]))|join (' ') }} {{ hname }}" >>/tmp/hosts
-{% endfor %}
-cat /etc/hosts | awk '/localhost/,/^$/' >/tmp/hosts-start
-{% if ansible_connection != "docker" %}
-mv /tmp/hosts-start /etc/hosts
-{% endif %}
-sort /tmp/hosts|uniq >>/etc/hosts
 #
 {% if clab.kind is not defined %}
-{# no need to disable ZTP in a container #}
+{% include 'linux/hosts.j2' +%}
+{# no need to create hosts file or disable ZTP in a container #}
 #
 # Disable ZTP
 #

netsim/ansible/templates/initial/frr.j2

@@ -20,17 +20,7 @@ export PS1="\h(bash)#"
 echo "Use sudo vtysh to connect to FRR daemon"
 echo
 SCRIPT
-{% endif %}
-#
-# Build hosts file
-#
-echo "# Created by netlab initial" >>/tmp/hosts
-{% for hname,hdata in hosts.items() %}
-echo "{{ (hdata.ipv4|default([]) + hdata.ipv6|default([]))|join (' ') }} {{ hname }}" >>/tmp/hosts
-{% endfor %}
-grep "Created by netlab" /etc/hosts || uniq /tmp/hosts|sort >>/etc/hosts
-
-{% if clab is not defined %}
+{% include 'linux/hosts.j2' +%}
 #
 # Configure system defaults on Ubuntu
 #

netsim/ansible/templates/initial/linux.j2

@@ -13,6 +13,7 @@ cat <<SCRIPT >/root/.bash_profile
 #
 export PS1="\h(bash)$ "
 SCRIPT
+hostname {{ inventory_hostname.replace("_","-") }}
 {% include 'linux/hosts.j2' +%}
 {% if netlab_linux_distro is defined %}
 {%   include 'linux/'+netlab_linux_distro+'.j2' +%}

netsim/ansible/templates/initial/linux/hosts-common.j2

@@ -0,0 +1,22 @@
+{% for hname,hdata in hostvars.items() %}
+{%   set intf_list = hdata.interfaces|default([]) %}
+{%   if hdata.loopback is defined %}
+{%     set intf_list = [ hdata.loopback ] + intf_list %}
+{%   endif %}
+{%   for intf in intf_list %}
+{%     set h_entry = hname %}
+{%     if intf.vrf is defined %}
+{%       set h_entry = intf.vrf + '.' + h_entry %}
+{%     endif %}
+{%     if intf.type != 'loopback' %}
+{%       set h_entry = intf.ifname.replace('/','-') + '.' + h_entry %}
+{%     endif %}
+{%     set need_host = hdata.loopback is not defined %}
+{%     if intf.ipv4|default(False) is string %}
+{{ intf.ipv4|ipaddr('address') }} {% if need_host %}{{ hname }} {% endif +%}{{ h_entry }}
+{%     endif %}
+{%     if intf.ipv6|default(False) is string %}
+{{ intf.ipv6|ipaddr('address') }} {% if need_host %}{{ hname }} {% endif +%}{{ h_entry }}
+{%     endif %}
+{%   endfor %}
+{% endfor %}

netsim/ansible/templates/initial/linux/hosts.j2

@@ -1,7 +1,6 @@
 #
 # Build hosts file
 #
-hostname {{ inventory_hostname.replace("_","-") }}
 {% if clab.kind is defined %}
 {# ... running in containerlab, have to rewrite /etc/hosts from scratch #}
 cat <<HOSTS >/etc/hosts
@@ -21,8 +20,6 @@ cat <<SCRIPT >/tmp/hosts
 #
 # Created by netlab initial
 #
-{% for hname,hdata in hosts.items() %}
-{{ (hdata.ipv4|default([]) + hdata.ipv6|default([]))|join (' ') }} {{ hname }}
-{% endfor %}
+{% include 'linux/hosts-common.j2' %}
 SCRIPT
 grep "Created by netlab" /etc/hosts || uniq /tmp/hosts >>/etc/hosts

netsim/ansible/templates/initial/linux/ubuntu.j2

@@ -45,6 +45,15 @@ else
   apt-get install -qq net-tools
   NEED_APT_UPDATE=
 fi
+if which traceroute; then
+  echo "traceroute already installed"
+else
+  if "$NEED_APT_UPDATE"; then
+    apt-get update -qq
+  fi
+  apt-get install -qq traceroute
+  NEED_APT_UPDATE=
+fi
 {% endif %}
 {% if netlab_lldp_enable|default(False) %}
 #

netsim/devices/cumulus.yml

@@ -10,6 +10,21 @@ libvirt:
 virtualbox:
   image: CumulusCommunity/cumulus-vx:4.3.0
   mtu: 1500
+clab:
+  mtu: 1500
+  kmods:
+  node:
+    kind: cvx
+    runtime: docker
+    config_templates:
+      interfaces: /etc/network/interfaces
+      hosts: /etc/hosts
+  image: networkop/cx:4.4.0
+  group_vars:
+    ansible_connection: docker
+    ansible_user: root
+external:
+  image: none
 group_vars:
   ansible_user: vagrant
   ansible_ssh_pass: vagrant
@@ -85,18 +100,4 @@ features:
     ripng: True
   vxlan: True
 
-clab:
-  mtu: 1500
-  kmods:
-  node:
-    kind: cvx
-    runtime: docker
-    config_templates:
-      interfaces: /etc/network/interfaces
-  image: networkop/cx:4.4.0
-  group_vars:
-    ansible_connection: docker
-    ansible_user: root
-external:
-  image: none
 graphite.icon: switch

netsim/outputs/ansible.py

@@ -14,7 +14,7 @@
 from ..augment import plugin
 from ..utils import templates,strings,log
 from ..utils import files as _files
-from ..data import global_vars
+from ..data import global_vars,append_to_list
 
 forwarded_port_name = { 'ssh': 'ansible_port', }
 
@@ -80,18 +80,19 @@ def get_host_addresses(topology: Box) -> Box:
     intf_list = node.interfaces
     if 'loopback' in node:                                  # Create a list of all usable interfaces
       intf_list = [ node.loopback ] + node.interfaces       # ... starting with loopback
+      for af in ['ipv4','ipv6']:
+        if af in node.loopback:
+          lb = str(netaddr.IPNetwork(node.loopback[af]).ip) # Extract IP address from the CIDR prefix
+          append_to_list(hosts[name],'loopback',lb)
 
     for intf in intf_list:                                  # Now iterate over interfaces
       h_name = f'{name}-{intf.vrf}' if 'vrf' in intf else name
       for af in ('ipv4','ipv6'):                            # ... and collect IPv4 and IPv6 addresses
         if not isinstance(intf.get(af,False),str):          # Is the IP address a string (= usable IP address)?
           continue
 
-        if not af in hosts[h_name]:                         # Edge case: first interface
-          hosts[h_name][af] = []                            # ... have to start with an empty list
-
         addr = str(netaddr.IPNetwork(intf[af]).ip)          # Extract IP address from the CIDR prefix
-        hosts[h_name][af].append(addr)                      # ... and append it to the list of usable IP addresses
+        append_to_list(hosts[h_name],af,addr)               # ... and append it to the list of usable IP addresses
 
   global_vars.set('hosts',hosts)                            # Cache the hosts dictionary
   return hosts

netsim/templates/provider/clab/cumulus/hosts-common.j2

@@ -0,0 +1 @@
+../../../../ansible/templates/initial/linux/hosts-common.j2

netsim/templates/provider/clab/frr/hosts-common.j2

@@ -0,0 +1 @@
+../../../../ansible/templates/initial/linux/hosts-common.j2

netsim/templates/provider/clab/linux/hosts-common.j2

@@ -0,0 +1 @@
+../../../../ansible/templates/initial/linux/hosts-common.j2

netsim/templates/provider/clab/linux/hosts.j2

@@ -8,6 +8,4 @@ ff00::0	ip6-mcastprefix
 ff02::1	ip6-allnodes
 ff02::2	ip6-allrouters
 #
-{% for hname,hdata in hosts.items() %}
-{{ (hdata.ipv4|default([]) + hdata.ipv6|default([]))|join (' ') }} {{ hname }}
-{% endfor %}
+{% include 'hosts-common.j2' %}

tests/topology/expected/bgp-unnumbered.yml

@@ -104,8 +104,10 @@ nodes:
     clab:
       binds:
       - clab_files/r1/interfaces:/etc/network/interfaces
+      - clab_files/r1/hosts:/etc/hosts
       config_templates:
       - interfaces:/etc/network/interfaces
+      - hosts:/etc/hosts
       kind: cvx
       runtime: docker
     device: cumulus
@@ -190,8 +192,10 @@ nodes:
     clab:
       binds:
       - clab_files/r2/interfaces:/etc/network/interfaces
+      - clab_files/r2/hosts:/etc/hosts
       config_templates:
       - interfaces:/etc/network/interfaces
+      - hosts:/etc/hosts
       kind: cvx
       runtime: docker
     device: cumulus
@@ -277,8 +281,10 @@ nodes:
     clab:
       binds:
       - clab_files/r3/interfaces:/etc/network/interfaces
+      - clab_files/r3/hosts:/etc/hosts
       config_templates:
       - interfaces:/etc/network/interfaces
+      - hosts:/etc/hosts
       kind: cvx
       runtime: ignite
     device: cumulus

tests/topology/expected/device-node-defaults.yml

@@ -121,8 +121,10 @@ nodes:
     clab:
       binds:
       - clab_files/rtr/interfaces:/etc/network/interfaces
+      - clab_files/rtr/hosts:/etc/hosts
       config_templates:
       - interfaces:/etc/network/interfaces
+      - hosts:/etc/hosts
       kind: cvx
       runtime: docker
     device: cumulus