Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

clab config not persistent after redeploy #1685

Open
KTodts opened this issue Oct 30, 2023 · 12 comments
Open

clab config not persistent after redeploy #1685

KTodts opened this issue Oct 30, 2023 · 12 comments
Assignees
@hellt
Labels
bug Something isn't working

Comments

@KTodts
Copy link

KTodts commented Oct 30, 2023

I noticed after upgrading my clab version from 43.0 to 47.1 that the config is not persistent after a redeploy of the topology.

Test would be, deploy topology for the first time --> make a config change --> destroy lab --> deploy again --> verify if change is persistent.

1 upgrade clab to version 47.1

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# clab version upgrade
A newer containerlab 0.47.1 is available. Release notes: https://containerlab.dev/rn/0.47.1
You are running containerlab 0.43.0 version
Downloading https://github.com/srl-labs/containerlab/releases/download/v0.47.1/containerlab_0.47.1_linux_amd64.deb
Preparing to install containerlab 0.47.1 from package
(Reading database ... 115507 files and directories currently installed.)
Preparing to unpack .../containerlab_0.47.1_linux_amd64.deb ...
Unpacking containerlab (0.47.1) over (0.43.0) ...
Setting up containerlab (0.47.1) ...

                           _                   _       _
                 _        (_)                 | |     | |
 ____ ___  ____ | |_  ____ _ ____   ____  ____| | ____| | _
/ ___) _ \|  _ \|  _)/ _  | |  _ \ / _  )/ ___) |/ _  | || \
( (__| |_|| | | | |_( ( | | | | | ( (/ /| |   | ( ( | | |_) )
\____)___/|_| |_|\___)_||_|_|_| |_|\____)_|   |_|\_||_|____/

    version: 0.47.1
     commit: 1da5ebd2
       date: 2023-10-24T22:09:40Z
     source: https://github.com/srl-labs/containerlab
 rel. notes: https://containerlab.dev/rn/0.47/#0471

2 Start with a fresh deployment by using --reconfgure

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# ll
total 72
drwxr-xr-x  6 root root 4096 Oct 26 09:31 ./
drwxr-xr-x 17 root root 4096 Oct 26 09:22 ../
-rwxr-xr-x  1 root root   37 Oct 26 09:22 00.show_lab.sh*
-rwxr-xr-x  1 root root   36 Oct 26 09:22 01.deploy_srl.sh*
-rwxr-xr-x  1 root root  335 Oct 26 09:22 03.initialize_hosts.sh*
-rwxr-xr-x  1 root root   49 Oct 26 09:22 10.gnmic_example.sh*
-rwxr-xr-x  1 root root   37 Oct 26 09:22 91.destroy_srl.sh*
-rwxr-xr-x  1 root root  310 Oct 26 09:22 98.check_xdp.sh*
-rwxr-xr-x  1 root root  386 Oct 26 09:22 99.repair_xdp.sh*
drwxr-xr-x  7 root root 4096 Oct 26 09:31 clab-lab/
drwxr-xr-x 10 root root 4096 Oct 26 09:22 config/
drwxr-xr-x  8 root root 4096 Oct 26 09:22 .git/
-rw-r--r--  1 root root   10 Oct 26 09:22 .gitignore
-rw-r--r--  1 root root 1025 Oct 26 09:22 gnmic_subscription.yaml
drwxr-xr-x  2 root root 4096 Oct 26 09:22 img/
-rw-r--r--  1 root root 1192 Oct 26 09:22 README.md
-rw-r--r--  1 root root 1413 Oct 26 09:30 topology.yml
-rw-r--r--  1 root root 1413 Oct 26 09:36 .topology.yml.bak
root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# clab dep -t topology.yml --reconfigure
INFO[0000] Containerlab v0.47.1 started
INFO[0000] Parsing & checking topology file: topology.yml
INFO[0000] Removing /home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab/clab-lab directory...
INFO[0000] Creating lab directory: /home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab/clab-lab
INFO[0000] Creating container: "host11"
INFO[0000] Creating container: "host10"
INFO[0000] Creating container: "host12"
INFO[0000] Creating container: "host20"
INFO[0000] Creating container: "dc-leaf-1"
INFO[0000] Creating container: "dc-spine-2"
INFO[0000] Creating container: "dc-spine-1"
INFO[0000] Creating container: "dc-leaf-2"
INFO[0002] Creating link: dc-spine-1:e1-2 <--> dc-leaf-2:e1-1
INFO[0003] Creating link: dc-spine-2:e1-2 <--> dc-leaf-2:e1-2
INFO[0003] Creating link: dc-spine-1:e1-1 <--> dc-leaf-1:e1-1
INFO[0003] Creating link: host20:eth1 <--> dc-leaf-2:e1-10
INFO[0004] Creating link: dc-spine-2:e1-1 <--> dc-leaf-1:e1-2
INFO[0004] Creating link: host10:eth1 <--> dc-leaf-1:e1-10
INFO[0004] Creating link: host11:eth1 <--> dc-leaf-1:e1-11
INFO[0004] Creating link: host12:eth1 <--> dc-leaf-1:e1-12
INFO[0005] Running postdeploy actions for Nokia SR Linux 'dc-spine-2' node
INFO[0005] Running postdeploy actions for Nokia SR Linux 'dc-leaf-2' node
INFO[0005] Running postdeploy actions for Nokia SR Linux 'dc-leaf-1' node
INFO[0005] Running postdeploy actions for Nokia SR Linux 'dc-spine-1' node
INFO[0023] Adding containerlab host entries to /etc/hosts file
INFO[0023] Adding ssh config for containerlab nodes
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+
| # |        Name         | Container ID |             Image             | Kind  |  State  |  IPv4 Address   |     IPv6 Address     |
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-dc-leaf-1  | e3236ca45408 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 2 | clab-lab-dc-leaf-2  | be9c94e81822 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.5/24  | 2001:172:20:20::5/64 |
| 3 | clab-lab-dc-spine-1 | ca2ba3def795 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.7/24  | 2001:172:20:20::7/64 |
| 4 | clab-lab-dc-spine-2 | 930b7ef359f4 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.9/24  | 2001:172:20:20::9/64 |
| 5 | clab-lab-host10     | 7eb4169c5d66 | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.6/24  | 2001:172:20:20::6/64 |
| 6 | clab-lab-host11     | e36457b5060d | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.8/24  | 2001:172:20:20::8/64 |
| 7 | clab-lab-host12     | dd238582526d | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 8 | clab-lab-host20     | c9e74db564c3 | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+

3 Log into the node. Make a change, commit and save config. (add default network-instance with a description)

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# ssh admin@clab-lab-dc-leaf-1
Warning: Permanently added 'clab-lab-dc-leaf-1' (ED25519) to the list of known hosts.
admin@clab-lab-dc-leaf-1's password:
Using configuration file(s): []
Welcome to the srlinux CLI.
Type 'help' (and press <ENTER>) if you need any help using this.
--{ [OLD STARTUP] + running }--[  ]--
A:dc-leaf-1# enter candidate
--{ [OLD STARTUP] + candidate shared default }--[  ]--
A:dc-leaf-1# network-instance default description test
--{ [OLD STARTUP] +* candidate shared default }--[  ]--
A:dc-leaf-1# commit now
All changes have been committed. Leaving candidate mode.
--{ [OLD STARTUP] + running }--[  ]--
A:dc-leaf-1# save startup
/system:
    Saved current running configuration as initial (startup) configuration '/etc/opt/srlinux/config.json'

--{ [OLD STARTUP] running }--[  ]--
A:dc-leaf-1#
EOF encountered

Connection to clab-lab-dc-leaf-1 closed.

4 Destroy topology

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# clab des -t topology.yml
INFO[0000] Parsing & checking topology file: topology.yml
INFO[0000] Destroying lab: lab
INFO[0001] Removed container: clab-lab-host11
INFO[0001] Removed container: clab-lab-host20
INFO[0002] Removed container: clab-lab-host12
INFO[0002] Removed container: clab-lab-host10
INFO[0002] Removed container: clab-lab-dc-leaf-2
INFO[0003] Removed container: clab-lab-dc-leaf-1
INFO[0003] Removed container: clab-lab-dc-spine-1
INFO[0003] Removed container: clab-lab-dc-spine-2
INFO[0003] Removing containerlab host entries from /etc/hosts file
INFO[0003] Removing ssh config for containerlab nodes
root@s4qd-pod75:

5 Deploy again. Note that the project directory, clab-lab, is being created again

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# clab dep -t topology.yml
INFO[0000] Containerlab v0.47.1 started
INFO[0000] Parsing & checking topology file: topology.yml
INFO[0000] Creating lab directory: /home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab/clab-lab
INFO[0000] Creating container: "host20"
INFO[0000] Creating container: "host12"
INFO[0000] Creating container: "host10"
INFO[0000] Creating container: "host11"
INFO[0000] Creating container: "dc-spine-1"
INFO[0000] Creating container: "dc-leaf-1"
INFO[0000] Creating container: "dc-leaf-2"
INFO[0000] Creating container: "dc-spine-2"
INFO[0002] Creating link: dc-spine-2:e1-2 <--> dc-leaf-2:e1-2
INFO[0002] Creating link: dc-spine-1:e1-2 <--> dc-leaf-2:e1-1
INFO[0003] Creating link: host20:eth1 <--> dc-leaf-2:e1-10
INFO[0003] Creating link: dc-spine-1:e1-1 <--> dc-leaf-1:e1-1
INFO[0003] Creating link: dc-spine-2:e1-1 <--> dc-leaf-1:e1-2
INFO[0004] Creating link: host11:eth1 <--> dc-leaf-1:e1-11
INFO[0004] Creating link: host10:eth1 <--> dc-leaf-1:e1-10
INFO[0004] Creating link: host12:eth1 <--> dc-leaf-1:e1-12
INFO[0004] Running postdeploy actions for Nokia SR Linux 'dc-leaf-1' node
INFO[0004] Running postdeploy actions for Nokia SR Linux 'dc-spine-1' node
INFO[0004] Running postdeploy actions for Nokia SR Linux 'dc-spine-2' node
INFO[0004] Running postdeploy actions for Nokia SR Linux 'dc-leaf-2' node
INFO[0022] Adding containerlab host entries to /etc/hosts file
INFO[0022] Adding ssh config for containerlab nodes
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+
| # |        Name         | Container ID |             Image             | Kind  |  State  |  IPv4 Address   |     IPv6 Address     |
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+
| 1 | clab-lab-dc-leaf-1  | 1202a5504106 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.9/24  | 2001:172:20:20::9/64 |
| 2 | clab-lab-dc-leaf-2  | e26778959c69 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.5/24  | 2001:172:20:20::5/64 |
| 3 | clab-lab-dc-spine-1 | 000173c3b88b | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.7/24  | 2001:172:20:20::7/64 |
| 4 | clab-lab-dc-spine-2 | f565e19ddaf0 | ghcr.io/nokia/srlinux:23.7.1  | srl   | running | 172.20.20.6/24  | 2001:172:20:20::6/64 |
| 5 | clab-lab-host10     | 6f2d1eb5f3de | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.11/24 | 2001:172:20:20::b/64 |
| 6 | clab-lab-host11     | 56691d463e2b | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.12/24 | 2001:172:20:20::c/64 |
| 7 | clab-lab-host12     | 2d61abfc36cc | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.10/24 | 2001:172:20:20::a/64 |
| 8 | clab-lab-host20     | 65db39a49fff | pklepikov/ubuntu-tools:latest | linux | running | 172.20.20.8/24  | 2001:172:20:20::8/64 |
+---+---------------------+--------------+-------------------------------+-------+---------+-----------------+----------------------+

6 Verify is config change is persistent after redeployment. (default network-instance is not configured)

root@s4qd-pod75:/home/ubuntu/gitlab/srl-dc-fabric-fundamentals-lab# ssh admin@clab-lab-dc-leaf-1
Warning: Permanently added 'clab-lab-dc-leaf-1' (ED25519) to the list of known hosts.
admin@clab-lab-dc-leaf-1's password:
Using configuration file(s): []
Welcome to the srlinux CLI.
Type 'help' (and press <ENTER>) if you need any help using this.
--{ [OLD STARTUP] + running }--[  ]--
A:dc-leaf-1# show network-instance summary
+-----------------------------------------------+------------------------+------------------------+------------------------+-----------------------------------------------+----------------------------------------------------------+
|                     Name                      |          Type          |      Admin state       |       Oper state       |                   Router id                   |                       Description                        |
+===============================================+========================+========================+========================+===============================================+==========================================================+
| mgmt                                          | ip-vrf                 | enable                 | up                     |                                               | Management network instance                              |
+-----------------------------------------------+------------------------+------------------------+------------------------+-----------------------------------------------+----------------------------------------------------------+
@hellt hellt added the bug Something isn't working label Oct 30, 2023
@hellt
Copy link
Member

hellt commented Oct 30, 2023

Thanks for a detailed report @KTodts
we will have a look at it

@steiler
Copy link
Collaborator

steiler commented Nov 2, 2023

I can not confirm this. The message Creating lab directory: ... is expected, we call the os.MakeAll functions, that takes a path and creates all the dirs along the path that do not exist, like mkdir with the -p option. So if the path towards the lab folder is there it is basically a noop.
Deploy:

~/projects/containerlab4$ sudo clab deploy -t lab-examples/srl01/
INFO[0000] Containerlab v0.47.2 started
INFO[0000] Parsing & checking topology file: srl01.clab.yml
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU='ל'
INFO[0000] Creating lab directory: /home/steiler/projects/containerlab4/clab-srl01
INFO[0000] Creating container: "srl"
INFO[0001] Running postdeploy actions for Nokia SR Linux 'srl' node
INFO[0022] Adding containerlab host entries to /etc/hosts file
INFO[0022] Adding ssh config for containerlab nodes
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+
| # |      Name      | Container ID |         Image         | Kind |  State  |  IPv4 Address  |     IPv6 Address     |
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+
| 1 | clab-srl01-srl | af0a98f1c3b0 | ghcr.io/nokia/srlinux | srl  | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+

Configure:

steiler@vbox:~/projects/containerlab4$ ssh clab-srl01-srl
Warning: Permanently added 'clab-srl01-srl' (ED25519) to the list of known hosts.
................................................................
:                  Welcome to Nokia SR Linux!                  :
:              Open Network OS for the NetOps era.             :
:                                                              :
:    This is a freely distributed official container image.    :
:                      Use it - Share it                       :
:                                                              :
: Get started: https://learn.srlinux.dev                       :
: Container:   https://go.srlinux.dev/container-image          :
: Docs:        https://doc.srlinux.dev/22-11                   :
: Rel. notes:  https://doc.srlinux.dev/rn22-11-1               :
: YANG:        https://yang.srlinux.dev/release/v22.11.1       :
: Discord:     https://go.srlinux.dev/discord                  :
: Contact:     https://go.srlinux.dev/contact-sales            :
................................................................

Using configuration file(s): []
Welcome to the srlinux CLI.
Type 'help' (and press <ENTER>) if you need any help using this.
--{ running }--[  ]--
A:srl# enter candidate
--{ candidate shared default }--[  ]--
A:srl# set network-instance default description "my funny description"
--{ * candidate shared default }--[  ]--
A:srl# set network-instance default admin-state enable
--{ * candidate shared default }--[  ]--
A:srl# commit now save
/system:
    Saved current running configuration as initial (startup) configuration '/etc/opt/srlinux/config.json'

All changes have been committed. Leaving candidate mode.
--{ running }--[  ]--
A:srl# quit
Connection to clab-srl01-srl closed.

Check:

steiler@vbox:~/projects/containerlab4$ grep -B3 -A2 funny clab-srl01/srl/config/config.json
    {
      "name": "default",
      "admin-state": "enable",
      "description": "my funny description"
    },
    {

Destroy:

steiler@vbox:~/projects/containerlab4$ sudo clab destroy -t lab-examples/srl01/
INFO[0000] Parsing & checking topology file: srl01.clab.yml
INFO[0000] Destroying lab: srl01
INFO[0000] Removed container: clab-srl01-srl
INFO[0000] Removing containerlab host entries from /etc/hosts file
INFO[0000] Removing ssh config for containerlab nodes

Redeploy:

steiler@vbox:~/projects/containerlab4$ sudo clab deploy -t lab-examples/srl01/
INFO[0000] Containerlab v0.47.2 started
INFO[0000] Parsing & checking topology file: srl01.clab.yml
INFO[0000] Creating docker network: Name="clab", IPv4Subnet="172.20.20.0/24", IPv6Subnet="2001:172:20:20::/64", MTU='ל'
INFO[0000] Creating lab directory: /home/steiler/projects/containerlab4/clab-srl01
INFO[0000] Creating container: "srl"
INFO[0000] Running postdeploy actions for Nokia SR Linux 'srl' node
INFO[0016] Adding containerlab host entries to /etc/hosts file
INFO[0016] Adding ssh config for containerlab nodes
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+
| # |      Name      | Container ID |         Image         | Kind |  State  |  IPv4 Address  |     IPv6 Address     |
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+
| 1 | clab-srl01-srl | 003c92e9f56e | ghcr.io/nokia/srlinux | srl  | running | 172.20.20.2/24 | 2001:172:20:20::2/64 |
+---+----------------+--------------+-----------------------+------+---------+----------------+----------------------+

Check file:

steiler@vbox:~/projects/containerlab4$ grep -B3 -A2 funny clab-srl01/srl/config/config.json
    {
      "name": "default",
      "admin-state": "enable",
      "description": "my funny description"
    },
    {

Check SRL:

steiler@vbox:~/projects/containerlab4$ ssh clab-srl01-srl
Warning: Permanently added 'clab-srl01-srl' (ED25519) to the list of known hosts.
................................................................
:                  Welcome to Nokia SR Linux!                  :
:              Open Network OS for the NetOps era.             :
:                                                              :
:    This is a freely distributed official container image.    :
:                      Use it - Share it                       :
:                                                              :
: Get started: https://learn.srlinux.dev                       :
: Container:   https://go.srlinux.dev/container-image          :
: Docs:        https://doc.srlinux.dev/22-11                   :
: Rel. notes:  https://doc.srlinux.dev/rn22-11-1               :
: YANG:        https://yang.srlinux.dev/release/v22.11.1       :
: Discord:     https://go.srlinux.dev/discord                  :
: Contact:     https://go.srlinux.dev/contact-sales            :
................................................................

Using configuration file(s): []
Welcome to the srlinux CLI.
Type 'help' (and press <ENTER>) if you need any help using this.
--{ running }--[  ]--
A:srl# info from running network-instance default
    network-instance default {
        admin-state enable
        description "my funny description"
    }
--{ running }--[  ]--
A:srl#

So, well... someone else maybe also try this, but I can't reproduce the issue.

@KTodts
Copy link
Author

KTodts commented Nov 2, 2023 •
edited
Loading

I don't know if it makes any difference but I'm using startup-config in the topology file.

edit:
My understanding was, that clab project dir has priority to the startup-config file. When you deploy for the first time there is no clab project dir, so it takes the config from startup-config. Once you alter the config and redeploy it should take the config from the clab project dir

@steiler
Copy link
Collaborator

steiler commented Nov 2, 2023

Hmm @hellt whats our definition of a startup-config?
Is that the if no config already exists startup-config or is it the always and in any case startup-config?

@steiler
Copy link
Collaborator

steiler commented Nov 2, 2023

ok, if a config is present in the nodes lab dir we should use that, if not e.g. --reconfigure we should download.
I'll figure out how we do that best

@hellt
Copy link
Member

hellt commented Nov 2, 2023

ah, startup-config takes precendence over the local dir @KTodts
The goal of a startup-config is to be the "startup config".

The idea was that our topology should be as declarative as possible, so if we have a startup config in the topo file, this file should be used as a startup config, and thus overriding any config present in the node lab directory.

@hellt
Copy link
Member

hellt commented Nov 2, 2023

Although this note here https://containerlab.dev/manual/nodes/#embedded-startup-config says otherwise :D

To be honest, I think we need to then use existing labdir config indeed

/cc @steiler @KTodts

@KTodts
Copy link
Author

KTodts commented Nov 2, 2023

Although this note here https://containerlab.dev/manual/nodes/#embedded-startup-config says otherwise :D

To be honest, I think we need to then use existing labdir config indeed

/cc @steiler @KTodts

This way after installing a project from a repo you have the possibility to build on top of that.

@steiler
Copy link
Collaborator

steiler commented Nov 2, 2023

Yes and the note says so... We do have the enforce-startup-config flag otherwise...
This behavior got screwed when I implemented the http(s) download of the config...
Mea culpa. Will fix it!

@gusman
Copy link

gusman commented Dec 19, 2023

I just found this issue has been occurred starting from version 0.45.0.

The main issue that I have experienced is even though I have saved my config before destroying the topology, the config is not persistent anymore. After executing deploy command it always copies or uses the config file from startup-config definition in topo-yaml file.,

@hellt
Copy link
Member

hellt commented Dec 19, 2023

yes, this is still unresolved.
I will link your internal message to be evaluated when we get to fix this

@mahmutaydin1
Copy link

@hellt I am having same issue. I have startup config in clab lab directory. When I deploy lab again, startup config is not seen.
have you any progress for this bug

thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hellt hellt

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@gusman @steiler @hellt @KTodts @mahmutaydin1