v0.27.3 not working on openwrt system

[xcf13363175]

v0.27.3 not working on openwrt system, while v0.27.2 works fine.

2024-04-20T22:56:55Z INFO client/cmd/service_controller.go:80: stopped Netbird service
2024-04-20T22:56:56Z INFO client/cmd/service_controller.go:24: starting Netbird service
2024-04-20T22:56:56Z INFO client/cmd/service_controller.go:64: started daemon server: /var/run/netbird.sock
2024-04-20T22:56:56Z INFO client/internal/connect.go:104: starting NetBird client version 0.27.3 on linux/arm64
2024-04-20T22:56:58Z WARN client/internal/wgproxy/factory_linux.go:15: failed to initialize ebpf proxy, fallback to user space proxy: bpf_link not supported (requires >= v5.7)
2024-04-20T22:56:58Z INFO client/internal/routemanager/manager.go:88: Routing setup complete
2024-04-20T22:56:58Z WARN client/firewall/nftables/route_linux.go:69: table 'filter' not found for forward rules
2024-04-20T22:56:59Z ERRO client/firewall/nftables/route_linux.go:82: failed to create containers for route: nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
2024-04-20T22:56:59Z ERRO client/firewall/create_linux.go:54: failed to create nftables manager: nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
2024-04-20T22:56:59Z ERRO client/internal/engine.go:286: failed creating firewall manager: nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
2024-04-20T22:56:59Z INFO client/internal/dns/host_linux.go:68: System DNS manager discovered: file
2024-04-20T22:57:02Z INFO client/internal/connect.go:247: Netbird engine started, my IP is: 100.114.106.xx/16
2024-04-20T22:57:02Z WARN signal/client/grpc.go:152: disconnected from the Signal Exchange due to an error: rpc error: code = Canceled desc = context canceled
2024-04-20T22:57:02Z ERRO signal/client/grpc.go:180: exiting the Signal service connection retry loop due to the unrecoverable error: context canceled
2024-04-20T22:57:02Z WARN management/client/grpc.go:172: exiting the Management service connection retry loop due to the unrecoverable error: context canceled
2024-04-20T22:57:03Z ERRO client/internal/dns/file_linux.go:144: Failed to remove netbird nameserver from /etc/resolv.conf.original.netbird on backup restore: parse backup resolv.conf: failed to open /etc/resolv.conf.original.netbird file: open /etc/resolv.conf.original.netbird: no such file or directory
2024-04-20T22:57:03Z ERRO client/internal/dns/server.go:195: restoring /etc/resolv.conf from /etc/resolv.conf.original.netbird: checking stats for /etc/resolv.conf.original.netbird file when copying it. Error: stat /etc/resolv.conf.original.netbird: no such file or directory
2024-04-20T22:57:03Z INFO client/internal/routemanager/manager.go:112: Routing cleanup complete
2024-04-20T22:57:03Z ERRO iface/bind/udp_mux_universal.go:85: error while reading packet: shared socked stopped
2024-04-20T22:57:03Z INFO client/internal/engine.go:224: stopped Netbird Engine
2024-04-20T22:57:03Z INFO client/internal/connect.go:261: stopped NetBird client
2024-04-20T22:57:07Z INFO client/cmd/root.go:161: shutdown signal received
2024-04-20T22:57:09Z INFO client/cmd/service_controller.go:80: stopped Netbird service

[xcf13363175]

It should be a problem that nftables cannot be started, causing openwrt to be unable to access all servers. ps. Currently using iptables
Is it possible not to force the use of nftables?

[xcf13363175]

v0.27.4 still the same. Can only return to use v0.27.2.

nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory

2024-05-01T07:00:18Z INFO client/cmd/root.go:161: shutdown signal received

[mlsmaycon]

@xcf13363175 can you try running the client in the foreground with the environment variable NB_SKIP_NFTABLES_CHECK=true?

sudo `NB_SKIP_NFTABLES_CHECK=true` netbird up -F -l debug 

[xcf13363175]

Thanks for the reply and the solution ideas. The reason why openwrt cannot be used in ≥ v0.27.3 has been found. Manually setting sysctl -w net.ipv4.conf.all.src_valid_mark=0
it can be used normally.
Hope that sysctl -w net.ipv4.conf.all.src_valid_mark=0
will be added in the new version

Attached are two versions of the log.
v0.27.2

2024-05-01T07:40:18Z INFO client/internal/connect.go:104: starting NetBird client version 0.27.2 on linux/arm64
2024-05-01T07:40:18Z DEBG client/internal/connect.go:165: connecting to the Management service api.netbird.io:443
2024-05-01T07:40:18Z DEBG client/internal/connect.go:173: connected to the Management service api.netbird.io:443
2024-05-01T07:40:20Z DEBG signal/client/grpc.go:92: connected to Signal Service: signal.netbird.io:443
2024-05-01T07:40:20Z DEBG client/internal/wgproxy/proxy_ebpf.go:38: instantiate ebpf proxy
2024-05-01T07:40:20Z DEBG client/internal/ebpf/ebpf/wg_proxy_linux.go:11: load ebpf WG proxy
2024-05-01T07:40:20Z WARN client/internal/wgproxy/factory_linux.go:15: failed to initialize ebpf proxy, fallback to user space proxy: bpf_link not supported (requires >= v5.7)
2024-05-01T07:40:20Z INFO client/internal/routemanager/manager.go:83: Routing setup complete
2024-05-01T07:40:20Z DEBG iface/tun_kernel_linux.go:70: adding device: wt0
2024-05-01T07:40:20Z DEBG iface/tun_kernel_linux.go:198: adding address 100.114.XX.XX/16 to interface: wt0
2024-05-01T07:40:20Z DEBG iface/tun_kernel_linux.go:86: setting MTU: 1280 interface: wt0
2024-05-01T07:40:20Z DEBG iface/wg_configurer_kernel.go:29: adding Wireguard private key
2024-05-01T07:40:20Z DEBG client/firewall/create_linux.go:51: creating an nftables firewall manager
2024-05-01T07:40:20Z WARN client/firewall/nftables/route_linux.go:69: table 'filter' not found for forward rules
2024-05-01T07:40:21Z ERRO client/firewall/nftables/route_linux.go:82: failed to create containers for route: nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
2024-05-01T07:40:21Z ERRO client/firewall/create_linux.go:54: failed to create nftables manager: nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
2024-05-01T07:40:21Z ERRO client/internal/engine.go:286: failed creating firewall manager: nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
2024-05-01T07:40:21Z DEBG iface/tun_kernel_linux.go:110: bringing up interface: wt0
2024-05-01T07:40:21Z DEBG iface/tun_kernel_linux.go:130: device is ready to use: wt0
2024-05-01T07:40:21Z INFO client/internal/dns/host_linux.go:68: System DNS manager discovered: file
2024-05-01T07:40:21Z DEBG signal/client/grpc.go:137: signal connection state READY
2024-05-01T07:40:21Z INFO signal/client/grpc.go:158: connected to the Signal Service stream
2024-05-01T07:40:21Z DEBG client/internal/engine.go:596: connecting to Management Service updates stream
2024-05-01T07:40:21Z DEBG management/client/grpc.go:120: management connection state READY

v0.27.4

2024-05-01T07:56:23Z INFO client/internal/connect.go:104: starting NetBird client version 0.27.4 on linux/arm64
2024-05-01T07:56:23Z DEBG client/internal/connect.go:165: connecting to the Management service api.netbird.io:443
2024-05-01T07:56:23Z DEBG client/internal/connect.go:173: connected to the Management service api.netbird.io:443
2024-05-01T07:56:25Z DEBG signal/client/grpc.go:92: connected to Signal Service: signal.netbird.io:443
2024-05-01T07:56:25Z DEBG client/internal/wgproxy/proxy_ebpf.go:38: instantiate ebpf proxy
2024-05-01T07:56:25Z DEBG client/internal/ebpf/ebpf/wg_proxy_linux.go:11: load ebpf WG proxy
2024-05-01T07:56:25Z WARN client/internal/wgproxy/factory_linux.go:15: failed to initialize ebpf proxy, fallback to user space proxy: bpf_link not supported (requires >= v5.7)
2024-05-01T07:56:25Z DEBG client/internal/routemanager/systemops_linux.go:558: Set sysctl net.ipv4.conf.all.src_valid_mark from 0 to 1
2024-05-01T07:56:25Z INFO client/internal/routemanager/manager.go:88: Routing setup complete
2024-05-01T07:56:25Z DEBG iface/tun_kernel_linux.go:70: adding device: wt0
2024-05-01T07:56:25Z DEBG iface/tun_kernel_linux.go:198: adding address 100.114.XX/16 to interface: wt0
2024-05-01T07:56:25Z DEBG iface/tun_kernel_linux.go:86: setting MTU: 1280 interface: wt0
2024-05-01T07:56:25Z DEBG iface/wg_configurer_kernel.go:27: adding Wireguard private key
2024-05-01T07:56:25Z DEBG client/firewall/create_linux.go:51: creating an nftables firewall manager
2024-05-01T07:56:25Z WARN client/firewall/nftables/route_linux.go:69: table 'filter' not found for forward rules
2024-05-01T07:56:25Z ERRO client/firewall/nftables/route_linux.go:82: failed to create containers for route: nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
2024-05-01T07:56:25Z ERRO client/firewall/create_linux.go:54: failed to create nftables manager: nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
2024-05-01T07:56:25Z ERRO client/internal/engine.go:286: failed creating firewall manager: nftables: unable to initialize table: conn.Receive: netlink receive: no such file or directory
2024-05-01T07:56:25Z DEBG iface/tun_kernel_linux.go:110: bringing up interface: wt0
2024-05-01T07:56:25Z DEBG iface/tun_kernel_linux.go:130: device is ready to use: wt0
2024-05-01T07:56:25Z INFO client/internal/dns/host_linux.go:68: System DNS manager discovered: file
2024-05-01T07:56:25Z DEBG signal/client/grpc.go:137: signal connection state READY

[mlsmaycon]

@lixmal can you have a look at this behavior with openwrt?

[lixmal]

@xcf13363175 you can set the env var NB_USE_LEGACY_ROUTING=true or even NB_DISABLE_CUSTOM_ROUTING=true (if you don't need default routes) for the daemon to disable the behavior.

I'll investigate the issue

[lixmal]

@xcf13363175 can you share the openwrt/kernel version that you use and which issue you've observerd with v0.27.4? The logs don't show any issues other than the nftables one.

I've tried OpenWrt 23.05.3 with netbird v0.27.3 and it works fine

[xcf13363175]

The version used is OpenWrt 23.05. The problem encountered is that when netbird ≥ v0.27.3 and passwall2 are opened at the same time, the kernel panic. When only netbird is opened, all versions are ok, or when netbird < v0.27.3 and passwall2 are opened at the same time, there is no problem.
The current solution is to open netbird and then manually execute sysctl -w net.ipv4.conf.all.src_valid_mark=0.

[lixmal]

@xcf13363175 have you opened a bug report with the maintainer(s) of passwall2 then?
Have you tried the env vars I have proposed earlier?

[xcf13363175]

@lixmal Trying to set the environment variables provided before still crashes. It can only be restarted by unplugging the power. It should be caused by the incompatibility of the upstream xray-core of passwall2. A bug report has been submitted. Xray-core has PR of net.ipv4.conf.all.src_valid_mark=1XTLS/Xray-core#2762 . I don’t know if it is because the net.ipv4.conf.all.src_valid_mark settings of the two software are different.

[lixmal]

@xcf13363175

@lixmal Trying to set the environment variables provided before still crashes.

The env vars disable settings and sysctl options.
Can you share how you set those env vars? They need to be set for the daemon, not the cli tool.
You can test it in foreground mode.

netbird service stop
NB_USE_LEGACY_ROUTING=true netbird up -F

[xcf13363175]

I used the provided command to start netbird in the foreground and it can be used normally. Thank you very much for your help. Found net.ipv4.conf.all.src_valid_mark = 0 after using the command. The previous test installed sudo and used sudo `NB_USE_LEGACY_ROUTING=true` netbird up -F -l debug and it could not be used normally.

[xcf13363175]

The problem has been solved, thanks again.