cockpit.socket crashes after attempt to access it over http/s port 9090

[remrengg]

Explain what happens

1. I try to access cockpit from the webbrowser.
2. Cockpit.socket crashes with this error:
   × cockpit.socket - Cockpit Web Service Socket
   Loaded: loaded (/usr/lib/systemd/system/cockpit.socket; enabled; preset: disabled)
   Active: failed (Result: trigger-limit-hit) since Tue 2024-09-24 15:33:43 CEST; 8s ago
   Duration: 22.153s
   Triggers: ● cockpit.service
   Docs: man:cockpit-ws(8)
   Listen: [::]:9090 (Stream)
   Process: 27228 ExecStartPost=/usr/share/cockpit/motd/update-motd localhost (code=exited, status=0/SUCCESS)
   Process: 27236 ExecStartPost=/bin/ln -snf active.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
   Process: 27267 ExecStopPost=/bin/ln -snf inactive.motd /run/cockpit/motd (code=exited, status=0/SUCCESS)
   CPU: 9ms
   systemd[1]: Starting Cockpit Web Service Socket...
   systemd[1]: Listening on Cockpit Web Service Socket.
   systemd[1]: cockpit.socket: Trigger limit hit, refusing further activation.
   systemd[1]: cockpit.socket: Failed with result 'trigger-limit-hit'.

Version of Cockpit

The one that opensuse leap 15.6 downloads automatically when you type zypper install cockpit on 23.09.2024

Where is the problem in Cockpit?

Networking

Server operating system

openSUSE

Server operating system version

Opensuse Leap 15.6

What browsers are you using?

Firefox

System log

systemd[1]: Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.
 cockpit.service: Job cockpit.service/start failed with result 'dependency'.
 Dependency failed for Cockpit Web Service.it'.

[martinpitt]

So cockpit.service failed because one of its dependencies failed. The most probable one is cockpit-ws-user.service, can you please check systemctl status cockpit-ws-user.service and journalctl -b -u cockpit-ws-user.service?

[remrengg]

systemctl status cockpit-ws-user.service gives Back that the Service is still running after the failiure. The journalctl command gives back this:
systemd[1]: cockpit-ws-user.service: Deactivated successfully.
systemd[1]: Stopped Dynamic user for cockpit-ws.
systemd[1]: Starting Dynamic user for cockpit-ws...
systemd[1]: Finished Dynamic user for cockpit-ws.

[martinpitt]

OK, it's not that then. Please check the others: cockpit-wsinstance-http.socket and cockpit-wsinstance-https-factory.socket -- or more generally, systemctl --failed?

[remrengg]

They both failed. cockpit-wsinstance-http.socket failed with that:
systemd[1]: Failed to listen on Socket for Cockpit Web Service http instance.
systemd[1]: cockpit-wsinstance-http.socket: Start request repeated too quickly.
systemd[1]: cockpit-wsinstance-http.socket: Failed with result 'exit-code'.

And cockpit-wsinstance-https-factory.socket with
Dependency failed for Socket for Cockpit Web Service https instance factory.
systemd[1]: cockpit-wsinstance-https-factory.socket: Job cockpit-wsinstance-https-factory.socket/start failed with result 'dependency'.

and sometimes there are these messages aswell:
cockpit-wsinstance-https-factory.socket: Failed to resolve user cockpit-ws: No such process
Dependency failed for Socket for Cockpit Web Service https instance factory.
systemd[1]: cockpit-wsinstance-https-factory.socket: Job cockpit-wsinstance-https-factory.socket/start failed with result 'dependency'.
systemd[1]: cockpit-wsinstance-https-factory.socket: Control process exited, code=exited, status=217/USER

systemctl --failed shows that:
UNIT LOAD ACTIVE SUB DESCRIPTION
● cockpit-wsinstance-http.socket loaded failed failed Socket for Cockpit Web Service http instance
● cockpit-wsinstance-https-factory.socket loaded failed failed Socket for Cockpit Web Service https instance factory
● cockpit.socket loaded failed failed Cockpit Web Service Socket

[martinpitt]

OK -- sorry, none of this is the actual reason for the failure, this is all just follow-ups. Can you please do

systemctl stop cockpit.socket cockpit.service
systemctl reset-failed
systemctl start cockpit.socket
journalctl -f

then try to log in, and copy&paste the whole output of journalctl here? Then press Control-C to stop it. Thanks!

[martinpitt]

Oh, wait:

cockpit-wsinstance-https-factory.socket: Failed to resolve user cockpit-ws: No such process

So it cannot resolve the DynamicUser=.

This means that /etc/nsswitch.conf does not have systemd in its passwd: line -- can you confirm? Your system may lack the systemd nss module. In Fedora/RHEL that's built into the systemd-libs rpm, but e.g. Debian has a separate libnss-systemd package. Opensuse LEAP may also have split this out, and thus cockpit-ws.rpm may miss that dependency.

[remrengg]

/etc/nsswitch.conf has compat in the passwd: line.

[martinpitt]

Right, and it needs systemd somewhere in addition to "compat" or " files".

@Nykseli @Lunarequest do you have an idea here? Does LEAP need an additional package dependency to ensure that nss-systemd is installed and enabled in nsswitch.conf? I.e. to ensure that services with DynamicUser=yes work?

[Nykseli]

This is a known issue and fix has been found for it [1] [2]

Afaik, the fix is not yet pushed to Leap 15.6

CC: @SludgeGirl

[1]: https://bugzilla.suse.com/show_bug.cgi?id=1230546
[2]: https://bugzilla.suse.com/show_bug.cgi?id=1230638

[martinpitt]

@Nykseli thanks -- FTR, this was changed in https://cockpit-project.org/blog/cockpit-317.html . It is intended that there are two separate system users cockpit-ws (dynamic now) and cockpit-wsinstance (still static because reasons). (wrt. https://bugzilla.suse.com/show_bug.cgi?id=1230546#c2)