After update from k3s version v1.30.4+k3s1 to v1.31.0+k3s1

[husarzlukasz]

Problem with secrets-store-csi-driver after update to v1.31.0+k3s1

Environmental Info:
K3s Version before update:
k3s version v1.30.4+k3s1
after update:
v1.31.0+k3s1

Node(s) CPU architecture, OS, and Version:
3x nodes: Linux 6.9.7+bpo-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.9.7-1~bpo12+1 (2024-07-03) x86_64 GNU/Linux

Cluster Configuration:
3 master/agent nodes

Describe the bug:
After upgrade to latest version of k3s i notice a lot of warnings of:

Warning:
 kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

 Warning  FailedMount             48s (x31 over 47m)    kubelet                     MountVolume.SetUp failed for volume "backend" : kubernetes.io/csi: mounter.SetUpAt failed to get CSI client: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

nothing changes in volumes configuration and csi-driver configuration:
csidriver was okay:

secrets-store.csi.k8s.io false true false <unset> false Ephemeral

Volumes:
  backend:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            secrets-store.csi.k8s.io
    FSType:            
    ReadOnly:          true
    VolumeAttributes:      secretProviderClass=backend

Delete pod with warning to recreate cause a pod stuck in init state because of error ^

Steps To Reproduce:
Have https://github.com/kubernetes-sigs/secrets-store-csi-driver deployed and used
k3s version v1.30.4+k3s1 upgrade to v1.31.0+k3s1

Expected behavior:
no warnings: driver name secrets-store.csi.k8s.io not found in the list of registered CSI drivers

Actual behavior:

Downgrade cluster to previous version resolved the problem

[brandond]

I would probably open an issue with the secrets-store-csi-driver project. They can suggest steps to troubleshoot. Did you perhaps need to upgrade it to be compatible with Kubernetes 1.31?

[husarzlukasz]

@brandond Thanks for the answer. I've created a ticket in the secrets-store-csi-driver project. I currently have the latest version ;)