Find a way to grab the new certificate generated by Lets'encrypt to update our cluster (or a cluster)

[cmoulliard]

TODO

Find a way to grab the new certificate generated by Lets'encrypt to update our cluster (or a cluster)

Current procedure is:

• Identify the VM running the certmanager and lets'encrypt for domain: snowdrop.dev
• ssh to the VM and get the tls and key files

passstore-vm-ssh.sh openstack snowdrop-k8s
alias k=kubectl
k get secret/qshift-snowdrop-dev-tls -n snowdrop-site -ojson | jq -r '.data."tls.crt"' | base64 -d > tls.crt
k get secret/qshift-snowdrop-dev-tls -n snowdrop-site -ojson | jq -r '.data."tls.key"' | base64 -d > tls.key

• Identify the ocp4 cluster where the console's certificate should be updated within the password store
• scp the files to the target cluster
• Recreate the secret used by ingress to access the ingress URL

Commands to be used

$ alias k=kubectl
$ mkdir cert-renew
$ pass openstack/ocp-qshift-wlg4j/kubeconfig > cert-renew/kubeconfig
$ cp <path_to_file>/tls.crt cert-renew/
$ cp <path_to_file>/tls.key cert-renew/
$ export KUBECONFIG=./kubeconfig
$ k -n openshift-ingress delete secret/qshift-console
$ k -n openshift-ingress create secret tls qshift-console --cert=cert-renew/tls.crt --key=cert-renew/tls.key # --dry-run="client" -oyaml

[cmoulliard]

If the k8s certificate expires, here is what you should do

1. ssh to the snowdrop-k8s vm

// Use password team and script: https://github.com/snowdrop/k8s-infra/tree/main/tools
passstore-vm-ssh.sh openstack snowdrop-k8s

2. Renew the certificate

k get po -A
Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2024-09-23T08:26:33Z is after 2024-08-08T07:37:44Z

[snowdrop@snowdrop-k8s ~]$ ls -la /etc/kubernetes/pki/
total 56
drwxr-xr-x 1 root root  480 Aug  9  2023 .
drwxr-xr-x 1 root root  142 Aug  9  2023 ..
-rw-r--r-- 1 root root 1289 Aug  9  2023 apiserver.crt
-rw-r--r-- 1 root root 1155 Aug  9  2023 apiserver-etcd-client.crt
-rw------- 1 root root 1675 Aug  9  2023 apiserver-etcd-client.key
-rw------- 1 root root 1675 Aug  9  2023 apiserver.key
-rw-r--r-- 1 root root 1164 Aug  9  2023 apiserver-kubelet-client.crt
-rw------- 1 root root 1675 Aug  9  2023 apiserver-kubelet-client.key
-rw-r--r-- 1 root root 1099 Aug  9  2023 ca.crt
-rw------- 1 root root 1675 Aug  9  2023 ca.key
drwxr-xr-x 1 root root  184 Aug  9  2023 etcd
-rw-r--r-- 1 root root 1115 Aug  9  2023 front-proxy-ca.crt
-rw------- 1 root root 1675 Aug  9  2023 front-proxy-ca.key
-rw-r--r-- 1 root root 1119 Aug  9  2023 front-proxy-client.crt
-rw------- 1 root root 1679 Aug  9  2023 front-proxy-client.key
-rw------- 1 root root 1679 Aug  9  2023 sa.key
-rw------- 1 root root  451 Aug  9  2023 sa.pub

[snowdrop@snowdrop-k8s ~]$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Aug  9 07:37:44 2023 GMT
            Not After : Aug  8 07:37:44 2024 GMT


[snowdrop@snowdrop-k8s ~]$ sudo kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
[renew] Error reading configuration from the Cluster. Falling back to default configuration

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed

Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.
[snowdrop@snowdrop-k8s ~]$ sudo systemctl restart kubelet

[snowdrop@snowdrop-k8s ~]$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -text |grep ' Not '
            Not Before: Aug  9 07:37:44 2023 GMT
            Not After : Sep 23 08:30:59 2025 GMT

3. Replace the home .kube/config file with the new generated

sudo cp /etc/kubernetes/admin.conf /home/snowdrop/.kube/config