Skip to content

Commit

Permalink
Integrate k3s and multus
Browse files Browse the repository at this point in the history
Signed-off-by: Manuel Buil <[email protected]>
  • Loading branch information
manuelbuil committed Feb 18, 2024
1 parent d8907ce commit 5073222
Show file tree
Hide file tree
Showing 11 changed files with 148 additions and 10 deletions.
44 changes: 44 additions & 0 deletions docs/adrs/multus.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Record architecture decisions

Date: 2024-02-14

## Status

Approved

## Context

### Multus

Multus is a CNI multiplexer that allows pods to have multiple network interfaces. We have users that are operating K3s + Multus but it is not super obvious how to configure it to work with K3s and how to add the additional pieces needed (e.g. IPAM or additional CNI plugins). We could facilitate this by creating an integration with Multus.

We we will wait a bit to include whereabouts. That project is using very old dependencies which will creep in CVEs

### Design suggestion

Add multus to the k3s-charts repo. That multus chart will consume the tarball we generate in rke2-charts, i.e. both rke2 and k3s will use the same chart with minimal diffs (e.g. the Chart name will be k3s-multus instead of rke2-multus).

Then, multus will be consumed as traefik:
* The chart gets downloaded with `make download`
* The chart tarball gets embedded in k3s binary with `go generate` and included in `pkg/static/zz_generated_bindata.go`
* The HelmChart manifest pointing to the chart tarball gets embedded in k3s binary with `go generate` and included in `pkg/deploy/zz_generated_bindata.go`

K3s will include a new `--multus` boolean flag. When that flag is true, we would leave the HelmChart manifest installing multus and whereabouts.

The multus chart will install a daemonset that:
* deploys the necessary binaries (multus and common CNI plugins) in each node
* generates the correct CNI plugin
* Installs the required CRDs

It sucks a bit that the daemonset stays dormant forever after doing the job instead of just dying, but the alternatives are worse

## Alternatives

* K3s creates a job that picks the multus and whereabouts CNI plugins from the `image-build-cni-plugins` and copies them to each node. However, configuring jobs to run on each node is not that easy and very error prone. Therefore, we decided to reject this idea

* K3s includes the multus and whereabouts CNI plugins as part of its multi-exec cni binary. However, the whereabouts binary is using very old dependencies which would creep in CVEs. Moreover, the size of the K3s binary would increase more than 10%, something not acceptable for a something that the vast majority of K3s users will not enable


## Decision

YES
14 changes: 14 additions & 0 deletions manifests/multus.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
name: multus
namespace: kube-system
spec:
chart: https://%{KUBERNETES_API}%/static/charts/multus-4.0.201+upv4.0.2-build2024020801.tgz
valuesContent: |-
config:
cni_conf:
confDir: /var/lib/rancher/k3s/agent/etc/cni/net.d
binDir: %{DATA_DIR}%
kubeconfig: /var/lib/rancher/k3s/agent/etc/cni/net.d/multus.d/multus.kubeconfig
6 changes: 6 additions & 0 deletions pkg/cli/cmds/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ type Server struct {
EtcdS3Folder string
EtcdS3Timeout time.Duration
EtcdS3Insecure bool
Multus bool
ServiceLBNamespace string
}

Expand Down Expand Up @@ -489,6 +490,11 @@ var ServerFlags = []cli.Flag{
Usage: "(experimental/components) Enable embedded distributed container registry; requires use of embedded containerd",
Destination: &ServerConfig.EmbeddedRegistry,
},
&cli.BoolFlag{
Name: "multus",
Usage: "(experimental/networking) Enable multus",
Destination: &ServerConfig.Multus,
},
NodeNameFlag,
WithNodeIDFlag,
NodeLabels,
Expand Down
6 changes: 6 additions & 0 deletions pkg/cli/server/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -173,6 +173,7 @@ func run(app *cli.Context, cfg *cmds.Server, leaderControllers server.CustomCont
serverConfig.ControlConfig.EncryptSecrets = cfg.EncryptSecrets
serverConfig.ControlConfig.EtcdExposeMetrics = cfg.EtcdExposeMetrics
serverConfig.ControlConfig.EtcdDisableSnapshots = cfg.EtcdDisableSnapshots
serverConfig.ControlConfig.Multus = cfg.Multus
serverConfig.ControlConfig.VLevel = cmds.LogConfig.VLevel
serverConfig.ControlConfig.VModule = cmds.LogConfig.VModule

Expand Down Expand Up @@ -396,6 +397,11 @@ func run(app *cli.Context, cfg *cmds.Server, leaderControllers server.CustomCont
serverConfig.ControlConfig.Disables["ccm"] = true
}

if !serverConfig.ControlConfig.Multus {
serverConfig.ControlConfig.Skips["multus"] = true
serverConfig.ControlConfig.Disables["multus"] = true
}

tlsMinVersionArg := getArgValueFromList("tls-min-version", serverConfig.ControlConfig.ExtraAPIArgs)
serverConfig.ControlConfig.TLSMinVersion, err = kubeapiserverflag.TLSVersion(tlsMinVersionArg)
if err != nil {
Expand Down
1 change: 1 addition & 0 deletions pkg/daemons/config/types.go
Original file line number Diff line number Diff line change
Expand Up @@ -157,6 +157,7 @@ type CriticalControlArgs struct {
FlannelIPv6Masq bool `cli:"flannel-ipv6-masq"`
FlannelExternalIP bool `cli:"flannel-external-ip"`
EgressSelectorMode string `cli:"egress-selector-mode"`
Multus bool `cli:"multus"`
ServiceIPRange *net.IPNet `cli:"service-cidr"`
ServiceIPRanges []*net.IPNet `cli:"service-cidr"`
}
Expand Down
23 changes: 23 additions & 0 deletions pkg/deploy/zz_generated_bindata.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

10 changes: 10 additions & 0 deletions pkg/server/server.go
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ import (
"context"
"fmt"
"os"
"os/exec"
"path"
"path/filepath"
"runtime/debug"
Expand Down Expand Up @@ -279,6 +280,13 @@ func stageFiles(ctx context.Context, sc *Context, controlConfig *config.Control)
dnsIPFamilyPolicy = "RequireDualStack"
}

// Find the /var/lib/rancher/k3s/data/${SHA}/bin/ directory. Same procedure we use in pkg/agent/config/config.go
hostLocal, err := exec.LookPath("host-local")
if err != nil {
return errors.Wrap(err, "failed to find host-local")
}
CNIBinDir := filepath.Dir(hostLocal)

templateVars := map[string]string{
"%{CLUSTER_DNS}%": controlConfig.ClusterDNS.String(),
"%{CLUSTER_DNS_LIST}%": fmt.Sprintf("[%s]", util.JoinIPs(controlConfig.ClusterDNSs)),
Expand All @@ -288,13 +296,15 @@ func stageFiles(ctx context.Context, sc *Context, controlConfig *config.Control)
"%{SYSTEM_DEFAULT_REGISTRY}%": registryTemplate(controlConfig.SystemDefaultRegistry),
"%{SYSTEM_DEFAULT_REGISTRY_RAW}%": controlConfig.SystemDefaultRegistry,
"%{PREFERRED_ADDRESS_TYPES}%": addrTypesPrioTemplate(controlConfig.FlannelExternalIP),
"%{DATA_DIR}%": CNIBinDir,
}

skip := controlConfig.Skips
if !skip["traefik"] && isHelmChartTraefikV1(sc) {
logrus.Warn("Skipping Traefik v2 deployment due to existing Traefik v1 installation")
skip["traefik"] = true
}
logrus.Infof("MANU - templateVars: %v", templateVars)
if err := deploy.Stage(dataDir, templateVars, skip); err != nil {
return err
}
Expand Down
31 changes: 27 additions & 4 deletions pkg/static/zz_generated_bindata.go

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions scripts/airgap/image-list-multus.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
docker.io/rancher/hardened-multus-cni:v4.0.2-build20240208
docker.io/rancher/hardened-cni-plugins:v1.4.0-build20240122
docker.io/rancher/mirrored-library-busybox:1.36.1
2 changes: 1 addition & 1 deletion scripts/download
Original file line number Diff line number Diff line change
Expand Up @@ -41,5 +41,5 @@ git clone --single-branch --branch=${VERSION_CONTAINERD} --depth=1 https://${PKG

for CHART_FILE in $(grep -rlF HelmChart manifests/ | xargs yq eval --no-doc .spec.chart | xargs -n1 basename); do
CHART_NAME=$(echo $CHART_FILE | grep -oE '^(-*[a-z])+')
curl -sfL ${CHARTS_URL}/${CHART_NAME}/${CHART_FILE} -o ${CHARTS_DIR}/${CHART_FILE}
curl -sfL ${CHARTS_URL}/${CHART_NAME}/${CHART_FILE} -o ${CHARTS_DIR}/${CHART_FILE}
done
18 changes: 13 additions & 5 deletions scripts/package-airgap
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,20 @@ cd $(dirname $0)/..

. ./scripts/version.sh

function createTarball() {
images=$(cat "${1}")
xargs -n1 docker pull <<< "${images}"
docker save ${images} -o dist/artifacts/${2}-${ARCH}.tar
zstd --no-progress -T0 -16 -f --long=25 dist/artifacts/${2}-${ARCH}.tar -o dist/artifacts/${2}-${ARCH}.tar.zst
pigz -v -c dist/artifacts/${2}-${ARCH}.tar > dist/artifacts/${2}-${ARCH}.tar.gz
}


airgap_image_file='scripts/airgap/image-list.txt'
images=$(cat "${airgap_image_file}")
xargs -n1 docker pull <<< "${images}"
docker save ${images} -o dist/artifacts/k3s-airgap-images-${ARCH}.tar
zstd --no-progress -T0 -16 -f --long=25 dist/artifacts/k3s-airgap-images-${ARCH}.tar -o dist/artifacts/k3s-airgap-images-${ARCH}.tar.zst
pigz -v -c dist/artifacts/k3s-airgap-images-${ARCH}.tar > dist/artifacts/k3s-airgap-images-${ARCH}.tar.gz
multus_airgap_image_file='scripts/airgap/image-list-multus.txt'
createTarball ${airgap_image_file} "k3s-airgap-images"
createTarball ${multus_airgap_image_file} "multus-airgap-images"
if [ ${ARCH} = amd64 ]; then
cp "${airgap_image_file}" dist/artifacts/k3s-images.txt
cp "${multus_airgap_image_file}" dist/artifacts/multus-images.txt
fi

0 comments on commit 5073222

Please sign in to comment.