containerOverrides missing in 0.2.* helm chart thanos CRD

[NissesSenap]

Describe the bug

The config to be able to override container configuration for the different parts of thanos is missing.

Steps to reproduce the issue:

Install 0.1.1 helm chart

helm upgrade -i thanos-operator --namespace monitor banzaicloud-stable/thanos-operator --version 0.1.1

Apply a Thanos CR that looks something like this:

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: Thanos
metadata:
  name: thanos-sample
spec:
  queryDiscovery: true
  query: {}
  rule:
    containerOverrides:
      volumeMounts:
        - name: thanos-data
          mountPath: ./data
        - mountPath: /etc/config/
          name: objectstore-secret
          readOnly: true
    workloadOverrides:
      volumes:
        - name: thanos-data
          emptyDir: {}
        - name: objectstore-secret
          secret:
            defaultMode: 420
            secretName: thanos-objstore-config
  storeGateway:
    containerOverrides:
      volumeMounts:
        - name: thanos-data
          mountPath: ./data
        - mountPath: /etc/config/
          name: objectstore-secret
          readOnly: true
    workloadOverrides:
      volumes:
        - name: thanos-data
          emptyDir: {}
        - name: objectstore-secret
          secret:
            defaultMode: 420
            secretName: thanos-objstore-config

Upgrade to helm chart 0.2.1

helm upgrade thanos-operator --namespace monitor banzaicloud-stable/thanos-operator --version 0.2.1
k apply -f https://raw.githubusercontent.com/banzaicloud/thanos-operator/chart/thanos-operator/0.2.1/charts/thanos-operator/crds/monitoring.banzaicloud.io_objectstores.yaml
k apply -f https://raw.githubusercontent.com/banzaicloud/thanos-operator/chart/thanos-operator/0.2.1/charts/thanos-operator/crds/monitoring.banzaicloud.io_receivers.yaml
k apply -f https://raw.githubusercontent.com/banzaicloud/thanos-operator/chart/thanos-operator/0.2.1/charts/thanos-operator/crds/monitoring.banzaicloud.io_storeendpoints.yaml
k apply -f https://raw.githubusercontent.com/banzaicloud/thanos-operator/chart/thanos-operator/0.2.1/charts/thanos-operator/crds/monitoring.banzaicloud.io_thanos.yaml
k apply -f https://raw.githubusercontent.com/banzaicloud/thanos-operator/chart/thanos-operator/0.2.1/charts/thanos-operator/crds/monitoring.banzaicloud.io_thanosendpoints.yaml
k apply -f https://raw.githubusercontent.com/banzaicloud/thanos-operator/chart/thanos-operator/0.2.1/charts/thanos-operator/crds/monitoring.banzaicloud.io_thanospeers.yaml

Expected behavior

Everything keeps on working like in 0.1.1

Screenshots

Additional context

Instead i get the following crashLoopBack due to I'm not able to write to disk due to a muttating webhooks that applies

    securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - NET_RAW
      readOnlyRootFilesystem: true

Thanks to this I get similar error in both:
statefulsets.apps thanos-sample-storeendpoint-receiver-rule
and
deployment thanos-sample-storeendpoint-receiver-store

➜ k logs thanos-sample-storeendpoint-receiver-store-868b54d476-6qznc              
level=info ts=2021-04-13T08:31:15.0960554Z caller=main.go:152 msg="Tracing will be disabled"
level=info ts=2021-04-13T08:31:15.0962121Z caller=factory.go:46 msg="loading bucket configuration"
level=info ts=2021-04-13T08:31:15.1869422Z caller=inmemory.go:172 msg="created in-memory index cache" maxItemSizeBytes=131072000 maxSizeBytes=262144000 maxItems=maxInt
level=error ts=2021-04-13T08:31:15.1872223Z caller=main.go:186 err="mkdir data: read-only file system\nmeta fetcher\nmain.runStore\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/store.go:280\nmain.registerStore.func1\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/store.go:119\nmain.main\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:184\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373\npreparing store command failed\nmain.main\n\t/go/src/github.com/thanos-io/thanos/cmd/thanos/main.go:186\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1373"

[pepov]

We switched to a new way of configuration overrides, which is in this case can be found under spec.storeGateway.deploymentOverrides and spec.rule.statefulsetOverrides. You would want to add override configuration just as you would configure a deployment or a statefulset:

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: Thanos
metadata:
  name: thanos-sample
spec:
  queryDiscovery: true
  query: {}
  rule:
    statefulsetOverrides:
      spec:
        template:
          spec:
            containers:
            - name: "rule"
              volumeMounts:
              - name: thanos-data
                mountPath: ./data
              - mountPath: /etc/config/
                name: objectstore-secret
                readOnly: true
            volumes:
            - name: thanos-data
              emptyDir: {}
            - name: objectstore-secret
              secret:
                defaultMode: 420
                secretName: thanos-objstore-config
  storeGateway:
      spec:
        template:
          spec:
            containers:
            - name: "store"
              volumeMounts:
              - name: thanos-data
                mountPath: ./data
              - mountPath: /etc/config/
                name: objectstore-secret
                readOnly: true
            volumes:
            - name: thanos-data
              emptyDir: {}
            - name: objectstore-secret
              secret:
                defaultMode: 420
                secretName: thanos-objstore-config

please let me know if this works for you now

[NissesSenap]

So I have been trying to get this to work for the last hour but not luck.
A small thing you missed deploymentOverrides in your example. Thanks btw for taking the time to converting my old yaml for me.

But my current problem is that it dosen't seem like the volumeMounts respects
readOnly: false on the volumeMounts.

Here you can see my current yaml bellow:

apiVersion: monitoring.banzaicloud.io/v1alpha1
kind: Thanos
metadata:
  name: thanos-sample
spec:
  queryDiscovery: true
  query: {}
  rule:
    statefulsetOverrides:
      spec:
        template:
          spec:
            containers:
              - name: "rule"
                volumeMounts:
                  - name: thanos-data
                    mountPath: ./data
                    readOnly: false
                  - mountPath: /etc/config/
                    name: objectstore-secret
                    readOnly: true
            volumes:
              - name: objectstore-secret
                secret:
                  defaultMode: 420
                  secretName: thanos-objstore-config
              - name: thanos-data
                emptyDir: {}
  storeGateway:
    deploymentOverrides:
      spec:
        template:
          spec:
            containers:
              - name: "store"
                volumeMounts:
                  - name: thanos-data
                    mountPath: ./data
                    readOnly: false
                  - mountPath: /etc/config/
                    name: objectstore-secret
                    readOnly: true
            volumes:
              - name: objectstore-secret
                secret:
                  defaultMode: 420
                  secretName: thanos-objstore-config
              - name: thanos-data
                emptyDir: {}

Here is a small snipet on how it looks in the pods and statefulSets:

        volumeMounts:
        - mountPath: ./data
          name: thanos-data
          readOnly: true
        - mountPath: /etc/config/
          name: objectstore-secret
          readOnly: true

and bellow you can find the entire statefulset.

➜ k get statefulsets.apps thanos-sample-storeendpoint-receiver-rule -o yaml      
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app.kubernetes.io/managed-by: thanos-sample
    app.kubernetes.io/name: rule
    monitoring.banzaicloud.io/storeendpoint: storeendpoint-receiver
  name: thanos-sample-storeendpoint-receiver-rule
  namespace: monitor
  ownerReferences:
  - apiVersion: monitoring.banzaicloud.io/v1alpha1
    controller: true
    kind: StoreEndpoint
    name: storeendpoint-receiver
    uid: d9b27d53-d963-4a5e-a71a-895a4e3070b0
  resourceVersion: "18908366"
  selfLink: /apis/apps/v1/namespaces/monitor/statefulsets/thanos-sample-storeendpoint-receiver-rule
  uid: 35165fa2-58f9-4ea9-9d57-8cc926b22513
spec:
  podManagementPolicy: OrderedReady
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/managed-by: thanos-sample
      app.kubernetes.io/name: rule
      monitoring.banzaicloud.io/storeendpoint: storeendpoint-receiver
  serviceName: ""
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/managed-by: thanos-sample
        app.kubernetes.io/name: rule
        monitoring.banzaicloud.io/storeendpoint: storeendpoint-receiver
      name: thanos-sample-storeendpoint-receiver-rule
      namespace: monitor
      ownerReferences:
      - apiVersion: monitoring.banzaicloud.io/v1alpha1
        controller: true
        kind: StoreEndpoint
        name: storeendpoint-receiver
        uid: d9b27d53-d963-4a5e-a71a-895a4e3070b0
    spec:
      containers:
      - args:
        - rule
        - --objstore.config-file=/etc/config/thanos.yaml
        - --log.level=info
        - --http-address=0.0.0.0:10902
        - --grpc-address=0.0.0.0:10901
        - --query=thanos-sample-query.monitor.svc:10902
        image: quay.io/thanos/thanos:v0.19.0-rc.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 2
          httpGet:
            path: /-/healthy
            port: 10902
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
        name: rule
        ports:
        - containerPort: 10902
          name: http
          protocol: TCP
        - containerPort: 10901
          name: grpc
          protocol: TCP
        readinessProbe:
          failureThreshold: 2
          httpGet:
            path: /-/ready
            port: 10902
            scheme: HTTP
          initialDelaySeconds: 5
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 5
        resources: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: ./data
          name: thanos-data
          readOnly: true
        - mountPath: /etc/config/
          name: objectstore-secret
          readOnly: true
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: objectstore-secret
        secret:
          defaultMode: 420
          secretName: thanos-objstore-config
      - emptyDir: {}
        name: thanos-data
  updateStrategy:
    rollingUpdate:
      partition: 0
    type: RollingUpdate
status:
  collisionCount: 0
  currentReplicas: 1
  currentRevision: thanos-sample-storeendpoint-receiver-rule-6bbbccb45b
  observedGeneration: 1
  replicas: 1
  updateRevision: thanos-sample-storeendpoint-receiver-rule-6bbbccb45b
  updatedReplicas: 1

As you can see the operator seems to set my volume to readOnly any way.
Any ideas why this might be?

[pepov]

Yes, unfortunately merging volumes and volumemounts does not work the way expected. In this case the original volume and volumemount is objectstore-secret which is overriden by the thanos-data volume defined in the override spec based on it's index. You have to put objectstore-secret volume and volumemount first and thanos-data second to make this work.