Make segment-cache size configurable and use emptyDir for it

[sbernauer]

Currently we the segment-cache location and size hardcoded to 300gb:

druid-operator/deploy/config-spec/properties.yaml

Line 725 in 4552503

value: "[{\"path\":\"/stackable/var/druid/segment-cache\",\"maxSize\":\"300g\"}]"

Also /stackable/var/druid/segment-cache is not mounted but instead belongs to the container root directory.

We could either put the cache on a disk or in a ramdisk (by using Memory medium for emptydir).
My suggestion is putting the it on disk as this matches the Druid docs

Segments assigned to a Historical process are first stored on the local file system (in a disk cache) and then served by the Historical process

So we need a emptyDir without setting a explicit medium (using disk). We should also set the sizeLimit to the cache size.

• segment-cache resides on emptyDir with correct sizeLimit
• segment-cache size configurable.
• segment-cache free percentage configurable. We default to 5% free percentage and set this as freeSpacePercent Druid attribute.
  CRD proposal

  historicals:
    roleGroups:
      default:
        replicas: 3
        config:
          resources:
            cpu:
              min: '200m'
              max: '4'
            memory:
              limit: '2Gi'
            storage:
              segmentCache: # Enum called e.g. "StorageVolumeConfig" (new)
                freePercentage: 5 # default: 5
                emptyDir: # struct EmptyDirConfig (new)
                  capacity: 10Gi
                  medium: "" # or "Memory"
                # OR
                pvc: # PvcConfig struct
                  capacity: 10Gi
                  storageClass: "ssd"

UPDATE: 04.11.12

Change of plan: since the operator framework doesn't support merging enum types currently, the solution above cannot be implemented. In agreement with others, a new temporary solution is proposed: an implementation with support for emptyDir storage will be made in this repository only. Later, when the framework is updated with the enum merging support, the complete solution from above will be implemented. This proposal is forward compatible with the one above from the user's perspective.

The manifest will look just like this (note the missing PVC configuration:

  historicals:
    roleGroups:
      default:
        replicas: 3
        config:
          resources:
            cpu:
              min: '200m'
              max: '4'
            memory:
              limit: '2Gi'
            storage:
              segmentCache:
                freePercentage: 5 # default: 5
                emptyDir:
                  capacity: 10Gi
                  medium: "" # or "Memory"

[lfrancke]

"on disk" -> is this an externally provided PV?

[sbernauer]

Nope, it's an emptyDir. Normally it's a spinning disk or ssd on the k8s node. A it's a cache there is no point saving it via a pvc

[soenkeliebau]

Integration test for this failed, so should be investigated some more.

https://ci.stackable.tech/job/druid-operator-it-custom/32/

[fhennig]

Maybe run it on AWS EKS 1.22 (nightly runs on that) instead of IONOS 1.24

[fhennig]

I've unassigned myself, since this will go into a bigger phase of "In Progress" again

[razvan]

Blocked by: stackabletech/operator-rs#497