How do I add a configmap to the volume of a sidecar?

[SandraGH5]

Adam Salow:
I'm trying to add a configmap to the volume of a sidecar. If I try like this:

sidecar_container = generated_pb2.Container(
        name="sidecar-proxy",
        image="myproxy",
        volume=generated_pb2.Volume(
            name="proxy-config",
            configMap=generated_pb2.ConfigMap(
                data={"proxy.yaml": json.dumps(proxy_yaml, indent=2).encode("utf-8")}
            ),
        ),
    )

I get the error Protocol message Volume has no "configMap" field.

If I try like this:

sidecar_container = V1Container(
        name="sidecar-proxy",
        image="myproxy",
        volume=V1Volume(
            name="proxy-config",
            config_map=V1ConfigMap(
                data={"proxy.yaml": proxy_yaml}
            ),
        ),
    )

flytepropeller gives the following error and my wf gets stuck in the queued state.

{"json":{"exec_id":"pn9f5pdl21","node":"n0","ns":"flyte-test-development","res_ver":"76563861","routine":"worker-5","src":"handler.go:359","tasktype":"sidecar","wf":"flyte-test-adam:development:workflows.proxy_sidecar.ProxySideCarWorkflow"},"level":"warning","msg":"Runtime error from plugin [sidecar]. Error: [BadTaskSpecification] Unable to unmarshal task custom [\u003cnil\u003e], Err: [json: cannot unmarshal object into Go struct field Container.containers.volumeMounts of type []v1.VolumeMount]","ts":"2021-06-10T21:08:31Z"}
{"json":{"exec_id":"pn9f5pdl21","node":"n0","ns":"flyte-test-development","res_ver":"76563861","routine":"worker-5","src":"handler.go:218","wf":"flyte-test-adam:development:workflows.proxy_sidecar.ProxySideCarWorkflow"},"level":"error","msg":"handling parent node failed with error: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [sidecar]: [BadTaskSpecification] Unable to unmarshal task custom [\u003cnil\u003e], Err: [json: cannot unmarshal object into Go struct field Container.containers.volumeMounts of type []v1.VolumeMount]","ts":"2021-06-10T21:08:31Z"}
{"json":{"exec_id":"pn9f5pdl21","node":"n0","ns":"flyte-test-development","res_ver":"76563861","routine":"worker-5","src":"executor.go:363","wf":"flyte-test-adam:development:workflows.proxy_sidecar.ProxySideCarWorkflow"},"level":"error","msg":"failed Execute for node. Error: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [sidecar]: [BadTaskSpecification] Unable to unmarshal task custom [\u003cnil\u003e], Err: [json: cannot unmarshal object into Go struct field Container.containers.volumeMounts of type []v1.VolumeMount]","ts":"2021-06-10T21:08:31Z"}
{"json":{"exec_id":"pn9f5pdl21","ns":"flyte-test-development","res_ver":"76563861","routine":"worker-5","src":"executor.go:379","wf":"flyte-test-adam:development:workflows.proxy_sidecar.ProxySideCarWorkflow"},"level":"warning","msg":"Error in handling running workflow [failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [sidecar]: [BadTaskSpecification] Unable to unmarshal task custom [\u003cnil\u003e], Err: [json: cannot unmarshal object into Go struct field Container.containers.volumeMounts of type []v1.VolumeMount]]","ts":"2021-06-10T21:08:31Z"}
{"json":{"exec_id":"pn9f5pdl21","ns":"flyte-test-development","res_ver":"76563861","routine":"worker-5","src":"executor.go:380","wf":"flyte-test-adam:development:workflows.proxy_sidecar.ProxySideCarWorkflow"},"level":"info","msg":"Handling Workflow [pn9f5pdl21] Done","ts":"2021-06-10T21:08:31Z"}

Flytekit version: 0.19.1

Code Sample

import os
import time
from k8s.io.api.core.v1 import generated_pb2
from flytekit import task, workflow
from flytekitplugins.pod import Pod
from kubernetes.client.models import (
    V1Capabilities,
    V1ConfigMap,
    V1Container,
    V1PodSpec,
    V1ResourceRequirements,
    V1SecurityContext,
    V1Volume,
    V1VolumeMount,
)
def generate_pod_spec_for_task():
    # Primary containers do not require us to specify an image, the default image built for flyte tasks will get used.
    primary_container = V1Container(name="primary")
    sidecar_volume = V1Volume(
        name="proxy-config", config_map=V1ConfigMap(data={"envoy.yaml": {}})
    )
    sidecar_container = V1Container(
        name="sidecar-proxy",
        image="envoyproxy/envoy:v1.18.3",
        volume_mounts=V1VolumeMount(
            name="proxy-config",
            mount_path="/data",
        ),
    )
    sidecar_container.args = ["envoy", "-c", "/config/envoy.yaml", "-l", "debug"]
    resources = V1ResourceRequirements(
        requests={"cpu": ".5", "memory": "500Mi"},
        limits={"cpu": ".5", "memory": "500Mi"},
    )
    primary_container.resources = resources
    sidecar_container.resources = resources
    pod_spec = V1PodSpec(
        containers=[primary_container, sidecar_container],
        volumes=sidecar_volume
    )
    return pod_spec

Although Pod tasks for the most part allow you to customize kubernetes container attributes, you can still use flyte directives to specify resources and even the image. The default image built for flyte tasks will get used unless you specify the `container_image` task attribute.

@task(
    task_config=Pod(
        pod_spec=generate_pod_spec_for_task(), primary_container_name="primary"
    ),
)
def sidecar_task() -> str:
    # The code defined in this task will get injected into the primary container.
    time.sleep(1000)
@workflow
def ProxySideCarWorkflow() -> str:
    s = sidecar_task()
    return s
if __name__ == "__main__":
    ProxySideCarWorkflow()

[SandraGH5]

katrina:
Just to double-check, are you generating the full pod spec similar to the example here

The protobuf objects in generated_pb2 are legacy and no longer the recommended way. It is better to simply use the k8s python client.

You need to specify them as a list - so change:

volume_mounts=V1VolumeMount(
            name="proxy-config",
            mount_path="/data",
        ),

to:

volume_mounts=[V1VolumeMount(
            name="proxy-config",
            mount_path="/data",
        )],

Also, volume definition might be wrong as well here

you want V1ConfigMapVolumeSource: https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1ConfigMapVolumeSource.md

see: https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Volume.md

(This thread has been referenced from Slack)
@katrogan
@jeevb
@wild-endeavor