[Bug] Copy secret into every namespace. #10708

eitah · 2024-07-19T15:59:11Z

Kyverno Version

1.12

Kubernetes Version

1.29

Kubernetes Platform

GKE

Description

Im trying to write a generate rule inspired by sync secrets which will take an incoming secret and replicate it everywhere. The issue I'm having is twofold

I can't use a clone policy targeting the incoming secret because of #8025
Instead I tried using a cloneList policy selecting all secrets in the originating namespace and copying them, but the rule does not copy the incoming request object, I assume because it doesn't exist yet.

A note: I see in the logs it can't reconcile the namespaces array into a single namespace, I think because I'm not passing it in correctly. I've tried {{ namespaces }} without the brackets, etc, but nothing seems to work. Is there an easy way to express "put this in every namespace you see"?

Steps to reproduce

Add a clone secrets policy per the below

  - name: sync-new-label-secrets
    match:
      any:
      - resources:
          kinds:
            - Secret
          operations:
            - CREATE
          selector:
            matchLabels:
              label-key: label-value
    context:
      - name: namespaces
        apiCall:
          urlPath: "/apis/networking.k8s.io/v1/Namespaces"
          jmesPath: "items[?metadata.labels.\"label-key\"=='label-value'].metadata.name"
    generate:
      apiVersion: v1
      namespace: "{{ namespaces[] }}"
      synchronize: true
      cloneList:
        namespace: origin-namespace
        kinds:
          - v1/Secret
        selector:
          matchLabels:
            label-key: label-value

Create a secret in origin-namespace
The secret is not synced to destination namespaces with the right labels.

Expected behavior

I would expect the secret to have been synced.

Screenshots

No response

Kyverno logs

^[[Akyverno-admission-controller-776987899-dfmzz kyverno 2024-07-19T15:55:24Z	INFO	setup.cluster-policy	logging/controller.go:45	resource added	{"type": "ClusterPolicy", "name": "sync-secrets"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	INFO	PolicyController	policy/policy_controller.go:181	policy created	{"uid": "366bfd08-0134-4490-904d-1837756679bf", "kind": "ClusterPolicy", "namespace": "", "name": "sync-secrets"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	INFO	PolicyController.handleMutate.sync-secrets	policy/mutate.go:15	update URs on policy event
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	INFO	PolicyController.handleGenerate.sync-secrets	policy/generate.go:21	update URs on policy event
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	INFO	PolicyController	policy/policy_controller.go:421	creating new UR for generate
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-d4bkn", "policy": "sync-secrets", "resource": "v1/Namespace//concourse-shared-secrets", "ur": "ur-d4bkn", "resourceVersion": "538873"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "ur": "ur-t7jbc", "resourceVersion": "538878"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\"],\"synchronize\":true},\"m|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "ur": "ur-t7jbc", "resourceVersion": "538881"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\"],\"synchronize\":true},\"m|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "ur": "ur-t7jbc", "resourceVersion": "538884"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\"],\"synchronize\":true},\"m|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:24Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "ur": "ur-t7jbc", "resourceVersion": "538887"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\"],\"synchronize\":true},\"m|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-8htxx", "policy": "sync-secrets", "resource": "v1/Namespace//new-ns01", "ur": "ur-8htxx", "resourceVersion": "538894"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "ur": "ur-t7jbc", "resourceVersion": "538895"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-t7jbc", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/foo", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\",\"new-ns01\"],\"synchroniz|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	INFO	background	generate/generate.go:458	created generate target resource	{"name": "ur-8htxx", "policy": "sync-secrets", "resource": "v1/Namespace//new-ns01", "target": "//new-ns01/"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "ur": "ur-wmrvj", "resourceVersion": "538904"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\",\"new-ns01\"],\"synchroniz|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "ur": "ur-wmrvj", "resourceVersion": "538907"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\",\"new-ns01\"],\"synchroniz|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "ur": "ur-wmrvj", "resourceVersion": "538910"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\",\"new-ns01\"],\"synchroniz|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "ur": "ur-wmrvj", "resourceVersion": "538913"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\",\"new-ns01\"],\"synchroniz|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "ur": "ur-wmrvj", "resourceVersion": "538916"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:25Z	ERROR	background	generate/generate.go:384	variable substitution failed for rule	{"name": "ur-wmrvj", "policy": "sync-secrets", "resource": "v1/Secret/concourse-shared-secrets/bar", "rule": "", "error": "v1.Rule.Generation: v1.Generation.ResourceSpec: Namespace: ReadString: expects \" or n, but found [, error found in kyverno/policies#10 byte of ...|mespace\":[\"concourse|..., bigger context ...|es.io/component\":\"concourse-team\"}}},\"namespace\":[\"concourse-shared-secrets\",\"new-ns01\"],\"synchroniz|..."}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:41Z	INFO	background	generate/generate.go:101	start processing UR	{"name": "ur-fqgd9", "policy": "sync-secrets", "resource": "v1/Namespace//new-ns01", "ur": "ur-fqgd9", "resourceVersion": "538975"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:55:52Z	INFO	PolicyController	policy/policy_controller.go:228	policy deleted	{"uid": "366bfd08-0134-4490-904d-1837756679bf", "kind": "ClusterPolicy", "namespace": "", "name": "sync-secrets"}
kyverno-admission-controller-776987899-dfmzz kyverno 2024-07-19T15:55:52Z	INFO	setup.cluster-policy	logging/controller.go:68	resource deleted	{"type": "ClusterPolicy", "name": "sync-secrets"}
kyverno-background-controller-86b9f95c96-c8f2q controller 2024-07-19T15:57:00Z	INFO	PolicyController.forceReconciliation	policy/policy_controller.go:366	reconciling generate and mutateExisting policies	{"scan interval": "1h0m0s"}

Slack discussion

No response

Troubleshooting

I have read and followed the documentation AND the troubleshooting guide.
I have searched other issues in this repository and mine is not recorded.

The text was updated successfully, but these errors were encountered:

welcome · 2024-07-23T11:22:07Z

Thanks for opening your first issue here! Be sure to follow the issue template!

realshuting · 2024-07-23T11:24:48Z

Hi @eitah - here's a similar request to copy a resource into multiple namespaces, you probably need the foreach, see:

kyverno/KDP#52 (comment)

realshuting · 2024-07-23T11:26:57Z

Related to #3542.

realshuting · 2024-08-01T07:51:39Z

With the proposed KDP, I craft a policy to achieve this use case;

spec:
  rules:
  - name: generate-network-policies
    match:
      any:
      - resources:
          kinds:
            - Secret
          operations:
            - CREATE
          selector:
            matchLabels:
              label-key: label-value
    context:
      - name: namespaces
        apiCall:
          urlPath: "/apis/networking.k8s.io/v1/Namespaces"
          jmesPath: "items[?metadata.labels.\"label-key\"=='label-value'].metadata.name"
    generate:
      foreach:
        - list: "{{namespaces}}"
          dataList:
            - metadata: "{{request.object.metadata}}"
              data: "{{request.object.data}}"

eitah · 2024-08-19T15:42:38Z

Thank you! When is Kyverno 13 expected? I saw from this page Milestone 85 is due, but that can't be the entire release, can it? https://github.com/kyverno/kyverno/milestone/85

eitah added the bug Something isn't working label Jul 19, 2024

realshuting transferred this issue from kyverno/policies Jul 23, 2024

realshuting added enhancement New feature or request generation Issues pertaining to the generate ability. end user This label is used to track the issue that is raised by the end user. and removed bug Something isn't working labels Jul 23, 2024

realshuting added this to the Kyverno Release 1.13.0 milestone Jul 23, 2024

realshuting added the release-high High issues which SHOULD be addressed in the specified milestone. These may get bumped. label Jul 24, 2024

realshuting self-assigned this Jul 24, 2024

realshuting modified the milestones: Kyverno Release 1.13.0, Kyverno Release 1.14.0 Oct 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] Copy secret into every namespace. #10708

[Bug] Copy secret into every namespace. #10708

eitah commented Jul 19, 2024

welcome bot commented Jul 23, 2024

realshuting commented Jul 23, 2024

realshuting commented Jul 23, 2024

realshuting commented Aug 1, 2024

eitah commented Aug 19, 2024

[Bug] Copy secret into every namespace. #10708

[Bug] Copy secret into every namespace. #10708

Comments

eitah commented Jul 19, 2024

Kyverno Version

Kubernetes Version

Kubernetes Platform

Description

Steps to reproduce

Expected behavior

Screenshots

Kyverno logs

Slack discussion

Troubleshooting

welcome bot commented Jul 23, 2024

realshuting commented Jul 23, 2024

realshuting commented Jul 23, 2024

realshuting commented Aug 1, 2024

eitah commented Aug 19, 2024