Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle large resource spec being annotated to last-handled-configuration #1087

Open
Bharat23 opened this issue Jan 9, 2024 · 3 comments
Open
Labels
question Further information is requested

Comments

@Bharat23
Copy link

Bharat23 commented Jan 9, 2024

Keywords

annotation, spec, last-handled-configuration, finalizers

Problem

When dealing with a resource that has a large spec, the operator is throwing an error

APIClientError('Pod \"<REDACTED>\" is invalid: metadata.annotations: Too long: must have at most 262144 bytes'

This is due to the fact that it is trying to store the spec in the last-handled-configuration annotation.
It is especially problematic during deletion as it blocks the deletion using the finalizer, which never gets removed. So, how does one go about configuring the operator to handle such a scenario?

Note:
As per the doc Troubleshooting
Restarting the operator doesn't solve the problem. Only running this command unblocks it.

kubectl patch kopfexample kopf-example-1 -p '{"metadata": {"finalizers": []}}' --type merge
@Bharat23 Bharat23 added the question Further information is requested label Jan 9, 2024
@Ramblurr
Copy link

wow this is unfortunate, literally the first lines of code I write with kopf and I hit this error:

import kopf
import logging

@kopf.on.create("secret")
def create_fn(body, **kwargs):
    logging.info(f"A handler is called with body")
kopf run -n database kopftest.py
[2024-02-16 09:26:51,554] kopf._core.engines.a [INFO    ] Initial authentication has been initiated.
[2024-02-16 09:26:51,556] kopf.activities.auth [INFO    ] Activity 'login_with_kubeconfig' succeeded.
[2024-02-16 09:26:51,557] kopf._core.engines.a [INFO    ] Initial authentication has finished.
[2024-02-16 09:26:51,735] root                 [INFO    ] A handler is called with body
[2024-02-16 09:26:51,735] kopf.objects         [INFO    ] [database/sh.helm.release.v1.crunchy-postgres-operator.v1] Handler 'create_fn' succeeded.
[2024-02-16 09:26:51,736] kopf.objects         [INFO    ] [database/sh.helm.release.v1.crunchy-postgres-operator.v1] Creation is processed: 1 succeeded; 0 failed.
[2024-02-16 09:26:51,780] kopf.objects         [ERROR   ] [database/sh.helm.release.v1.crunchy-postgres-operator.v1] Throttling for 1 seconds due to an unexpected error: APIClientError('Secret "sh.helm.release.v1.crunchy-postgres-operator.v1" is invalid: metadata.annotations: Too long: must have at most 262144 bytes', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'Secret "sh.helm.release.v1.crunchy-postgres-operator.v1" is invalid: metadata.annotations: Too long: must have at most 262144 bytes', 'reason': 'Invalid', 'details': {'name': 'sh.helm.release.v1.crunchy-postgres-operator.v1', 'kind': 'Secret', 'causes': [{'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}]}, 'code': 422})
Traceback (most recent call last):
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/errors.py", line 148, in check_response
    response.raise_for_status()
  File "/nix/store/xrgwijximlapr1rkwwwnirf510h3kvc3-python3.11-aiohttp-3.9.1/lib/python3.11/site-packages/aiohttp/client_reqrep.py", line 1059, in raise_for_status
    raise ClientResponseError(
aiohttp.client_exceptions.ClientResponseError: 422, message='Unprocessable Entity', url=URL('https://10.9.9.120:6443/api/v1/namespaces/database/secrets/sh.helm.release.v1.crunchy-postgres-operator.v1')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_core/actions/throttlers.py", line 44, in throttled
    yield should_run
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_core/reactor/processing.py", line 130, in process_resource_event
    applied = await application.apply(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_core/actions/application.py", line 60, in apply
    await patch_and_check(
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_core/actions/application.py", line 131, in patch_and_check
    resulting_body = await patching.patch_obj(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/patching.py", line 47, in patch_obj
    patched_body = await api.patch(
                   ^^^^^^^^^^^^^^^^
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/api.py", line 155, in patch
    response = await request(
               ^^^^^^^^^^^^^^
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/auth.py", line 45, in wrapper
    return await fn(*args, **kwargs, context=context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/api.py", line 85, in request
    await errors.check_response(response)  # but do not parse it!
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/a7b1cwfh2x4i757srdds0jddzwnxv1l2-python3.11-kopf-1.37.1/lib/python3.11/site-packages/kopf/_cogs/clients/errors.py", line 150, in check_response
    raise cls(payload, status=response.status) from e
kopf._cogs.clients.errors.APIClientError: ('Secret "sh.helm.release.v1.crunchy-postgres-operator.v1" is invalid: metadata.annotations: Too long: must have at most 262144 bytes', {'kind': 'Status', 'apiVersion': 'v1', 'metadata': {}, 'status': 'Failure', 'message': 'Secret "sh.helm.release.v1.crunchy-postgres-operator.v1" is invalid: metadata.annotations: Too long: must have at most 262144 bytes', 'reason': 'Invalid', 'details': {'name': 'sh.helm.release.v1.crunchy-postgres-operator.v1', 'kind': 'Secret', 'causes': [{'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}, {'reason': 'FieldValueTooLong', 'message': 'Too long: must have at most 262144 bytes', 'field': 'metadata.annotations'}]}, 'code': 422})
[2024-02-16 09:26:52,781] kopf.objects         [INFO    ] [database/sh.helm.release.v1.crunchy-postgres-operator.v1] Throttling is over. Switching back to normal operations.
^C[2024-02-16 09:26:52,830] kopf._core.reactor.r [INFO    ] Signal SIGINT is received. Operator is stopping.

@nolar
Copy link
Owner

nolar commented Feb 16, 2024

Kopf is designed to work "out of the box" for the majority of cases, i.e. of typically small resources — without any special configuration or external dependencies.

For this, it uses annotations to store its per-object state. However, if Kubernetes prohibits storing big values, Kopf can do nothing here. I am open to suggestions though.

What an operator developer can do, is to configure their own state storage (2 storages of 2 types, actually), as documented here:

It can be e.g. a MySQL/Postgres/any other AWS RDS database, or even a filesystem (if the volume is persisted and shared across multiple instanced of Kopf or on restarts). All of this requires some extra configuration to start an operator and therefore is not the default "out of the box" setup.

There is also the Kopf-provided storage to store the state in the status field — but since a few years ago, it requires that the CRD schema explicitly declares those fields as allowed to store any arbitrary value with x-kubernetes-preserve-unknown-fields: true. So it is not the default either (but it was back then). See the blue note section in the link above.

@baracoder
Copy link

baracoder commented Jun 24, 2024

I have a problem https://github.com/telekom-mms/trivy-dojo-report-operator which uses kopf and works with VulnerabilityReport resources that can get very big. But the reports have a field report.updateTimestamp which would be sufficient to check for changes. Is it possible to configure, which fields get persisted for change detection?
Edit: Would it work if a hash of the resources is saved and used for comparison?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

4 participants