Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve] Implement negative cache to avoid repetitive failure #52

Open
aymkhalil opened this issue Jan 17, 2023 · 0 comments
Open

[improve] Implement negative cache to avoid repetitive failure #52

aymkhalil opened this issue Jan 17, 2023 · 0 comments

Comments

@aymkhalil
Copy link
Collaborator

In pulsar transformation, some errors are only detected at runtime. Those could lead the function to be stuck with high CPU utilization until the error is resolved. Some of those errors are sticky in nature an require user intervention. For example:

2023-01-17T20:32:50,787+0000 [**/**/**] WARN  org.apache.pulsar.functions.instance.JavaInstanceRunnable - Encountered exception wh │
│ java.lang.IllegalStateException: Flatten requires non-null schemas!                                                                                │
│     at com.datastax.oss.pulsar.functions.transforms.FlattenStep.validateAvro(FlattenStep.java:68) ~[?:?]                                           │
│     at com.datastax.oss.pulsar.functions.transforms.FlattenStep.process(FlattenStep.java:43) ~[?:?]                                                │
│     at com.datastax.oss.pulsar.functions.transforms.TransformFunction.process(TransformFunction.java:259) ~[?:?]                                   │
│     at com.datastax.oss.pulsar.functions.transforms.TransformFunction.process(TransformFunction.java:249) ~[?:?]                                   │
│     at com.datastax.oss.pulsar.functions.transforms.TransformFunction.process(TransformFunction.java:121) ~[?:?]                                   │
│     at org.apache.pulsar.functions.instance.JavaInstance.handleMessage(JavaInstance.java:94) ~[com.datastax.oss-pulsar-functions-instance-2.10.2.3 │
│     at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:321) ~[com.datastax.oss-pulsar-functions-instance-2 │
│     at java.lang.Thread.run(Thread.java:829) ~[?:?]

Flatten strictly work with schema, and until the user fix their schema or transform, the error will happen non-stop.

@aymkhalil aymkhalil changed the title [fn][improve] Implement negative cache to avoid repetitive failure [improve] Implement negative cache to avoid repetitive failure Feb 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant