-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
CAMEL-20227: camel-kafka - Kafka offset advances when using Pausable … #14745
base: main
Are you sure you want to change the base?
Conversation
…EIP and messages are lost when resumed
🌟 Thank you for your contribution to the Apache Camel project! 🌟 🤖 CI automation will test this PR automatically. 🐫 Apache Camel Committers, please review the following items:
|
Argh yeah this test is a bit wrong as you want to be able to consume all 15 messages, but in the middle the consumer is paused |
…EIP and messages are lost when resumed
…EIP and messages are lost when resumed
okay I improved this so the pause and resume will not lose messages and that the pause is waiting a poll cycle, otherwise it keeps looping asap which is not desireable (eat cpu and goes too fast) |
Cool. It's a holiday here, but I will try to take a look later today |
At first it wasn't very clear to me how the user would be losing messages after the pause. The KafkaConsumer documentation says that But looking more closely to the reproducer, I think I kinda understand the problem ... It's in part because of a misuse of the API (and, poor documentation for this particular feature). In particular, our documentation states: "The pausable EIP is meant to be used as a support mechanism when there is an exception somewhere in the route that prevents the exchange from being processed. More specifically, the check called by the pausable EIP should be used to test for transient conditions preventing the exchange from being processed." Specifically, the trigger for those transient problems should be pulled by an exception, so that we have a failure in the processing result. Otherwise, we never cause the pause, thus leading to them losing the messages. So, to make it short: the reproducer from the user is wrong. |
That said, I think we do have a room for improvement on the API. I am not sure if we should be allowing the code to resume the assignment if pause hasn't been called. Maybe we could throw an exception explaining that this is a misuse of the API, and trying to force a resume without an underlying transient problem is not supported. |
Yeah okay that can be better explain in the docs / javadoc / and the API is using a predicate so an exception is not thrown. If the intend is to pause/resume kafka via some logic then use |
@davsclaus actually, this is different from pausing the route per se. The idea for the Nonetheless, let's see if we can brainstorm some improvements to this once you are back. There are some specific requirements for pausing the KafkaConsumer (such as: it must be done from the same thread as the one calling 'poll'), but if we could do this using the |
…EIP and messages are lost when resumed
Description
Target
camel-3.x
, whereas Camel 4 uses themain
branch)Tracking
Apache Camel coding standards and style
mvn clean install -DskipTests
locally and I have committed all auto-generated changes