POLL_ADD completion is sent after IORING_OP_POLL_REMOVE returned with 0

[romange]

I use IORING_OP_POLL_ADD method to subsribe to POLLERR | POLLHUP.

I noticed that sometimes I get a completion event even after I call IORING_OP_POLL_REMOVE that cancels the subscription. I verified that IORING_OP_POLL_REMOVE returns 0, i.e. it should have been canceled POLL_ADD request, but it does not happen. My kernel version is 5.15. Did anyone experience similar problems with IORING_OP_POLL_REMOVE?

[axboe]

You always get a completion for a canceled request, for both the cancelation request and the targeted request. That isn't new, it's always been like that. If the event to cancel is racing with the completion, you may get a valid return. You should get -EALREADY for the POLL_REMOVE in this case, and normal completion for the POLL_ADD.

Maybe I'm missing what you are really describing, so can you describe it in a bit more detail? Both in terms of expectations, but also in terms of what you're seeing that differs from that.

[romange]

Jens, sorry - it was a bug in my code. In rare cases the socket is migrated to another thread, thus it being handled by a different io_uring. I did not account for it when calling POLL_REMOVE. Please delete or close this question.

[axboe]

All good, thanks for checking and confirming.

[romange]

My code looks like this:

poll_id = socket->AddPollEvent(POLLERR | POLLHUP,  my_callback);  // calls POLL_ADD underneath 

IoLoop(socket);   //  ... read/write socket processing  here ...* /
   
int result = socket->CancelPoll(poll_id);   // calls POLL_REMOVE, and waits for the completion of cancellation request. returns the result. 
printf("result %d\n", result);

And what I see is:

1. CancelPoll result is zero, i..e POLL_REMOVE returned 0 as if it canceled the target request successfully.
2. After CancelPoll result is printed, I see printings in my_callback function. I see that POLL result is 0x2010 =
   POLLRDHUP | POLLHUP.

My expectation is that my_callback will be called with a negative result -ECANCELED because POLL_REMOVE succeeded with result 0. (pls assume that I do some wrapping around io_uring code to translate io_uring completions to callback invocations).

(Additional info: I have not encountered this problem before. I suspect it happens if I call shutdown(SHUT_RDWR)
on the socket inside IoLoop but I am not sure).

[romange]

Sorry, maybe it's a problem on my end. Let me double-check again if I do not do anything weird with threads.

[axboe]

Closing