Seeking example of applying io_uring_prep_cancel properly

[tofes]

On a single io_uring, I'm in the process of testing cancel I/O but keep getting errno 2 (ENOENT) on cancel request, would really appreciate if anyone can show me what is the proper way of using io_uring_prep_cancel(). I'm on kernel 5.5 which supports IORING_OP_ASYNC_CANCEL (14) opcode.

In my test, I did a read request first from a client to a server, since the server is not writing anything I will get an errno 11 (EAGAIN) back which is expected. The corresponding SQE and CQE are shown below:

sqe Read Request
{
opcode : 1
flags : 4
ioprio : 0
fd : 51
off : 0
addr : 0x709d15d2d2b0
len : 1
user_data : 0x70fe02aa88d8 (1)
}

cqe wait
{
user_data : 0x70fe02aa88d8 (1)
res : -11
}

Then I issued a cancel request of the above read request as below before submitting the request:
(The userdata is the void * which has the same address passed into the to be cancelled read request.)

    io_uring_prep_cancel(sqe, userdata, 0);
    io_uring_sqe_set_data(sqe, userdata);

I used the io_uring_sqe_set_data() because without using it the sqe->user_data value is 0, it appears that io_uring_prep_cancel() does not set correct user_data value, it sets the sqe->addr instead.

The corresponding SQE and CQE for the cancel request are shown below:

sqe Cancel Request
{
opcode : 14
flags : 0
ioprio : 0
fd : -1
off : 0
addr : 0x70fe02aa88d8
len : 0
user_data : 0x70fe02aa88d8 (2)
}

cqe wait
{
user_data : 0x70fe02aa88d8 (2)
res : -2
}

Also, I had IOSQE_ASYNC and IOSQE_IO_LINK sqe flags set, will they be a problem in this test case?

Why would the result of the cancel request be not found ENOENT in this case instead of 0 or EALREADY?

[axboe]

If I'm reading this right, you issue a request, and that completes with -11 as cqe->res. That means the request is no longer active, it has completed. So your cancelation will return -ENOENT, as no such request exists (it already completed).

[tofes]

hmm... does this mean I can only cancel before the io_uring_wait_cqe returned 0? Not sure how I can create such a case for unit testing a cancel request.

[axboe]

It only makes sense to attempt cancelation of a request that is still in progress. Since the read you're trying to cancel already completed, not quite sure what you expect the cancelation request to do? The request is already completed and gone.

[axboe]

This has nothing to do with io_uring_wait_cqe(). If it fails with -11, then it has already completed before you even call io_uring_wait_cqe().

As a general comment, you really should move to a newer base. 5.5 is really ancient, particularly in io_uring terms.

[tofes]

I'm understanding that if the operation is already completed, its result is put in the CQ ring before I call io_uring_wait_cqe(), so issuing a cancel to the SQ ring is not going to cancel this operation. hmm... I want to test if my cancel operation works correctly, so I guess this covers the "if not found" case, but for the "found" case, I will have to get the cancel operation result back before the readv/writev result is available. Is there a way to be certain the io_uring has not completed the readv/writev yet so I can issue the cancel request and quickly check its result?

The older kernel version is for backward compatibility for our product :). I will use newer kernel version as I go.

[axboe]

I'm guessing your socket is marked nonblock, just use a blocking socket?

[tofes]

Thanks for the acute observation, a socket is usually blocking by default but not in our system. Also thanks for the patient explanation and timely engagement!
I updated the socket to be blocking, and cancelling a blocked readv operation resulted in EALREADY (better found than not found ;)). I also see an EINTR in the result for the same user_data when calling io_uring_wait_cqe() again, assume it is the original readv operation's result.
A note/observation, the Cancel operation and the original operation being cancelled use the same user_data, so when looking at the CQE it's not clear which CQE is for which SQE. Maybe there is a way to differentiate them? I can tell them apart this time using the errno.

[axboe]

Those sound like expected results - you get -EALREADY since it was already in progress, and the original request errors with -EINTR because it was interrupted.

A note/observation, the Cancel operation and the original operation being cancelled use the same user_data, so when looking at the CQE it's not clear which CQE is for which SQE. Maybe there is a way to differentiate them? I can tell them apart this time using the errno.

Well, you're the one that picked user_data for the cancelation request, so just pick a different value?

[tofes]

I see, the io_uring_prep_cancel() used sqe->addr to tell which operation it was cancelling, so we can still use user_data to tell which sqe operation the cqe result was for :).

[axboe]

Yes, sqe->addr is the user_data to look for and cancel, you can still put whatever you want in sqe->user_data for the cancelation request.

[ioquatix]

Is there a difference between ECANCELED or EINTR?

[axboe]

Generally ECANCELED means that it was actively canceled, either through ring shutdown or someone issuing a cancelation request for the req. EINTR generally means that processing it was interrupted by a signal.

[axboe]

If io-wq is handling the request, we do signal it to interrupt execution of the request. But that should yield EALREADY on the cancelation side.