io_uring_prep_send_bundle (IORING_RECVSEND_BUNDLE) : buffer ring updates, "done" buffers and zero copy

[nigoroll]

I am trying to understand how to make best use of the new-ish bundle feature. I have three questions which I am having a hard time figuring out answers for from the documentation and example/test code.

Buffer ring updates?

io_uring_prep_send(3) states that a bundle send "keeps sending from that buffer group until it runs out of buffers.", but does this imply that more buffers can be added while the send is in flight?

"done" buffers

io_uring_prep_send(3) states that "A single CQE is posted for the send", yet recvsend_bundle-inc.c checks for IORING_CQE_F_MORE, which seems to suggest that multiple CQEs were returned? Is this a documentation error? If not, it seems I do not understand the test code? If in fact only a single CQE is returned, is there any other way to learn about buffers which io_uring no longer needs for a send operation?

zero copy

is there an option to use zero copy also with send bundles?

Thank you,

Nils

[axboe]

io_uring_prep_send(3) states that a bundle send "keeps sending from that buffer group until it runs out of buffers.", but does this imply that more buffers can be added while the send is in flight?

This ties in with the IORING_CQE_F_MORE as well. When bundle is set, if:

1. There are more buffers to send, and
2. The current send was successful

then another attempt will be made. When this happens. IORING_CQE_F_MORE will be set in the posted CQE, as another attempt at sending will be made. This is identical to other kinds of multishot requests, if more CQEs are expected for this request, then the MORE flag will be set. However, a send bundle isn't persistent in the sense that it stays around and triggers again when there's space in the buffer. It'll only last for as many sends as is required to either hit a full condition on the socket, or we run out of buffers to send. So while you could theoretically have someone added buffers WHILE your send is in progress and before it finishes (and hence checks for more buffers), I'd say that sounds pretty unlikely. And normally those buffers are probably added by the ring owner anyway, which is doing the send itself.

I think this explains both your questions? For zero-copy, currently no way to use them with bundles. That could be added, just haven't been done yet. Would be useful...

[nigoroll]

I think this explains both your questions?

Yes, and more, thank you very much. In my particular case, at first sight the bundle facility sounded like maintaining a separate state was avoidable, but in particular a send bundle isn't persistent in the sense that it stays around and triggers again when there's space in the buffer helped a lot, because you clarified that indeed the ring owner needs to be prepared to "resume" at any point, depending on the returned result.