performance characteristics of IORING_OP_{SEND/WRITE/SENDMSG}

[romange]

I've used IORING_OP_{SEND,RECV}MSG so far just because they are the most powerful API calls for socket communication.

Now, based on discussions here, I learned that there are some performance implications when using vectored calls.
If we are discussing this I would like to understand which calls are "heavier" than others.

1. Should I prefer IORING_OP_SEND over IORING_OP_WRITE when using socket writes?
2. Should I default to flattening small vectors with small msg_iovlen lengths (say overall sum less than K bytes) into a single buffer?
3. Is it also true for RECV/READ counterparts as well?

[axboe]

1. Yes, use the send/recv variants over write/read if you're dealing with a socket. Various performance features or optimizations are also only available through the network specific variants.
2. Non-vectored requests are always more efficient. That's true even for inline execution, but only marginally so. It's specifically true for requests that need to go async (eg arm internal poll to retry when data/space is available, and to a lesser extent io-wq punting).
3. Yes