io_uring_prep_send_zc() performing a little worst than io_uring_prep_send()

[jmillan]

I'm trying out io_uring_prep_send_zc() and comparing it with io_uring_prep_send() in my server.

In order to enable zero copy I do as follows:

Enable SO_ZEROCOPY in the socket file descriptor

// Enable zero-copy in the file descriptor.
int on = 1;

err = setsockopt(this->fd, SOL_SOCKET, SO_ZEROCOPY, &on, sizeof(on));

Register buffers into io_uring

 // Register buffers.
for (size_t i{ 0 }; i < DepLibUring::QueueDepth; ++i)
{
 this->iov[i].iov_base = malloc(1500);
 this->iov[i].iov_len = 1500;
}

err = io_uring_register_buffers(std::addressof(this->ring), this->iov, DepLibUring::QueueDepth);

Prepare send

io_uring_prep_send_zc(sqe, sockfd, this->iov[idx].iov_base,  this->iov[idx].iov_len, 0, 0);
io_uring_prep_send_set_addr(sqe, addr, addrlen);

sqe->ioprio |= IORING_RECVSEND_FIXED_BUF;
sqe->buf_index = idx;

I'm handling notifications accordingly, making the buffers available when receiving the IORING_CQE_F_NOTIF in cqe-flags.

Tracing the process with and without zero copy I see that there is a slightly more CPU usage when using ZC. These are some observations comparing the traces:

• There is a function io_import_fixed being called, taking a little time. That is no present in NO_ZC. It's expected.
• There is a function io_send_zc_prep being called, takes a little more time than the analogous io_sendmsg_prep.
• ip_append_data takes considerably less time in ZC in comparison with NO_ZC. I can imagine this is where CZ shines.
• ip_send_skb function:
  • __ip_local_out seems to take the same time in both cases.
  • ip_finish_output takes more time in ZC than in NO_ZC.

From these observations I see that the benefit of using ZC is in ip_append_data, but for some reason it takes more time than NO_ZC in the other mentioned functions, so in overall there is no benefit.

In my tests I'm sending around 200K UDP (audio and video) packets through a socket.

I would definitely like to understand this numbers better, as everything pointed that ZC would show improvements compared to NO_ZC.

Here the most significant part of the traces flamegraph for a visual comparison:

NO_ZC

ZC

[jmillan]

NOTE: I can provide the flamegraph .svg files in case there is any interest.

[isilence]

In order to enable zero copy I do as follows:

Enable SO_ZEROCOPY in the socket file descriptor

// Enable zero-copy in the file descriptor.
int on = 1;

err = setsockopt(this->fd, SOL_SOCKET, SO_ZEROCOPY, &on, sizeof(on));

SO_ZEROCOPY is not needed for io_uring zc and even harmful. Just a general note, it wouldn't change performance.

I'm handling notifications accordingly, making the buffers available when receiving the IORING_CQE_F_NOTIF in cqe-flags.

Tracing the process with and without zero copy I see that there is a slightly more CPU usage when using ZC. These are some observations comparing the traces:

That's how it often goes, CPUs are pretty good at copying small buffers, assuming it's <=1500B per request in your test, and zerocopy also adds some additional overhead. The kernel's networking stack is also pretty heavy in general, saving on small copies wouldn't make a drastic change. You might try to enable gso (e.g. UDP_SEGMENT), or if possible bump up the request size (together with network mtu).

* There is a function `io_import_fixed` being called, taking a little time. That is no present in `NO_ZC`. It's expected.

* There is a function `io_send_zc_prep` being called, takes a little more time than the analogous `io_sendmsg_prep`.

* `ip_append_data` takes considerably less time in `ZC` in comparison with `NO_ZC`. I can imagine this is where `CZ` shines.

Yes, that's where it copies data.

* `ip_send_skb` function:
  
  * `__ip_local_out` seems to take the same time in both cases.
  * `ip_finish_output` takes more time in `ZC` than in `NO_ZC`.

That's interesting, I don't see immediately why. Can you share the svg of the profiles as you offered?

[jmillan]

SO_ZEROCOPY is not needed for io_uring zc and even harmful. Just a general note, it wouldn't change performance.

Thanks for the info 👍 .

You might try to enable gso (e.g. UDP_SEGMENT),

I'll take a look at this.

or if possible bump up the request size (together with network mtu).

This is a constraint for the service, there are no packets sent or received >1500B.

Can you share the svg of the profiles as you offered?

Of course, yes.

EDIT: In order to see properly visualize the .svg's you may need to download and open locally.

NO_ZC

ZC

[jmillan]

After reading about UDP_SEGMENT, it's something that does not apply to our server, nor the MTU increase.

If I may briefly explain the nature of this server, it's basically a RTP (Real Time Protocol) relay that receives traffic (ie: audio, video) from one socket and relays such traffic to N other sockets. Each relayed packet is slightly modified, as we encrypt the data for each destination based on the uniquely selected keys.

In a nutshell, the nature of this traffic is real time media, the traffic received comes out of capturing a media device (microphone, webcam, etc), encoding the media, packaging the content into RTP packets and sending it to the server who will deliver such traffic to N endpoints that will un-package the data, extract the media and play it out of the corresponding media device (speaker, screen, etc).

This been said, each network packet needs to be delivered as it comes, so using UDP_SEGMENT is not an option as we can't wait to receive multiple packets in order to deliver them.

liburing has so far enhanced the performance by reducing the system calls for sendto. Since we relay one packet to N destinations, liburing allows us to prepare all sending packets and effectively send them within a single call to io_uring_submit().

I thought ZC was also going to help here, but as shown it does not. I still can't see how ip_finish_output2 takes more time with ZC than without it. Do you have any idea on how could this be further debugged?

[isilence]

First to note, we're looking at ~0.5% differences in functions down the stack, so it might be naturally noisy and hard to capture precise ratios. I also see that coredumping takes ~0.5%, which wouldn't usually show up in long enough testing. Let me ask, for how long you was sampling it? Would be great to have >1 min runs.

With that aside, let's zoom into start_xmit -- 6.86% (non-zc) vs 7.43% (zc), which is 0.57% difference, and let's assume it's a real difference, but not normalization, noise and other side effects.

1. virtqueue_add_outbuf() does extra allocations in case of zerocopy and costs extra 0.26%. I didn't check what's happening in virtqueue, but the most likely reason is that for normal sends the kernel will allocate a buffer and try to pack all headers and user data in it, so we pass only 1 page to virtqueue per send. For zerocopy, headers will still stay in separately allocated in kernel buffer, but user data would point to a user page, so we have 2 pages there per request. And looks it's matched with a free() in virtqueue_get_buf_ctx_split().

2. Now, I mentioned zerocopy brings some extra overhead, one of them is those notification completions. We can see them in
   ... -> start_xmit() -> free_old_xmit_skbs() -> ... -> skb_release_data() -> io_uring_zerocopy_callback() -> __io_req_task_work_add()
   There is not much I can do about that, most of it is adding ourselves into a list (1 cmpxchg when unconteded). The first version of the zerocopy API was a bit more complicated but allowed having a single notification CQEs per multiple sends. I might return it back if there is a really good case for that, but it's only 0.1% here.

It'll also try to wake up the task, but you're lucky here because it already runs in the task context and so doesn't have to actually wake it up, which is pretty expensive. It's usually not like that with other drivers, especially for real hw. In fact, the trace is only for the task but doesn't cover irqs, which might be calling io_uring_zerocopy_callback and adding overhead that we don't see.

3. Another part of it is in do_syscall_64() -> syscall_exit_to_user_mode() -> ... -> task_work_run(), ~0.25%.

There is a feature that can help a bit with 2) and 3) called IORING_SETUP_DEFER_TASKRUN, but it should go together with IORING_SETUP_SINGLE_ISSUER and adds some limitations on the userspace, i.e. only one task can use the ring, but seems the test/app is single threaded anyway. The user is also has to do a syscall to get completions (e.g. via io_uring_wait_cqes()).

Also several notes not related to zerocopy:

• You may try registering the ring fd, see IORING_ENTER_REGISTERED_RING and IORING_REGISTER_RING_FDS.
• Try out IORING_REGISTER_FILES
• Tying io_uring into epoll via eventfd incurs overhead.
• You can try to convert recvmsg(2) to io_uring, and take a look at multishot recv.

[jmillan]

I've repeated the test to be longer than 1 minute order to remove the possible noises.

NOTES:

• IORING_REGISTER_FILES is used now.
• IORING_REGISTER_RING_FDS not used as I got Operation not supported as response to io_uring_register_ring_fd().
  • kernel version: 6.1.0-13
• Encryption has been disabled now, it's a legitimate case and was providing too much noise on the traces.

As you can see there is now a 2% difference between ZC and NO_ZC, being ZC more efficient.

ZC

NO ZC

Probably now we are seeing the benefits of using ZC compared to not using it, for our use case (small buffers). I hope everything makes sense.

Also several notes not related to zerocopy:

You may try registering the ring fd, see IORING_ENTER_REGISTERED_RING and IORING_REGISTER_RING_FDS.

Tried, but got the mentioned error. Maybe it's only enabled for newer kernels.

Try out IORING_REGISTER_FILES

New traces do contain it.

Tying io_uring into epoll via eventfd incurs overhead.

This is a convenience as our server lies on top of libuv event loop. The server does not handle threads. I'm registering an eventfd to the io_uring and letting libuv pool such eventfd and notify me whenever there are CQEs.

You can try to convert recvmsg(2) to io_uring, and take a look at multishot recv.

We are actually using recvmmsg. Maybe we could take advantage of io_uring multishot recv, but I was trying the sending side as the server relies N packets for each one it receives, so the highest impact can happen in the sending side. Also our server works for other OOSS apart from Linux, so I'd rather keep using libuv for reception.

[isilence]

As you can see there is now a 2% difference between ZC and NO_ZC, being ZC more efficient.

Sweet, glad to see that.

Probably now we are seeing the benefits of using ZC compared to not using it, for our use case (small buffers). I hope everything makes sense.

Getting rid of virtio allocations might be a pretty easy win here. From a quick look it seems because of this line:

https://elixir.bootlin.com/linux/v6.7-rc3/source/drivers/virtio/virtio_ring.c#L251

I'd bump the threshold to 2 and see if it helps.

diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c
index 51d8f3299c10..d6170cbd9e49 100644
--- a/drivers/virtio/virtio_ring.c
+++ b/drivers/virtio/virtio_ring.c
@@ -248,7 +248,7 @@ static bool virtqueue_use_indirect(const struct vring_virtqueue *vq,
 	 * If the host supports indirect descriptor tables, and we have multiple
 	 * buffers, then go indirect. FIXME: tune this threshold
 	 */
-	return (vq->indirect && total_sg > 1 && vq->vq.num_free);
+	return (vq->indirect && total_sg > 2 && vq->vq.num_free);
 }
 
 /*

[jmillan]

I'd bump the threshold to 2 and see if it helps.

I'll try this when possible and report the results 👍

[jmillan]

@isilence,

Appologizes.., It's been too long since I did compile a Linux kernel. I imagine in this case only virtio module needs to be compiled. Would you point me somewhere with the doc to compile a module out of the recent Linux sources?

Another question: how comes virtio is being used? Is it due to Linux being virtualized in this case, or does io_uring use it on every case?

Thanks

[isilence]

how comes virtio is being used? Is it due to Linux being virtualized in this case, or does io_uring use it on every case?

It's because it's a VM and it's configured to use virtio for net, unrelated to io_uring. And I forgot to mention, that last diff won't give you anything, if you use VMs only for testing, but it's still would be useful in general to fix it up if numbers back the diff.