What is the role of io_uring_sq.array, can someone explain it in detail?

[uuuuu-diwu]

The tail and head in sq have already indexed sqe, why do we need to store the index in the array, and it is very strange that the subscript and value are the same when storing the array, can anyone help?

[axboe]

It's to enable submitting entries from the array of sqes out-of-order. This can be useful in some cases where the application IO unit embeds certain elements that require it to use the same entry every time, which is why it was done that way.

[uuuuu-diwu]

thanks a lot for your answer! Is the following understanding correct? Getting a sqe and filling sqe is a two-step operation and not atomic, so the kernel cannot determine that the sqe between head and tail is ready, so the user needs to put the ready sqe index to the array to tell the kernel

[axboe]

No, that's not related to that. The sqe doesn't become visible to the kernel until the sq ring head is incremented, which is unrelated to how the sqes are indexed as per the original question.

[uuuuu-diwu]

Great, I understand, I just looked at the source code and found that liburing maintains a user-level tail and head by itself, and updates the tail and head to the kernel's tail and head during __io_uring_flush_sq

[electfreak]

@axboe I do my best to but can't understand it. Especially the "use the same entry every time" thing. Does it mean "reuse submission queue entry", like submitting it again without filling all the fields twice?

As for order, isn't it kind of parallel in asynchronous IO? If we were sending more than one entry, why would we think about the order? They come to CQ in plain order and I thought it's because they're running independently.

Am I right that submitting entries from the array of sqes out-of-order implies that we reorder indices like

sring->array[index + 1] = index;
sring->array[index]  = index + 1

to reorder two adjacent entries?

Could you please provide some small code samples for example you describe?

[isilence]

"out-of-order" here means that SQEs don't have to be placed sequentially in memory in the SQ, I don't think there was a deeper meaning to that. With re-indexing you can prepare some SQEs in advance and only point to them while submitting, i.e. potentially saves zeroing and initializing 64 bytes (==sizeof(SQE)) per request. See a simplified pseudo code example below.

Note that it's a part of the kernel API and has never been exposed by liburing. And now there is a IORING_SETUP_NO_SQARRAY flag, which removes the indirect indexing and is automatically set by liburing.

do {
	sqe = get_sqe();

	// sizeof SQE is 64B, it has to initialise it all
	switch (get_event()) {
	case READ_DATA:
		io_uring_prep_read(sqe, read_fd, ...);
		break;
	case WRITE_DATA:
		io_uring_prep_write(sqe, write_fd, ...);
		break;
	}
} while (submit_and_process_completions());

const read_sqe_index = 0;
const write_sqe_index = 1;

// init sqes in advance
read_sqe = sqes_array[read_sqe_index];
io_uring_prep_read(read_sqe, ...);
write_sqe = sqes_array[write_sqe_index];
io_uring_prep_write(write_sqe, ...);

do {
	// get_event() would usually be derived from completions
	switch (get_event()) {
	case READ_DATA:
		sq_index_array[sq->next++] = read_sqe_index;
		break;
	case WRITE_DATA:
		sq_index_array[sq->next++] = write_sqe_index;
		break;
	}
} while (submit_and_process_completions());