How to handle registered file rotations? (Opening a new file that all later ops should use)

[RedBeard0531]

My application is attempting to write to an append-only file, and rotate to a new file once it hits a certain size. To achieve this, I have a loop which roughly looks like this (very much psudo-code, do the obvious translation from function calls to SQE allocation and filling. Also assume all operations specify IOSQE_FIXED_FILE to used the registered file, rather than treating as an FD):

const int file = 0; // Always using a single registered file.
off_t offset = 0;
while (true) {
  for (auto w : readyToWrite) {
    write(file, w, offset);
    offset += w.size();
  }
  if (lastFsyncHasCompleted) {
    fsync(file, IOSQE_IO_DRAIN);
    if (offset >= kFileRotateSize) {
      open(file, makeNextFileName(), IOSQE_IO_DRAIN); // Also closes the already open file.
      offset = 0;
      nop(IOSQE_DRAIN); // instead of marking next op (whatever it is) with DRAIN
  }
  io_uring_enter();
  // Process completion events
}

I have a few questions about if this is valid and the best approach:

• Do I need to mark the openat SQE with DRAIN to avoid racing with earlier operations and closing the file under them? Ideally, I'd like it to be able to run concurrently with the fsync, if it wouldn't interfere. I know I could maintain a rotation of registered file indexes in userspace and manually close them, but I'm not sure if it is worth the application level complexity to do so to avoid a small pipeline bubble in the rare event of file rotations.
• Do I need the nop(DRAIN) (or marking the next SQE with drain) to ensure that all later operations use the new file rather than old one? I assume that at least this or the open_at must use DRAIN.
• Is it valid to use nop(DRAIN) rather than marking the next SQE with DRAIN? The docs say that nop is primarily for testing, but it at least seems like a good way to make all later ops wait for some operation to finish without needing to add if (needDrain) { needDrain=false; sqe->flags |= IOSQE_IO_DRAIN; } to every single SQE that uses the registered files (there are a handful and it isn't always known which the next will be upfront).
• Does IO_FEAT_LINKED_FILE imply that ops on or after a DRAIN point also get late file assignment? The docs only talk about IOSEQ_IO_LINK, but I can't use that because my dependencies will often cross submissions.
• Is there a better pattern for appending to (and fsyncing) rotating files?

[isilence]

First of all, I always discourage people for using DRAIN, as its performance is miserable, maintaining it is hard, there will cases when using drain once in past would be slowing down the ring for its entire lifetime, and there are even features incompatible with DRAIN (e.g. IOSQE_CQE_SKIP_SUCCESS). Instead, userspace can count the number of inflight requests for a file.

* Do I need to mark the `openat` SQE with `DRAIN` to avoid racing with earlier operations and closing the file under them? Ideally, I'd like it to be able to run concurrently with the fsync, if it wouldn't interfere. I know I _could_ maintain a rotation of 

All requests can get executed asynchronous in parallel, so unless you put some ordering requests might execute in random order.

submit(sqe1=OP_WRITE, sqe2=OP_CLOSE);

This may close first and write will fail with -EBADF or so. Same with sync(); close();

submit(sqe1=OP_WRITE,fd_slot=0, sqe2=OP_OPEN,target_fd_slot=0);

It may successfully open first, replace the file in the table, and then the write will be issued to the new file.

registered file indexes in userspace and manually close them, but I'm not sure if it is worth the application level complexity to do so to avoid a small pipeline bubble in the rare event of file rotations.

There is IORING_FILE_INDEX_ALLOC, in which case io_uring will allocate a slot for you, but you'd still need to close it.

* Do I need the `nop(DRAIN)` (or marking the next SQE with drain) to ensure that all later operations use the new file rather than old one? I assume that at least this or the `open_at` must use `DRAIN`.

You can only safe the new file once you see the open completion, that's it unless you do some ordering with DRAIN or IOSQE_IO_LINK.

* Is it valid to use `nop(DRAIN)` rather than marking the next SQE with `DRAIN`? The docs say that nop is primarily for testing, but it at least _seems_ like a good way to make all later ops wait for some operation to finish without needing to add `if (needDrain) { needDrain=false; sqe->flags |= IOSQE_IO_DRAIN; }` to every single SQE that uses the registered files (there are a handful and it isn't always known which the next will be upfront).

Yes, it's valid

* Does `IO_FEAT_LINKED_FILE` imply that ops on or after a `DRAIN` point also get late file assignment? The docs only talk about `IOSEQ_IO_LINK`, but I can't use that because my dependencies will often cross submissions.

It should be. I'd encourage you to write a test and hopefully donate it to liburing :)

* Is there a better pattern for appending to (and fsyncing) rotating files?