minor incompatibility between recvmsg multishot and kTLS

[lano1106]

https://github.com/openssl/openssl/blob/master/include/internal/ktls.h#L152

openssl tricks recvmsg() into writing 5 bytes into the user buffer so that it can recreate the TLS header based on the ancillary data returned to avoid recopying the whole thing.

As a side note, this is required only for TLSv1.2. This is not used with TLSv1.3...

Because of the way that recvmsg_out is packed into the provided buffer, this is not possible when doing the recvmsg call with io_uring recvmsg multishot.

I'll try to artificially increase the controllen to see if I can still do it. This is hackery but probably possible.

Not really an issue per se but this is something that I wanted to bring to your radar if this type of trickery is common enough to justify facilitating this type of operation explicitly.

[DylanZA]

this is interesting. I believe that should work as the struct order is [hdr][name][control].

Since the buffer as a whole is provided to io_uring in a pool, I think it would be difficult to do this more generally.

[lano1106]

I have written code to try out...

/*
 * getPayloadBuffer()
 */
char *bio_iouring_params::getPayloadBuffer(bidEntry &entry) noexcept
{
    auto *bidBuf{getBidBuffer(entry.bid)};

    if (!(entry.read_flags & EV_IOURING_KTLS_READ))
        return bidBuf;

    /*
     * the io_uring inline functions manipulating the recvmsg multishot
     * result do not need to have the exact same msghdr than the one
     * used to initiate the request. It only needs to have one
     * that has the exact same value for the fields msg_namelen & msg_controllen
     * so that it can navigate correctly into the buffer with the correct
     * offsets.
     */
    static struct msghdr s_msgh = {
        .msg_name = nullptr,
        .msg_namelen = 0,
        .msg_iov = nullptr,
        .msg_iovlen = 0,
        .msg_control = nullptr,
        .msg_controllen = CMSG_SPACE(sizeof(unsigned char)),
        .msg_flags = 0
    };
    /*
     * assume that recvmsg_out cannot be nullptr
     * If the assumption turns out to be false, it will be noticed
     * immediately with a SEGV.
     */
    auto *recvmsg_out{io_uring_recvmsg_validate(bidBuf, entry.bytes_read,
                                                &s_msgh)};
    auto *payload{io_uring_recvmsg_payload(recvmsg_out, &s_msgh)};
    /*
     * the initial bytes_read value included the space used by the recvmg_out
     * headers. We now set it to the payload size only.
     */
    entry.bytes_read =
    io_uring_recvmsg_payload_length(recvmsg_out,
                                    entry.bytes_read, &s_msgh);
#ifndef OPENSSL_NO_KTLS
    /*
     * A hack is required to work with kTLS.
     * recvmsg() with kTLS is passing payload buffer 5 bytes in front of the
     * user buffer so that it can recreate the TLS header in front of the TLS
     * payload.
     *
     * Also note that the code only prepend the SSL header if control data
     * is present but it should always be present in msgs coming from kTLS.
     */
    struct cmsghdr *cmsg = io_uring_recvmsg_cmsg_firsthdr(recvmsg_out, &s_msgh);
    if (cmsg) {
        if (cmsg->cmsg_type == TLS_GET_RECORD_TYPE) {

            payload -= SSL3_RT_HEADER_LENGTH;
            payload[0] = *((unsigned char *)CMSG_DATA(cmsg));
            payload[1] = TLS1_2_VERSION_MAJOR;
            payload[2] = TLS1_2_VERSION_MINOR;
            /* returned length is limited to msg_iov.iov_len above */
            payload[3] = (entry.bytes_read >> 8) & 0xff;
            payload[4] = entry.bytes_read & 0xff;
            entry.bytes_read += SSL3_RT_HEADER_LENGTH;
        }
    }
#endif
    return static_cast<char *>(payload);
}

I am confident that it will work fine. Once all the needed info has been extracted, there seems to be plenty of space in front of the payload to play with...

Note the comment above s_msgh declaration. Before understanding that point, I was keeping track of all the inflight multishot recvmsg struct msgh in a hash table. This was an unnecessarily complicated setup. I have a similar static global struct msghdr on the sqe issuing side.

For the benefit of all users, I think that a small note in the documentation about this could be helpful!