Slow Cached Performance of io_uring

[Hemanth-Eduru]

Hi,

I observed the following w.r.t io_uring all the data is served from Linux page Cache (Buffered Mode and no data read directy from disk) and test are with sequential access

• At smaller read sizes (say 2K,4k,8K) the performance of io_uring is better or comparable to pread
• Performance of the io_uring degrades w.r.t to pread once the Read size grows and
• For the Higher read sizes the performance drops with QD

Test details: io_uring with batch submit and batch size same as QD.
Below graph info : Performance of io_uring with respect to pread, The Horizontal axis corresponds to Read (block) size and QD and Vertical axis corresponds to relative performance of io_uring compared to pread (which is also lablled in brackets in graphs) Red lines indicate where performance is slower than pread. (Below screeshot contains performance with 1,2,4 threads)

Also verified the performance (to ensure that my code is not the culprit) with https://github.com/axboe/fio/blob/master/t/io_uring.c and the performance number for io_uring I got with my code and with t/io_uring are similar

Profiling (using vtune amplifier) shows io_uring is more memory bound (69%) when compared to pread (46%)

Are my observations correct? if they are correct whats the reason for the same?

Machine Details

CPU : Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz
L1d cache:                          640 KiB
L1i cache:                          640 KiB
L2 cache:                           20 MiB
L3 cache:                           27.5 MiB
RAM: 128 GB
Ubuntu 20.04 -> 5.15.0-83-generic

Code :

.....................................io_uring....................................................
for (size_t i = 0; (i < (maxBlockCount / QD)) && (ioType == IO_URING_BATCH); i++) {
        for (int p = 0; p < QD; p++) {
            struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
            if (sqe == NULL) {
                fprintf(stderr, "Unable to get SQE\n");
                exit(EXIT_FAILURE);
            }
            off_t idx = isRandomRead == 0 ? i : randomIdx[currThreadIdx][i];
            off_t offsetToRead = (idx * QD + p) * blockSize;
            io_uring_prep_read(sqe, fd, buf[p], blockSize, offsetToRead);
            io_uring_sqe_set_data(sqe, buf[p]);
        }
        auto ret = io_uring_submit(&ring);
        if (ret < 0) {
            fprintf(stderr, "io_uring_submit: %s\n", strerror(-ret));
            exit(EXIT_FAILURE);
        }
        for (int p = 0; p < QD; p++) {
            struct io_uring_cqe *cqe;
            auto ret = io_uring_wait_cqe(&ring, &cqe);
            if (ret < 0) {
                fprintf(stderr, "io_uring_peek_cqe: %s\n", strerror(-ret));
                exit(EXIT_FAILURE);
            }
            void *data = io_uring_cqe_get_data(cqe);
            // auto shaRes = sw::sha512::calculate(data, blockSize); // hash of a string, or
            // printf("i->%9lu and p-> %9d Sha512SUM at input is %s\n", i, p, shaRes.c_str());

            if (cqe->res < 0) {
                fprintf(stderr, "cqe failed -->: %s\n", strerror(-cqe->res));
                exit(EXIT_FAILURE);
            }
            // if (cqe->res < blockSize) {
            //     fprintf(stderr, "cqe failed: less data\n");
            //     exit(EXIT_FAILURE);
            // }
            readSize += cqe->res;
            io_uring_cqe_seen(&ring, cqe);
        }
    }
...............................................................pread........................

    for (size_t i = 0; (i < maxBlockCountPread) && (ioType == IO_PREAD); i++) {
        auto idx = isRandomRead == 0 ? i : randomPreadIdx[currThreadIdx][i];
        off_t offsetToRead = (idx)*blockSize;
        pread(fd, buf[0], blockSize, offsetToRead);
        readSize += blockSize;
    }

[redbaron]

Could you test on more recent version and also try XFS

[Hemanth-Eduru]

I repeated the test on ubuntu 22.04 with 6.2.0-32-generic (but on a different machine) results are similar.

Regarding xfs does the file system matters if all the data is cached in RAM (via linux page cache) ?

[redbaron]

Regarding xfs does the file system matters if all the data is cached in RAM (via linux page cache) ?

According to this comment it does

[Hemanth-Eduru]

Got it.
But the reason why I wanted to this test is that i want get most out of IO and in my app I don't know what will be my block size (it will vary depends on query my application is a columr DB and number of concurrent queries will be less)

And now I am getgiy that at smaller sizes opting for more QD will be helpful and and larger sizes opting for smaller QD will be helpful...

So Instead of design my app with const QD I can use the above logic to get best performance..

Will also do more tests and will update here..

[axboe]

My guess is that you're running way more ios on the io_uring side row higher thread/depth counts, and that's blowing your TLB. As a quick test, you could use the same buffer for each io within a tread regardless of queue size. That of course won't make sense, but it that performs much better, this is the reason.

[axboe]

Of course another way to look at this is to utilize huge pages for buffers, you could carve all your IO buffers out of a single huge page. That would also drastically reduce the TLB pressure, and I strongly suspect you'll see the same performance when threads and/or queue depth is ramped up.

[Hemanth-Eduru]

I just did a quick test by sharing the same buffer within a thread and suddenly performance is not degrading with increase in the QD.

For example when compared to original test with bs=512K and QD=64 the performance has improved by 40-50%, although no change for QD=1, for obvious reasons.

Will also repeat the test with hugepages and will update..

[Hemanth-Eduru]

I did the test even with HugePages (with each Queue getting a non overlapping portion of the allocated buffer) (via madvice and cat /proc/meminfo shows usage in AnonHugePages when I am running the benchmark) not much change in the performance. (This is the first time I am actually working with huge Pages so I could have messed up too)

size_t alignment = 2 * oneMB; /* HugePage Size */
size_t requiredSize = QD * blockSize;
requiredSize =
    ((requiredSize % alignment) == 0) ? requiredSize : (alignment) * ((requiredSize / alignment) + 1);
if (posix_memalign(&(buf), alignment, requiredSize)) {
    printf("Unable to Align buffers to %d failed with error\n", blockSize);
    exit(EXIT_FAILURE);
}
madvise(buf[0], requiredSize, MADV_HUGEPAGE);
.
.
.
.
char *data = (char *)buf;
size_t readSize = 0;
for (size_t i = 0; (i < (maxBlockCount / QD)) && (ioType == IO_URING_BATCH); i++) {
    for (int p = 0; p < QD; p++) {
        struct io_uring_sqe *sqe = io_uring_get_sqe(&ring);
        if (sqe == NULL) {
            fprintf(stderr, "Unable to get SQE\n");
            exit(EXIT_FAILURE);
        }
        off_t idx = isRandomRead == 0 ? i : randomIdx[currThreadIdx][i];
        off_t offsetToRead = (idx * QD + p) * blockSize;
        io_uring_prep_read(sqe, fd, data + p * blockSize, blockSize, offsetToRead);
        io_uring_sqe_set_data(sqe, data + p * blockSize);
    }
    auto ret = io_uring_submit(&ring);
    if (ret < 0) {
        fprintf(stderr, "io_uring_submit: %s\n", strerror(-ret));
        exit(EXIT_FAILURE);
    }
    .
    .
    .

[axboe]

As per your previous test, I think we can conclusively say that it's a TLB issue. The huge page idea is just a potential solution for you if you want to use more buffers with io_uring than you do on the sync side. Going into details on how to use huge pages is outside the scope of this project, it's somewhat of a mess imho. I'd suggest to mark the issue as answered/closed.

[axboe]

Ran a quick test on a local box here, buffered IO and file fully cached. Here's what I see:

1 thread, bs=4k
pread 8.11GiB/sec
io_uring 8.39GiB/sec

4 threads, bs=4k
pread 26.88GiB/sec
io_uring 27.72GiB/sec

1 threads, bs=512k
pread 22.99GiB/sec
io_uring 23.05GiB/sec

4 threads, bs=512k
pread 53.81GiB/sec
io_uring 53.81GiB/sec

This is NOT using registered buffers, because otherwise io_uring is a lot faster. As an example, 4 threads using 4k:

io_uring 29.15GiB/sec

This is using a XFS, 10G file, on an amd 7950x. I'm using t/io_uring for the tests, ala:

taskset -c 0-3 t/io_uring -O0 -d1 -s1 -c1 -p0 -b$bs -S$pread -B0 -n$threads /data1/file

with bs, pread,threads being the only thing changed. -B0 turns out registered buffers which is otherwise the default, outside of the last time it is off for the first ones.

[Hemanth-Eduru]

Hi
In orginal test I ran it on disk with ext4, tomorrow will format the disk to xfs and will repeat the test.
Also one thing to note that even in my test if I run the io_uring with QD of 1 the performance is almost same as that of pread but problem is that performance is degrading at higher QD.

[Hemanth-Eduru]

Is this because we are touching memory b/w limit per thread and trying to request more causing memory bottleneck? (As shown by vtune amplifier proffiler)