Reduce wasted memory in ChunkedSliceOutput

[chenyangfb]

Description

Currently the minimal buffer size of ChunkedSliceOutput is 8kb, this leads to lots of wasted memory with large number of output buffer, each with a few bytes. For example, with 10k output stream with 10 bytes, the total size of ChunkedSliceOutput is 80MB while the logical size is 100kb.

This PR reduce wasted memory in ChunkedSliceOutput by allowing smaller buffer size in ChunkedSliceOutput. The default value is still 8kb.

Impact

Improve memory usage and reduce wasted memory

Test Plan

Tested with Spark workload writing flat map column.

Control use 8kB chunk buffer size while test use 1kB chunk buffer.
Total memory usage from ChunkedSliceOutput reduced from 2GB to 700MB and memory usage per object reduced from ~8kB to ~2kB.

Before

After

Contributor checklist

• Please make sure your submission complies with our development, formatting, commit message, and attribution guidelines.
• PR description addresses the issue accurately and concisely. If the change is non-trivial, a GitHub Issue is referenced.
• Documented new properties (with its default value), SQL syntax, functions, or other functionality.
• If release notes are required, they follow the release notes guidelines.
• Adequate tests were added if applicable.
• CI passed.

Release Notes

General change
Allow setting chunk size in ChunkedSliceOutput, it can reduce memory in the cases of large number of small output buffer

[sdruzkin]

1. How does it impact writer performance? Smaller initial chunks will result in additional 2-3 mem copy operations.
2. The change you made will result in smaller initial size, but what would happen after a few batches are fed into the writer? In my understanding there will be no impact on reducing the memory consumption, except the perf regression because of additional mem copy.
3. Won't it make more sense to set the initial size to 0, but then grow to 4k at the first write?

Please update the PR Release Notes.

[chenyangfb]

Thanks @sdruzkin
This diff added support for setting chunk size, but the default size is the same as before (8kB).
Smaller initial chunks such as 1kB is aimed for cases with large number of output stream such as flat map with 10k output stream and typical bytes per output stream range from 10 bytes to a few kB.

So for those cases, since the typical bytes per output stream is small (but not zero), the writer performance impact is minimal but the memory saving is big and set the initial size to 0 won't reduce memory.

I have a few other changes for setting the config in Spark and propagate it to presto-orc, not sure if I should share it here.

I plan to do more Spark shadow testing to measure the performance impact with different chunk size.

[sdruzkin]

Revert this unnecessary change

[sdruzkin]

I'm confused. You are claiming that the old min chunk size was 8kb, but here in the Orc compression output slice it's 4kb. Can you please clarify this?