Make the buffer size configurable?

[ksolana]

The writer::new function has hardcoded buffer value.

    /// Creates a new `Writer`.
    ///
    /// All output from the given operation will be forwarded to `writer`.
    pub fn new(writer: W, operation: D) -> Self {
        Writer {
            writer,
            operation,

            offset: 0,
            // 32KB buffer? That's what flate2 uses
            buffer: Vec::with_capacity(32 * 1024),

            finished: false,
            finished_frame: false,
        }
    }

Can we make the buffer size configurable? I'd like to experiment with different buffer sizes and see if that helps with compression ratio/speed.

[ksolana]

Related hail-is/hail#14033

[gyscos]

Hi, and thanks for the request!

Indeed, it would likely make sense to expose a way to set a custom buffer size. I'm surprised by the 12% saving reported in that hail ticket though. I think they changed the internal block size, not just the output buffer size. This buffer is not going to affect compression ratio, only potentially reduce the number of times things need to be copied between zstd's own internal buffer, this output buffer, and the actual writer destination.

[ksolana]

This buffer is not going to affect compression ratio, only potentially reduce the number of times things need to be copied between zstd's own internal buffer, this output buffer, and the actual writer destination.

Yeah that sounds reasonable. Because speed is desired for certain workloads e.g., anza-xyz/agave#2729, it'll be great to expose the buffer size option to the user.