Online Data-Reduction / Compression

[besnardjb]

Tracing

Currently the data are stored in memory and therefore space might be exhausted. You may consider:

1. Compression
2. Flush to disk (when needed)
3. Periodical aggregation

On a second step:

You write multiple files at the end would it be possible to write only one performing data-sorting in parallel ?

[dssgabriel]

Compression

Interpol now outputs JSON a more compact but less readable format. This enable a compression rate of up to 30%. Ideas for further reduced trace sizes included compressing the JSON output (zlib?) or changing the output format altogether.

Flush to disk

Flush to disk has not been implemented yet, we need to think about how we could do that efficiently.

Parallel sorting

Traces are now sorted in parallel thanks to the rayon crate and outputted as a single JSON file (implemented in #14).