zstd dependency?

[TheChymera]

I see that the code mentions zstd quite a bit, but does not list it in the dependences.
Is this an ommission, or does the package want to use its own zstd?
If so, is there any way to tell it to use the system zstd distribution?

chymera@decohost ~/src/numcodecs $ ag zstd -o | wc -l
703
chymera@decohost ~/src/numcodecs $ ag zstd requirements* -o | wc -l
0

[joshmoore]

Hi @TheChymera. zstd is being pulled in via the c-blosc/ submodule. I'll defer to others whether or not there could be a way to use a different version.

[martindurant]

I'd like to note the package cramjam, which includes several de/compression algorithms optimised for numpy buffers and in a single small static binary. It does not blosc, so no blosc frame headers. This is what fastparquet uses.

[jakirkham]

Yeah as Josh says this is currently compiled into the binaries we ship via Blosc. So no additional dependency needed.

Martin, to your point, we discussed in issue ( #70 ) breaking out compressors as dependencies, which would have been one way to make this pure Python. Though I think adding binary wheels and some of the additional compressor code added since, probably makes pure Python less relevant (or maybe even feasible). That said, no objection to anyone picking up this refactoring work up if it is of interest 🙂