-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Compact type encoders #137
Comments
I see a few question we should answer before starting this.
|
After having implemented half a dozen set of encoders, I don't know how useful this will be. In each instance, I have had to tweak the encoding to the use case, while using the mango encoders where appropriate. Actually, the simple encoders in mango have been the ones that I have used the most for an actual compact string encoder. For instance, for one of my lexicoders I have had to create an encoder which creates a more compact number encoding. The only downside is that it won't encode numbers > 10^128. For our usecases this is more than acceptable and nets us a use space savings on the encoding for most numbers. For our over-the-wire encodings we also have a novel approach that helps number fit into the ascii text range. We simply encode all numbers into base 36 (MAX_RADIX). This usually halves the size of the data without requiring any special libraries for special handling. For each of these we also used a 1 char alias with no special meaning. This allowed for a much smaller footprint over the wire and in storage. While these encoding worked well for us, I doubt they would be very universal |
Thrift and Avro both supply really good standards for fast and lightweight compression of various different data formats for sending over the wire. For cases where we don't need lexicographically sortable types but instead just want something lightweight and compact to, for instance, place in the Value of an Accumulo key/value, I think it would be great it we could get a set of CompactTypeEncoders going.
The "Types" sunsection of the thrift paper would be a good place to start:
https://thrift.apache.org/static/files/thrift-20070401.pdf
The text was updated successfully, but these errors were encountered: