Can Daft parse .proto files?

[IvanLiuTW]

Hi there,

The .proto had appeared on the picture in the Readme, but I am unable to find a related function in DataFrame section of Daft Documentation.

Is it in another section of the documentation or how can I use daft to parse .proto file into a dataframe?

Thank you!

[jaychia]

EDIT: Daft 0.0.13 now supports all the features requested in this thread - DataFrame.explode and DataFrame.from_files

Yes we intend to support this in some form, but the protobuf usecase does need some speccing out.

Specifically since protobufs are more commonly used as a wire format than a file format, I'd love to learn your use-case a little better and we can file this as a feature request to work on soon.

1. Where should Daft be retrieving the protos from? Object store like S3 vs database vs a gRPC API?
2. Are you employing any chunking/serialization to store multiple protobufs as a single file?
3. If you are indeed storing them on disk, how are you serializing your protos?

[jaychia]

Thanks! Did a little digging and it seems to be a bug in our query optimizer code. I'll make an issue and tag you when we resolve it next release!

Edit: tracking issue #258

[jaychia]

Just fixed and this will be released with 0.0.14 over the weekend!

[IvanLiuTW]

Hi Jay,

I just tried the new version, and everything went very well! I truly appreciated that you took my use case seriously and also showed me how to do it via Daft. It does help a lot. I will keep using Daft, and also sharing my feedback!

Thank you very much. :)

[jaychia]

Thanks for the feedback - let us know if there are any other features you're looking for that we can provide.

[IvanLiuTW]

Sure, no problem!