Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make comm_p tree building streaming #1360

Open
vmx opened this issue Nov 11, 2020 · 0 comments
Open

Make comm_p tree building streaming #1360

vmx opened this issue Nov 11, 2020 · 0 comments
Labels
enhancement New feature or request

Comments

@vmx
Copy link
Contributor

vmx commented Nov 11, 2020

Description

Currently the comm_p calculation needs an mmap'ed file of the leaves. A tree is then built bottom up.

Instead you could use some input stream and build the tree on the fly as you are really only interested in the root of the tree.

Calculating comm_p is currently used in preparing storage deals and if the IPFS integration for Lotus is used.

Acceptance criteria

comm_p doesn't need the input data as a single file, but can consume it in streaming fashion.

Risks + pitfalls

The questions is what a "stream" is, especially if you want to interact with it over an FFI.

Where to begin

Start with the current way of having a file with all the leaves and stream that into a tree builder. Once that works you can think about how to make the input a stream.

@cryptonemo cryptonemo added cryptocomputelab CryptoComputeLab work enhancement New feature or request and removed cryptocomputelab CryptoComputeLab work labels Jan 21, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants