Callbacks for Storage.Find

[roman-khimov]

Summary or problem description
We have some limitations in the System.Storage.Find API:

• no way to get some part of the key, RemovePrefix and KeysOnly flags only work for some subset of cases
• Deserialize and Pick0/Pick1 flags are good, but if contract stores something in the field number 2 or 3, or if it has some structured fields 0/1 and needs to go deeper there is no way to do that
• no way to filter results, in some cases only a subset of data is really needed
• no way to get some data from other DB fields that can be relevant

Do you have any solution you want to propose?
These limitations can be solved with some callback code used by Iterator.Next. In its essence it's a code that can take a key and a value from parameters and return any other case-specific value (or throw an exception if this KV pair needs to be skipped). Either a bytecode (given that we have some dynamic capabilities now with #2756), or a contract (reminding of Oracle callbacks). Contract-based callback scheme is much easier to implement, but bytecode gives a bit more flexibility. Something like

System.Storage.FilteredFind(context, prefix, options, method) -> Iterator

(options are still needed at least for #2789), or

System.Storage.FilteredFind(context, prefix, options, script) -> Iterator

with a method/script like

fun(key, value) -> value

Another approach could be adding a new IteratorWrapper entity like

System.Iterator.Wrap(iterator, method) -> Iterator

or

System.Iterator.Wrap(iterator, script) -> Iterator

with a method/script like

fun(iterator) -> (value, ok) // a struct/array, we need some way to signal that it's over

To keep Storage.Find API the way it is now and do the same thing in this layer. The fun function will then be called on every Iterator.Next invocation (and call internal iterator's Next() as appropriate) until it returns false in ok, but while it's true value is the thing to be returned from Iterator.Value.

This wrapper/filter is to be executed with ReadOnly flags. For on-chain cross-contract invocations GAS is counted naturally for it, while RPC session code could use the GAS that is not yet used by this session (MaxGasInvoke-GasConsumed and counting down with every session iteration).

Neo Version

• Neo 3

Suggested implementation version

• 3.6.0

Where in the software does this update applies to?

• Interops
• SDK