Create LambdaStackItem and discussions on Function Pointers

[igormcoelho]

UPDATE: this changed from LambdaStackItem to a FunctionPointer stack item, during progress.

---

Design: #190 (comment)

---

I think we will need a very nice new stack item type, called LambdaStackItem :)

@erikzhang I've been thinking a lot, and a lot, and I've been convinced that OP_EVAL is not a good idea, due to its insecure and "poor-programming" nature: neo-project/neo#328

However, we have an alternative. We can have a StackItem type that carries a Script, and a specification for inputs/outputs.

We will require a new opcode: NEWLAMBDA, that specifies inputs/outputs, and a Script (read as regular sequence of bytes), read from stack. I think it's important to read from stack (rather than direct parameters), because we can make it much more powerful (not using only compiled code, but also those passed by users on contract).

This is much safer than OP_EVAL. This "Lambda" StackItem will not process recursion, nor any types of CALL (loops are fine, "perhaps"). Perhaps, it's better not to allow SYSCALLS too, although I'm not fully sure of it. Certainly, storage operations are not desirable, but some basic JSON, serialization, deserialization, etc, may be welcome inside a lambda.

Hopefully, this may help us solve this: neo-project/neo#880 (comment)

[igormcoelho]

@lightszero @shargon what do you think? do you think this is feasible?

P.S.: I still want to abolish one of Array or Struct (probably Struct). I have some new arguments to go back on that discussion.. just wait a little bit more ;)

[igormcoelho]

@vncoelho I think lambdas will solve best our problems too.

[vncoelho]

I will call you soon for understanding it better, brother.
Trying to hunt that P2P duplicated tasks and invpayloads.

[igormcoelho]

Guys, I just saw that, receiving scripts on runtime (via stack) to add on a LambdaStackItem will cause issues, as we won't be able to pre-calculate the estimated script time (some use could pass different things as parameter, build a lambda, and execute it on verification).
This thing will be less powerful, but yet very powerful, as we can hardcode on it its script and input/outputs. It's much safer too, and still allows us to build and re-use lambdas on code (generated by on compile-time).

So, new syntax:

NEW_LAMBDA 0102 0102 018b # one input int (01 0x02) , one output int (01 0x02), script INC 0x8b.
DUP
PUSH3
SWAP
CALL_LAMBDA
SWAP
CALL_LAMBDA

This script should return 3+1+1=5 😃

[erikzhang]

neo-project/neo#284

[erikzhang]

Maybe FunctionPointer?

[vncoelho]

@erikzhang, it also sounds good, could we point to a pre-defined function of the SC that access storage?

[erikzhang]

In VM, we only have the function pointers, which could be used in current contract only.

We can add a new SYSCALL System.Runtime.CreateCallback in NeoContract, which accepts the pointer, and creates a wrapper of the pointer. We name it as a Callback.

The contract can pass the callback to other contracts. So callbacks can be used across contracts.

[vncoelho]

This is quite powerful, @erikzhang. Let's discuss with @igormcoelho to check if it will be enough to cover the cases he thought.

[erikzhang]

One thing should be kept in mind. If A calls to B and sends a callback to B, and B calls the callback.

A -> B -> A(callback)

Can the callback function access A's static fields?

[erikzhang]

class A : SmartContract
{
    private static int MyInt;
    public static void Main()
    {
        MyInt = 2;
        Callback callback = CreateCallback(TestMethod);
        B.Main(callback);
    }
    private static void TestMethod()
    {
        //What is the value of `MyInt`?
    }
}
class B : SmartContract
{
    public static void Main(Callback callback)
    {
        callback();
    }
}

[igormcoelho]

Interesting Erik, my only concern is this (regarding callback):
PUSH1
PUSH2
ADD
Runtime.CreateCallback

Could we do this? If we can, this breaks all optimizations... if we could use direct parameters on this, ok, at least it's not a relative jump, but like this it's a relative jump, and we should not have that. Do you think direct parameters is feasible at least on this case?

I'd like to have both: direct parameter callback (on NeoContract) and Lambda item on neo-vm, since both can help in distinct manners. Let's see.

[erikzhang]

Could we do this? If we can, this breaks all optimizations...

Of course not. CreateCallback accept the function pointer only.

PUSHPOINTER Func1
SYSCALL "System.Runtime.CreateCallback"
RET
Func1:
//Do something
RET

[erikzhang]

PUSHPOINTER accepts address only. It doesn't pop the stack.

[igormcoelho]

I'm a bit confused, I'll try to think better on it.

[erikzhang]

PUSHPOINTER is necessary. Because it is lightweight, it can be used inside the contract.

And SYSCALLs accept parameters from stack only.

[vncoelho]

Func1 is set on SC, direct parameter, yes? Or dynamic from stack?

[igormcoelho]

SYSCALLs accept parameters from stack only

Ok, I'll assume this.

PUSHPOINTER is necessary

If we cannot have direct parameters on interop, we must find another way to do it... one way is to do it by a PUSHPOINTER opcode, that only accept direct parameter.

I think I get that, it must solve the problem 👍

Yet, if we may create a new stack item type to hold FunctionPointer, please consider the possibilty of using this same stack item type to hold a script, so that we may use it easily for lambda functions. This will help us implement lightweight invocations, not necessarily written on main script, but with local impact only. I think it's complementary to callbacks.

[shargon]

Can the callback function access A's static fields?

In c# yes, so my vote is yes... always will be danger if you send a callback to other contract, if you have your StorageContext static... you will be dead, but in other languages it works like this

[erikzhang]

We need two new opcodes:

enum OpCode : byte
{
    PUSHA = 0x0A, //Push the address of a function onto the stack.
    CALLA = 0x36, //Pop an address from the stack, and call the function.
}

[igormcoelho]

oh! very interesting point Erik, I was just going to mention that would be strange having only PUSH without any consumption method on neo-vm, for function pointers. Now it's fine. Agree with PUSHA and CALLA.
Do you think it's nice to use it for general calls now Erik? Or only for callback situations? I mean, is there any semantic advantage of doing:
PUSHA address
CALLA

instead of:

CALL address ?

[erikzhang]

Two instructions vs one instruction.

[igormcoelho]

If we want to simplify, we just disable static access on CALLA FunctionPointer, and it would be safe enough for the user to provide and create its own "lambda" on Entry (so just PUSHA and CALLA would suffice).
Later, let's choose a code for FunctionPointer, to receive it as NEP-3 parameter.

[vncoelho]

@erikzhang, I just saw that the title was changed.

This is not yet the functionality we were needing with lambda. Are you still considering a design for lambda calls?

[erikzhang]

I don't know what's the difference?

[shargon]

You mean, instead of absolute (to the start of NVM), put relative to the current position of executing code (like a jump)? Both work for me I guess, it should be just arithmetics.

No, offset from the beginning 0, it's an address but for prevent to have two different concept for "address" i think that we can call it offset.

[erikzhang]

@shargon It's not an offset. It is an absolute address.

[vncoelho]

@erikzhang,

I don't know what's the difference?

Lambda would allow us to cache a function on the stack (opcodes representing a desired function) coming from parameters of an Invoke:

Main(byte[] script, byte[] param)
LOAD script - loads in the stack of pointers
LAMBDA script with param - use the pointers to call the script with param as parameters

This is different than opeval, which would dynamically load.

[erikzhang]

Does it accept script from stack? Can it be passed to another contract?

[vncoelho]

Not from the stack, just temporary from parameters.

It can be called as a normal callback with desired params.

[erikzhang]

Then they are equivalent and can be converted to each other.

[erikzhang]

LOADSCRIPT <some script>
CALLA

PUSHA L1
CALLA
RET
L1:
//some script
RET

They are equivalent.

[vncoelho]

How to have a dynamic function on a predefined Smart Contract?

The case is when someone wants to Evaluate something with a dynamic function that comes from an invocation (older or send at the same time that caller wants to evaluate).

Is this possible with the current solution?

[vncoelho]

In your example, @erikzhang , can L1 behave dynamically? We would need to setup it.

[erikzhang]

I don't know what do you mean.

[igormcoelho]

They are equivalent.

I agree Erik, in this case yes... our initial goal was to be able to inspect the lambda code, and take a hash of it (to guarantee that user has passed correct script). Taking content of a NVM function is not possible, since we don't know where it ends...
But in the end, I think we can implement it using Witness @vncoelho ... we force user to send specific witness (on verification script), with specific hash, for every operation we want to verify... at least it will work for us.

[erikzhang]

If you don't allow the script from stack, and not allow to pass the pointer to another contract, they are equivalent.

[vncoelho]

I am not sure, @erikzhang, they look different to us...ahauahahau
L1 looks like to be a pointer to a deployed function that can not be modified, static function behavior in terms of logic.

[igormcoelho]

It's just the capability to inspect the Lambda content (or at least the hash of it), and verify input/output counts, that's the small difference for us.
LOADSCRIPT <some script>
INSPECTSCRIPT -> pushes script to stack, this was the original intention
CALLA

This was necessary, otherwise we cannot know what user actually attached for execution...

---

Other challenge: user expects a lambda function as parameter, that received two string and returns an integer, we cannot check that.

---

what we wanted:

Entry:
User implements some function, and loads it via PUSHA, sending to our contract.

Our contract gets the "function" (now just a function pointer), verify that it matches our "format", and allows its execution or not. Right now, we won't be able to do that using callbacks, because we won't be able to inspect its code, nor parameter/return counts (even for reading).

---

But Witness verification script can solve it, but much harder, and only for boolean results:

Entry:
User invokes our contract informing the script.

Contract verifies witness list, against the hash of the passed script. If they match, we allow execution of the contract. Limitations will be that no storage access, or result different than bool will be allowed. But we can manage @vncoelho

[vncoelho]

@erikzhang, it is ok, let's use the witness system for that. But the correct Lamda or OpEval would be the more correct and efficient way.

If we could allow the real lambda or an isolated opeval it would be great.
The witness is an opeval, but just with bool return.

[igormcoelho]

Oh, merged? nice!! Let's move on.

[shargon]

But the correct Lamda or OpEval would be the more correct and efficient way.

I don't like OpEval 😅

[vncoelho]

You are using it on all NEO invocations with Witness, perhaps you like 😅

[igormcoelho]

Congratulations @erikzhang at least now we have a good design to make filters operational 💪