Add operation OP_EVAL for direct execution of opcodes

[igormcoelho]

Dear all, I've been thinking that a very useful operation would be to have the ability of executing a given list of opcodes directly (inside a smart contract). It would be quite similar to the current APPCALL, but instead of calling a hash or interop string, it would be a bytearray of operations. I imagine there must be reasons for not having it now (like preventing the creation of very generic deployed contracts, or security issues when more than one value is returned on stack), but we can discuss what are the issues and possible fixes, possibilities for this operation cost (it could be expensive to disencourage random uses), so let me present the huge advantages :)
(1) After seeing the new vision for NEO 3.0, perhaps without UTXO and only NEP-5 assets, I imagine it would be very useful to have "NEP-5 addresses" that perform different types of ownership verification. For example, multisig addresses, time-lock addresses, all of them could possibly be recorded in this "new" NEP-5 as raw contracts, and the verification of withdraw could be directly performed by directly invoking this new OPCODE_EXECUTE [operation_list] (pretty much what we have already for UTXO assets)
[edit] this is not needed for nep5, only for advanced metaprogramming.

(2) It would natively support pretty much of we have right now, and give much more amazing use cases for NEO... it's like supporting anonimous lambda functions, just imagine that :)

=========

Current proposal is to implement it directly on NeoVM. InteropCalls will be blocked, also method Calls, Dynamic Invoke, or recursive Op_eval. It will be nearly Turing-incomplete (besides the JUMP that will still be allowed).

[erikzhang]

Do you mean like the eval() function in JavaScript?

[igormcoelho]

Just like that... :) if the stack was really isolated, differently from a CALL where the instruction list is maintained (only the instruction counter changes; and the context copied). We would need to copy the "new instruction list" to a "instruction list stack", and after execution, it returns to the original instruction list.
Code injection won't be a problem if a limited number of elements are returned to stack (one, two or many). It could be like inception movie, a dream inside a dream, inside a dream :D

Example validating pubkey 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a:

...
PUSHBYTES abababababbabababa... SIGNATURE FROM PARAMETER OR CALL
PUSHBYTES 21031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4aac
INCEPTION   # (will execute opcodes: 21 031a6c6fbbdf02ca351745fa86b9ba5a9452d785ac4f7fc2b7548ca2a46c4fcf4a / ac)
JMPIF ... # use stack top to validate data (passed as parameter)
...

[erikzhang]

I think there is a proposal in BIP for OP_EVAL:
https://github.com/bitcoin/bips/blob/master/bip-0012.mediawiki

There are some discussions on bitcointalk:
https://bitcointalk.org/index.php?topic=46538

[erikzhang]

Thinking it over, this is actually some form of dynamic invocation, just more lightweight.

[igormcoelho]

I'll take a deep look at the discussions Erik, but with one thing in mind: as far as I know, Bitcoin is not Turing-complete, right? That makes a lot of difference in this scenario... without loops, I personally don't see much of use in Bitcoin, but on Neo it's even more important than dynamic invoke (in my opinion).
If Neo wants to avoid people just deploying dummy contracts for using OP_EVAL, it is easy to "limit" the power of OP_EVAL (limiting script size, valid operations, ...), in order to fit it in the economic model. But I have seen many scenarios (even in scichain.org) that we would need many auxiliary contracts, but it's impossible to deploy them all, for all possible situations... this thing would be very helpful. And if UTXO is banned, then it will become super important for doing amazing things with NEP-5 addresses. I'll inform me a bit more, then I come back.

[edit] I was wrong, this is not needed for nep5, only for advanced metaprogramming.

[igormcoelho]

Ok Erik, took some time to study BIP-12 (OP_EVAL). The intention of Gavin Andresen in 2011 was to create new types of standard "scriptPubKeys" on bitcoin, that is, new ways of spending bitcoin, besides private-public key validation. On 2012 he suceeded to pass BIP-16 (P2SH - pay to script hash), which accomplishes similar behavior found on Neo Verification trigger, so BIP-12 was deemed useless, because everything OP_EVAL did could be done using P2SH.

What's the difference from Neo? We really need OP_EVAL, even having Verification trigger. The situation is that we have Turing complete computation, and this abstract allows us to have Turing machines inside turing machines.... so there will be infinitely many more situations that this will be useful. I guess we can write a NEP on it if you want, I will organize my arguments in two points now: (1) one important use case (2) how to implement and how to deal with security/safety concerns.

(1) Important use case: automated redeem script inside NEP-5 tokens
Currently, NEP-5 is implemented in a turing machine ecosystem (Neo), but it does not allow that users have this same power. That is why we cannot have multisig contracts working as NEP-5 tokens, or even password locked contracts. If we have OP_EVAL, the funds will be locked not in a "scripthash", but in a "script" (with more than 20 bytes). So, a transfer may pass parameters: "script" owner, "script" destination, value, and an array of values, that will be used to validate the transfer transaction (invocation). So, for compatibility reasons, 20 byte "scripts" could be matched the way they are now (against CHECKSIG), but could also work as: PUSH( 21 + pubkey + CHECKSIG ) + OP_EVAL. That would allow the construction of an imense number of validators, as long as the final stack contains a single boolean value (otherwise it could fail, in NEP-5 scenario).
[edit] this is not needed for nep5, only for advanced metaprogramming.

(2) how to implement
The OP_EVAL could follow exactly BIP-12 proposal initially, it takes a script passed as a bytearray on top of the stack and executes it. The author does not give details, but knowing Neo, I believe we need to instantiate a new execution stack with a new set of instructions (script), totally independent from the original code (to maintain total context isolation). That is not hard, and will keep the scripts to a limited size of 1024 bytes (max size of bytearray I guess). It's a limitation, but should be enough for most practical applications, regarding these initial use cases in mind. One big concern in my mind is that contract Storage could be "invaded" by a hacking OP_EVAL script. I believe it's nearly impossible to close all gaps here, so in my opinion, a safe use of OP_EVAL would require blocking StorageContext Put (and global variables if they exist), keeping the execution in a limited scenario. Recursion in OP_EVAL would not be possible, because it's not named, but still, someone could invoke 1024 OP_EVAL (one inside the other). I guess that's not a problem, as execution stack size is already limited, and this operation should cost some GAS. Finally, taking into account the nature of executing without any collateral effects, this is the pure spirit of functional programming and anonymous/lambda functions, so I believe this would be the most natural way of implementing OP_EVAL in a high level language. Something like:

delegate int del(int i);  
static void Main(string[] args)  
{  
    del myDelegate = x => x * x;  
    int j = myDelegate(5); //j = 25  
} 

Should become something like:

PUSH "GETITEM ... MUL ..." # regular instructions generated by function int f(x) { return x*x; }
OP_EVAL
SETITEM ... # continue regular code

Question is, how hard is it to code it on C# Neo compiler? xD
Even without a proper compiling support, we could already do that "manually" (with partial compiler support) right now (as regular independent Verification contracts), to achieve improved NEP-5 "addresses" for example.

[shargon]

Very dangerous OpCode, i don't like :P

[localhuman]

Agree with @shargon. Also, it is possible to do multisig-like Smart Contracts with the current verification routine.

[igormcoelho]

@localhuman the proposal is to allow a NEP-5 address to be a multisig, or even to allow more complicated token spending logics that verification allows, but NEP-5 does not (Verification can be nearly abolished together with utxo, we need a programmable alternative inside contracts). The idea is to give control to the token owner in how it should be spent.

[edit] I was wrong, you are right my friend @localhuman, nep5 can do anything.

@shargon don't be afraid my friend, I thought about it a lot, trust me xD hahhaha if we disallow storage writes completely (and even function CALLs/dynamicInvokes), and execute in a separate stack (such as a regular CALL, but with different "script"), there are no attacks possible (JMP cannot go out of scope, infinite loops will spend gas, recursive behavior is not possible without CALL, no dynamic invoke...), it's very limited, but very powerful anyway. If you are still afraid of the returning elements in the stack, we can create OP_EVAL1, which allows a single element in stack in the end (that's also enough for all use cases I imagined as a regular Lambda function).

[localhuman]

If you want an NEP5 contract to be multisig, you can attach extra script hashes to the TX Attrs and run them all through CheckWitness, requiring X amount of sigs. This can be performed in the Verification trigger or Application trigger.

[igormcoelho]

But it's not just MultiSig, it's all different kinds of logic that verification permits today (and we won't have anymore). We cannot predict all of them beforehand... for example, how to enable a time-lock or password-lock contract inside NEP-5? Time-lock is a quite real need for many purposes. The current ICO template does not even allow multisig (I guess it should), but other features will keep failing, and preventing us to abolish utxo.

[edit]: I was completely wrong, ignore this comment :)

[shargon]

@igormcoelho Give me a eval and i could take your coins muahaha. Now seriously, is very very dangerous operation, need to be full isolated. I don't like it xD

[igormcoelho]

Ok @shargon, as soon as I have time, I'll implement this feature, and store some coins in neocompiler.io network. You will have one week to steal the coins, and if you cannot, you will have to upvote the feature and put a heart in this comment xD

[vncoelho]

kkkkkkkkkkkkkkkkkkkkkk, you are the best, @shargon and @igormcoelho. aheuahueahuea
I will convert this into a hash and put into the blockchain.
1 Neo and 1 GAS will be awarded by me to the winner.

[shargon]

Maybe you @igormcoelho do the things in a safe way, and use well this OpCode, but this not means that the opCode is safe, you can use a knife for eat meat or for kill people, but give the knife to the people, and maybe someone will be murdered 🔪

[shargon]

The question is: is needed?

[igormcoelho]

I get your point @shargon, but I really think that's needed to implement neo and gas as nep-5. Perhaps there's another way, that I haven't seen yet.

[edit] I was wrong, you were right @shargon, nep5 does not need this. please ignore this comment :)

[shargon]

I talk about the eval opcode, neo and gas will be a nep5 i think that this is fine, but we don't need the eval opcode for this, right?

[igormcoelho]

Well, for me its the only way I think @shargon, otherwise we will lose passwordlock, timelock and other elaborate account types. The default nep5 doesnt even support multisig... we need to fix that too.

[edit] I was wrong, this is not needed for nep5, please ignore this comment :)

[erikzhang]

The default nep5 can support every account types without OP_EVAL.

[igormcoelho]

ok @erikzhang , if you say that, I believe :)

Anyway, I still think that's useful for giving more power to smart contracts, when I'm able to code that we can discuss if it's worth going on or not.

[vncoelho]

@erikzhang, how is the evolution of the idea surrounding the OP_EVAL?
1- We see it as a very important operation for several applications.

2 - Recently, we also saw an application that provides the possibility of Quantum Security, in the sense that each Account can choose its desired encryption.
On the other hand, @igormcoelho told me that cryptographic curves are very expensive for the Neo-VM and should be probably implemented natively in the Neo Core library.

Should we proceed with a draft of this new Operation?

[igormcoelho]

You are right Erik (and @localhuman), NEP-5 can support every possible account type, I took some time to fully understand that. It's such an amazing design, I learn new things every day. Still, this feature is very important for metaprogramming, for games and other mutable rules based contracts. It can be implemented safely,but I prefer to implement safe invocations on NeoVM first , that will help

[erikzhang]

If we implement OP_EVAL, I am afraid that the contract to use it in the future will start to worry about endless injection vulnerabilities.

[vncoelho]

I am afraid as well but I trust you, @igormcoelho, @shargon, a little bit on me, etc...
It is a very important feature and good projects will aggregate value to the Ecosystem by using it.

I believe we can do some encapsulation for isolating it in safe manner, even if we need a dedicated Application Layer for it.

[shargon]

Is very hard to control what code do you want to eval, so the injection vulnerabilities should be very easy if you use OP_EVAL

[vncoelho]

Nothing is impossible.
We know the power of it, @shargon, that is why we need it and we believe several other will in the future.

[igormcoelho]

@jsolman , just to clarify this point by point, to leave no misunderstandings here since I'm a huge defender of Neo economic model and I would never propose something do could do harm to it :)

OpEval allows anyone to write code without deployment of a contract that makes use of 10 GAS for application code that could potentially write to storage without any deployment cost. This is contrary to the economic model of the platform. Isn’t that a problem?

No, it's not a problem. Let's suppose OP_EVAL is fully unconstrained and has zero gas cost, only operations cost (note that we haven't talked about its pricing before on this discussion. we could possibly add expensive GAS cost to OP_EVAL execution, what breaks this argumentation, but let's suppose it's zero). This was the first scenario I considered months ago, which is not good for Neo for two reasons: (1) it's unsafe and possibly lead to many problems; (2) as argued, may give the possibility of someone re-deploying a contract "for free". Ok, number (1) won't happen because we will limit it strongly, so let's talk about (2). If unconstrained OP_EVAL is allowed + storage, someone could indeed code a program which just reads code from storage and executes it, however, keep in mind that possibly zero applications on blockchain could take advantage of that, because it basically says that your software may change constantly (a constant migration) which makes no sense from the "trust" point of view of dApps. Even Migrations are undesired, or unnacceptable, when you design a trustless application. Would you buy a token which is written over an OP_EVAL in a storage? I would never :)

On the other hand, OP_EVAL is clearly an interesting feature, as it has been proposed to Bitcoin in its early days... and it was denied. I took some time to understand why, and the reason is that Bitcoin adopted a similar verification scheme that we have on Neo, which is basically an OP_EVAL that returns a boolean. So, we have OP_EVAL already on Neo, and it costs zero (having no storage access, but allowing interop, which is more than I want). The small difference from Verification Scripts to what we want here, is simply the capability to return other values than Bool, and process these values inside an Application (or even Verification). I liked a lot the term "lightweight dynamic invoke" that Erik used, so we could say it's an "interop/call-free lightweight dynamic invoke".

Anyway, metaprogramming makes a lot of sense for Neo, much more than Bitcoin (that already has its Turing-incomplete verifications with no storage), and even Ethereum (that has Turing-complete computations with storage, but are bound to Solidity language). On Neo, we can have a beautiful and safe metaprogramming working, on C#, Python, which is very unique. This discussion is quite fruitful, and although we need this feature soon, we will wait to propose an implementation only when community is positive regarding its adoption, or perhaps do not implement it at all, if it's deemed harmful or unnecessary.

[vncoelho]

Great description, broda.
Let's encapsulate this and put fire on the bomb.
AHueaheuaea

After a first template Pull Request we gonna be able to discuss further those minor details about safety.
In addition, other discussions are going to appear during the journey.

[shargon]

My two cents:

• This should be moved to neo-vm
• This should be a syscall
• I don't like it xD

[igormcoelho]

@shargon my guess here is to allow safe execution inside a Turing-incomplete environment: neo-project/neo-vm#151

[igormcoelho]

This should be moved to neo-vm
This should be a syscall

😕
How can it be both at the same time? hahaha

[lock9]

Let me close this! Please, I hate this issue! 😂
We don't need this 😬

[vncoelho]

@lock9, this is high priority to us, we are going to do this anyway.
In this sense, if it is done here in the core much better for everyone, in my opinion. Thus, I believe that this flexible and powerful operation should be here.

[vncoelho]

@shargon, regarding your last comment.
Some of it was already accomplished.

• It will be a syscall as the other default non-essential computational operations
• thus, neo-vm will not consider it anymore (similarly to what we did with the other operations)
• It is not a matter of like, the important thing is what it enables us to do...aehuaheuaea ajajaj

I think that after limiting the Verification (or defining its scope for NEO3) we are going to be able to design the limit encapsulation for this call.

[vncoelho]

@jsolman, regarding your last comment, I think that the economic model should be more focus on operations than in the deploy itself, thus, the scope of the op_eval will generate the desired sys_fees.

[igormcoelho]

@vncoelho I don't know anymore brother hahahah I've seen so many complaints everywhere regarding eval() operations, that I don't know anymore if this is a desired practice.
One thing is certain: this is useful to us. But let's think about it...

[vncoelho]

How to eval a simple Travelling Salesman Problem regarding two different objective functions?

• One for distance and one for travelling time?
• One for distance and one for the energy cost of that trip on electric vehicles?
• One for distance and one for amount of Sunlight in that route?

Maybe as a verification as we previously discussed.
However, it will become impracticable in terms of costs I think, as well as for OnChain Cryptographic functions.

[lock9]

This "EVAL" thing is like non-typed programming languages: they look a good idea at first, but they are not, and sooner than later, you will pay for this bad decision 😂

[shargon]

I think that we can close this, many of us are disagree with eval opcode :P , at least i will remove it from the neo3 milestone

[vncoelho]

I think that remove from Milestone is plausible, @shargon.
But as I said, we are going to design this op_eval because we need it and it will happen soon.

[igormcoelho]

I finally agree with you, this is a bad idea.

[igormcoelho]

Vitor, as long as deploy is very cheap without storage, problem is solved for us. Theres no reason for expensive deploys, lets fix that.

[erikzhang]

How to implement lambda?

[lock9]

@igormcoelho could you help to answer Erik? I have no idea

[igormcoelho]

How to implement lambda?

I think we should compile lambda module as a regular function, and pass its address to function.
We could experiment a little bit on how Func<> is compiled, and we try to reuse that knowledge.

[igormcoelho]

I think that neo-project/neo-vm#190 may be a much better (and beautiful) solution ;)