[feat] implement CompleteMerkleTree (#12)

* checkpoint

* checkpoint≈

* checkpoint with initial assembly getRoot func

* initial implementation of CompleteMerkle.sol

* remove unused file

* remove diff testing file for another PR

* update to complete merkle tests

* initial optimization pass at getProof

* implement differential and unit tests for complete merkle

* clean up old and dev comments

* rename tests to be very explicit

* make CompleteMerkle a MurkyBase; differential documentation updates

* add natspec to CompleteMerkle.sol

* rely on MurkyBase for verifyProof; update testing dependencies; --via-ir, optimize in pipeline

* README update for CompleteMerkle

* final clean up

.github/workflows/run_tests.yml

@@ -26,11 +26,17 @@ jobs:
       - name: Run Fuzzed Unit Tests
         run: forge test --no-match-path src/test/StandardInput.t.sol --fuzz-runs 10000
 
-      - name: Run Differential Tests
+      - name: Run Murky Differential Tests
         run: |
           npm --prefix differential_testing/scripts/ install
           npm --prefix differential_testing/scripts/ run compile
-          forge test --ffi --contracts differential_testing/test/DifferentialTests.t.sol --fuzz-runs 512 
+          forge test --ffi --contracts differential_testing/test/DifferentialTests.t.sol --fuzz-runs 512
+        
+      - name: Run Complete Differential Tests
+        run: |
+          npm --prefix differential_testing/scripts/ install
+          npm --prefix differential_testing/scripts/ run compile
+          forge test --ffi --contracts differential_testing/test/CompleteDifferentialTests.t.sol --fuzz-runs 512 
 
       - name: Run Standard Gas Snapshotting
         run: forge snapshot --gas-report --ffi --match-path src/test/StandardInput.t.sol

.gitignore

@@ -3,6 +3,7 @@ out/
 *.txt
 differential_testing/**/node_modules
 differential_testing/**/*.js
+differential_testing/**/*.json
 differential_testing/data/input
 
 !remappings.txt

README.md

@@ -3,9 +3,16 @@
 ![Slither](https://github.com/dmfxyz/murky/actions/workflows/slither.yml/badge.svg?event=push)
 
 
-Murky contains contracts that can generate merkle roots and proofs. Murky also performs inclusion verification. Both XOR-based and a concatenation-based hashing are currently supported.
+### Overview
+Murky contains contracts that can generate merkle roots and proofs. Murky also performs inclusion verification. A couple of default implementations are available out-of-the-box:
 
-The root generation, proof generation, and verification functions are all fuzz tested (configured 5,000 runs by default) using arbitrary bytes32 arrays and uint leaves. There is also standardized testing and differential testing.
+1. [`Merkle.sol`](./src/Merkle.sol) is the original Murky implementation. It implements the tree as a [Full Binary Tree](https://xlinux.nist.gov/dads/HTML/fullBinaryTree.html).
+
+2. [`CompleteMerkle.sol`](./src/CompleteMerkle.sol) is a merkle tree implementation using [Complete Binary Trees](https://xlinux.nist.gov/dads/HTML/completeBinaryTree.html). Some external libraries, particulary front-end or off-chain ones, use this type of tree.
+
+By default, both trees use sorted concatentation based hashing; you can also "bring your own" hashing function by inherting from [`MurkyBase.sol`](./src/common/MurkyBase.sol).
+
+The root generation, proof generation, and verification functions are all fuzz tested (configured 10,000 runs by default) using arbitrary bytes32 arrays and uint leaves. See [testing](#testing).
 
 > Note: Code is not audited (yet). Please do your own due dilligence testing if you are planning to use this code!
 
@@ -34,11 +41,6 @@ bool verified = m.verifyProof(root, proof, data[2]); // true!
 assertTrue(verified);
 ```
 
-### Notes
-* `Xorkle.sol` is implemented as a XOR tree. This allows for greater gas efficiency: hashes are calculated on 32 bytes instead of 64; it is agnostic of sibling order so there is less lt/gt branching. Note that XOR trees are not appropriate for all use-cases*.
-
-* `Merkle.sol` is implemented using concatenation and thus is a generic merkle tree. It's less efficient, but is compatible with [OpenZeppelin's Prover](https://github.com/OpenZeppelin/openzeppelin-contracts/blob/master/contracts/utils/cryptography/MerkleProof.sol) and other implementations. Use this one if you aren't sure. Compatiblity with OZ Prover is implemented as a fuzz test.
-
 ### Script
 `Merkle.s.sol` is implemented using `forge-std` for quick and simple interaction with the core contracts. The script reads from `script/target/input.json`, generates merkle proof using `Merkle.sol` and then outputs at `script/target/output.json`.
 
@@ -57,17 +59,12 @@ When measuring a change's performance impact, please ensure you are benchmarking
 forge snapshot --ffi --match-path src/test/StandardInput.t.sol
 ```
 
-Passing just standardized tests is not sufficient for implementation changes. All changes must pass all tests, preferably with 10,000+ fuzz runs.
-
-There is also early support for [differential  testing](./differential_testing/).
+Passing just standardized tests is not sufficient for implementation changes. All changes must pass all tests, preferably with 10,000 fuzz runs. Slither analysis must also pass.
 
 > * It's possible that an improvement is not adequetly revealed by the current standardized data. If that is the case, new standard data should be provided with an accompanying description/justification.
 
+There is also [differential testing](./differential_testing/).
+
 #### Latest Gas
 ![gas report](./reports/murky_gas_report.png)
-
-[Gas Snapshots](./.gas-snapshot) are run only on the standardized tests. See [Testing](#testing).
-
----
-#### TODO
-- [ ] \* Do a writeup on the use-cases for XORs.
+---

differential_testing/README.md

@@ -1,7 +1,7 @@
 ## Differential Testing
 Differential testing is used to compare Murky's solidity implementation to reference implementations in other languages. This directory contains the scripts needed to support this testing, as well as the differential tests themselves.
 
-Currently, the only reference implementation is adapted from the [Uniswap/merkle-distributor](https://github.com/uniswap/merkle-distributor) implementation. It is written in javascript.
+Currently two reference implementations are tested. The first is adapted from [Uniswap/merkle-distributor](https://github.com/uniswap/merkle-distributor), and the second is from [OpenZeppelin/merkle-tree](https://github.com/OpenZeppelin/merkle-tree). Both are written in Javascript.
 
 
 ### Setup
@@ -14,16 +14,25 @@ npm run compile
 
 ### Test the javascript implementation
 From the scripts directory:
+1. To test that the Uniswap/merkle-distributor differential test will work:
 ```sh
 npm run generate-root
 ```
 
+2. To test that the OpenZeppelin/merkle-tree differential test will work:
+```sh
+npm run generate-complete-root ../data/complete_root_input.json
+npm run generate-complete-proof ../data/complete_proof_input.json
+```
+
+If all commands output data without exception, then you are ready to run the differential tests. 
 ### Run the differential test using foundry
 Now you can run the tests.  
-From the root of the Murky repo, run:
+From the **root** of the Murky repo, run:
 ```sh
-forge test --ffi -c differential_testing/test/DifferentialTests.t.sol
+forge test --ffi --contracts differential_testing/test/
 ```
+> Note: The differential tests can take some time to run. An extended period of time without output does not necessarily indicate a problem.
 
 
 

differential_testing/scripts/generate_complete_proof.ts

@@ -0,0 +1,13 @@
+import { StandardMerkleTree } from "@openzeppelin/merkle-tree";
+import { ethers } from 'ethers';
+
+const fs = require('fs');
+const input_file = process.argv[2];
+const input = JSON.parse(fs.readFileSync(input_file, 'utf8'));
+const one_word = "0x0000000000000000000000000000000000000000000000000000000000000001"
+const leafs = input['leafs'].map((bytes32) => [bytes32, one_word]);
+const indexToProve = input['index'];
+const tree = StandardMerkleTree.of(leafs, ["bytes32", "bytes32"], { sortLeaves: false }); // NO DEFAULT SORTING LEAVES
+const proof = tree.getProof(indexToProve);
+process.stdout.write(ethers.utils.defaultAbiCoder.encode(['bytes32[]'], [proof]));
+

differential_testing/scripts/generate_complete_root.ts

@@ -0,0 +1,9 @@
+import { StandardMerkleTree } from "@openzeppelin/merkle-tree";
+const fs = require('fs');
+const input_file = process.argv[2];
+const input = JSON.parse(fs.readFileSync(input_file, 'utf8'));
+const one_word = "0x0000000000000000000000000000000000000000000000000000000000000001"
+const leafs = input['leafs'].map((bytes32) => [bytes32, one_word]);
+const tree = StandardMerkleTree.of(leafs, ["bytes32", "bytes32"], { sortLeaves: false }); // NO DEFAULT SORTING LEAVES
+process.stdout.write(tree.root);
+

differential_testing/scripts/generate_root.ts

@@ -16,5 +16,5 @@ const encodedData = ethers.utils.defaultAbiCoder.encode(["bytes32[129]"], [data]
 if (!fs.existsSync("../data/")) {
     fs.mkdirSync("../data/");
 }
-fs.writeFileSync("../data/input", encodedData);
+fs.writeFileSync("../data/merkle_input.txt", encodedData);