THIS IS THE ORIGINAL README of the repo, for the comparasion to Certroa see: https://github.com/Certora/fuzz-vs-fv-showdown/blob/master/README.md
A comparison of solidity fuzzing tools Foundry, Echidna & Medusa. This challenge set is not intended to be an academically rigorous benchmark but rather to present the experiences of an auditor "in the trenches"; the goal is finding the best performance "out of the box" with as little guidance & tweaking as possible.
Ensure you are using a recent version of Foundry which includes PR6530.
Install a more recent build of Echidna.
Compile the latest Medusa source code as it is experimental and under active development.
Configure solc-select for Echidna & Medusa:
solc-select install 0.8.23
solc-select use 0.8.23
To compile this project:
forge build
Every exercise has a basic
some optionally an advanced
fuzz configuration for Foundry, Echidna & Medusa. The basic
configuration does not guide the fuzzer at all; it simply sets up the scenario and allows the fuzzer to do whatever it wants. The advanced
configuration guides the fuzzer to the functions it should call and helps to eliminate invalid inputs which result in useless fuzz runs.
In basic
configuration Foundry, Echidna & Medusa are able to break the simpler invariant but not the more valuable and difficult one. In advanced
configuration all 3 fuzzers can break both invariants. All 3 fuzzers reduce the exploit chain to a very concise & optimized transaction set and present this to the user in an easy to understand output. As a result they are tied and there is no clear winner.
Echidna in basic
configuration can frequently break both invariants while Foundry in basic
configuration can sometimes break the easier invariant but never the harder one. Foundry using advanced
configuration is able to break both invariants if given an extreme amount of targeting. Medusa in basic
configuration can always break both invariants and achieves this much faster than Echidna, making Medusa the clear winner.
Foundry, Echidna & Medusa in basic
mode are able to easily break the invariant, resulting in a tie.
In basic
configuration Foundry, Echidna & Medusa are all able to break the easier invariant but not the more difficult one. All Fuzzers are able to provide the user with a minimal transaction set to generate the exploit. Hence they are tied, there is no clear winner. Please note that the fuzz solvers for this challenge are not able to be publicly released at this time.
In basic
configuration Foundry & Echidna can only break the easier and more valuable invariant which leads to a Critical exploit but not the harder though less valuable invariant which leads to a High/Medium. However Medusa is able to almost immediately break both invariants in unguided basic
mode, making Medusa the clear winner. Please note that the fuzz solvers for this challenge are not able to be publicly released at this time.
Both Echidna & Foundry are unable to break the assertion in this stateless fuzzing challenge. Medusa used to be able to break it almost instantly but has regressed in performance after recent changes and is now unable to break it. Hence there is no current winner.
Foundry & Echidna are able to break the assertion in this stateless fuzzing challenge, but Medusa is unable to break it.
All 3 Fuzzers configured in advanced
guided mode attempted to break 16 invariants on Beta Finance Omni Protocol. Medusa is typically able to break 2 invariants within 5 minutes (often much sooner on subsequent runs) though on the first run can take a bit longer. Echidna can sometimes break 1 invariant within 5 minutes and Foundry appears to never be able to break any invariants within 5 minutes. Hence Medusa is the clear winner. The fuzzers written for this challenge were contributed to Beta Finance.