Should bulloak support fuzz tests?

[alexfertel]

Following the discussion in #19, the current proposal is to extend bulloak so that it when it finds something like:

pragma solidity [VERSION];

contract FooTest {
  modifier whenStuffCalled() {
    _;
  }

  function test_RevertWhen_StuffCalled() external whenStuffCalled
  {
    // The concrete test calls the fuzz test with a hardcoded value
    test_RevertWhen_StuffCalled(100);
  }

  function testFuzz_RevertWhen_StuffCalled(uint256 x) external whenStuffCalled
  {
    // then the test logic goes here
  }
}

We require that the fuzz test adheres to Foundry's best practices and that the name matches a non-fuzz test in the file.

---

My first thoughts are that if:

• bulloak should not support generating fuzz tests, then we should not enforce rules on fuzz tests when bulloak check is implemented. This is because:
  • The above feels wonky to me.
    • The user needs to choose a concrete value which defeats the purpose of having the fuzz test.
    • The non-fuzz test feels like unnecessary clutter.
  • If we choose not to support fuzz tests, but later do support it, staying compatible with the rule might make us less flexible.
• bulloak should generate fuzz tests as well, then we should tie it to the spec tree and not previous tests. This is to keep things consistent between concrete & fuzz tests, more explicit, and more flexible for users.

I'm down to support generating fuzz tests, whatever that might look like. However, I'm not a solidity dev & I'm not strongly opinionated on the above.

cc: @PaulRBerg @mds1

[jparklev]

I haven't worked with this .tree pattern before, so I don't yet have a strong opinion on whether or not fuzz test generation should be supported. With a high amount of uncertainty, I lean towards just having concrete tests, since that fits my conceptual model of this pattern, and if that's the case, I generally agree with this comment from Paul in #19:

I would prefer the tool to be less restrictive and allow any additional tests on top of what is defined in the tree file. It should still be possible to look at the order of the tree-related tests while ignoring the added-on stuff. The reason for this is general developer UX. It is often the case that I wrote a dummy test function just to check if something works as expected.

[mds1]

After thinking on this a bit I tend to agree:

To start, keep the tool less restrictive, with the idea being that for now bulloak is focused only on concrete tests and their spec. The rationale being (1) simplicity, and (2) your .tree spec should be thorough enough that concrete tests cover most/all edge cases. Developers can then add fuzz tests as needed.

Similarly, bulloak check would only consider concrete tests, and will allow additional tests of any kind. A --strict flag can remove this restriction by enforcing all concrete tests have a requirement in the .tree file, and that all fuzz tests have a name that matches a concrete test with the addition of the *Fuzz* part of the test name. Stateful fuzz (invariant) tests are ignored for now (they often are in a different file/folder anyway)