crazy_train: fuzz based, parameterized, reproducible automated e2e fuzz testing for Loco

[jondot]

Since we are majorly CLI driven for productivity (as is Rails), We are modeling a tool, in the spirit of the original jepsen effort, in the sense of providing:

• Randomized/fuzz execution of the Loco CLI for discovering unforeseen sequence of steps and / or parameters that users run and lead to unexpected errors
• Reproducible test plan runs

This is in favor of making sure that users can "drive" through the CLI in any combination of commands.

The Rails CLI had 20 years to evolve, and we now want to build a piece of infrastructure that will make us "speed up" that evolution for Loco.

crazy_train will create endless, randomized run plans with predefined steps which map to the Loco CLI. Example:

gen_model [parameters]
db_reset
db_seed 
build 
routes
start (+check listen, then kill)

which forms a "language" of steps, each step can randomize and track its various parameters.

crazy_train will also completely replace many of our CI testing flows, by providing "predefined" plans (which are not randomized, but hand-edited and executed)

This can answer questions such as:

• what happens if we run seed twice?
• what happens if we generate thousands of fields in the generate model command?
• what happens when we use chinese (wide unicode) for names of fields in scaffold?
• what happens with the combination: reset, migrate, reset, seed, reset?
• is there any combination of commands we're not seeing right now, that can create an unexpected error?
• etc..

credit for the name of this tool: #790

Commands: cargo xtask fuzzy ...

• New template
• Scaffold
• Generate model
• Generate controller

[kaplanelad]

https://github.com/loco-rs/crazy-train