From 5c4f4ac305498237ee6c02a882aa35fa584b7ef4 Mon Sep 17 00:00:00 2001 From: Alexander Peters Date: Wed, 2 Oct 2024 13:59:01 +0200 Subject: [PATCH] docs: Add initial doc for system tests (#22002) --- docs/build/building-apps/06-system-tests.md | 58 +++++++++++++++++++++ docs/build/building-modules/16-testing.md | 30 ++++------- 2 files changed, 67 insertions(+), 21 deletions(-) create mode 100644 docs/build/building-apps/06-system-tests.md diff --git a/docs/build/building-apps/06-system-tests.md b/docs/build/building-apps/06-system-tests.md new file mode 100644 index 000000000000..eb6d61ffea46 --- /dev/null +++ b/docs/build/building-apps/06-system-tests.md @@ -0,0 +1,58 @@ +--- +sidebar_position: 1 +--- + +# System Tests + +System tests provide a framework to write and execute black box tests against a running chain. This adds another level +of confidence on top of unit, integration, and simulations tests, ensuring that business-critical scenarios +(like double signing prevention) or scenarios that can't be tested otherwise (like a chain upgrade) are covered. + +## Vanilla Go for Flow Control + +System tests are vanilla Go tests that interact with the compiled chain binary. The `test runner` component starts a +local testnet of 4 nodes (by default) and provides convenient helper methods for accessing the +`system under test (SUT)`. +A `CLI wrapper` makes it easy to access keys, submit transactions, or execute operations. Together, these components +enable the replication and validation of complex business scenarios. + +Here's an example of a double signing test, where a new node is added with the same key as the first validator: +[double signing test example](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/fraud_test.go) + +The [getting started tutorial](https://github.com/cosmos/cosmos-sdk/blob/v0.52.0-beta.1/tests/systemtests/getting_started.md) +contains a step-by-step guide to building and running your first system test. It covers setting chain state via genesis +or +transactions and validation via transaction response or queries. + +## Design Principles and Guidelines + +System tests are slower compared to unit or integration tests as they interact with a running chain. Therefore, certain +principles can guide their usage: + +- **Perspective:** Tests should mimic a human interacting with the chain from the outside. Initial states can be set via + genesis or transactions to support a test scenario. +- **Roles:** The user can have multiple roles such as validator, delegator, granter, or group admin. +- **Focus:** Tests should concentrate on happy paths or business-critical workflows. Unit and integration tests are + better suited for more fine-grained testing. +- **Workflows:** Test workflows and scenarios, not individual units. Given the high setup costs, it is reasonable to + combine multiple steps and assertions in a single test method. +- **Genesis Mods:** Genesis modifications can incur additional time costs for resetting dirty states. Reuse existing + accounts (node0..n) whenever possible. +- **Framework:** Continuously improve the framework for better readability and reusability. + +## Errors and Debugging + +All output is logged to `systemtests/testnet/node{0..n}.out`. Usually, `node0.out` is very noisy as it receives the CLI +connections. Prefer any other node's log to find stack traces or error messages. + +Using system tests for state setup during debugging has become very handy: + +- Start the test with one node only and verbose output: + + ```sh + go test -v -tags=system_test ./ --run TestAccountCreation --verbose --nodes-count=1 + ``` + +- Copy the CLI command for the transaction and modify the test to stop before the command +- Start the node with `--home=/tests/systemtests/testnet/node0//` in debug mode +- Execute CLI command from shell and enter breakpoints diff --git a/docs/build/building-modules/16-testing.md b/docs/build/building-modules/16-testing.md index f2fafa36fd9d..65ed52a8a142 100644 --- a/docs/build/building-modules/16-testing.md +++ b/docs/build/building-modules/16-testing.md @@ -86,38 +86,26 @@ https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/tests/integration/bank ## Simulations -Simulations uses as well a minimal application, built with [`depinject`](../packages/01-depinject.md): +Simulations fuzz tests for deterministic message execution. They use a minimal application, built with [`depinject`](../packages/01-depinject.md): :::note -You can as well use the `AppConfig` `configurator` for creating an `AppConfig` [inline](https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/slashing/app_test.go#L54-L62). There is no difference between those two ways, use whichever you prefer. +Simulations have been refactored to message factories ::: -Following is an example for `x/gov/` simulations: +An example for `x/bank/` simulations: ```go reference -https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L406-L430 +https://github.com/cosmos/cosmos-sdk/blob/release/v0.52.x/x/bank/simulation/msg_factory.go#L13-L20 ``` -```go reference -https://github.com/cosmos/cosmos-sdk/blob/v0.50.0-alpha.0/x/gov/simulation/operations_test.go#L90-L132 -``` - -## End-to-end Tests +## System Tests -End-to-end tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). -They must test the whole application flow, from the user perspective (for instance, CLI tests). They are located under [`/tests/e2e`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/e2e). +System tests are at the top of the [test pyramid](https://martinfowler.com/articles/practical-test-pyramid.html). +They test the whole application flow as black box, from the user perspective. They are located under [`/tests/systemtests`](https://github.com/cosmos/cosmos-sdk/tree/main/tests/systemtests). - -For that, the SDK is using `simapp` but you should use your own application (`appd`). -Here are some examples: +For that, the SDK is using the `simapp` binary, but you should use your own binary. +More details about system test can be found in [building-apps](https://docs.cosmos.network/main/build/building-apps/06-system-tests.md) -* SDK E2E tests: . -* Cosmos Hub E2E tests: . -* Osmosis E2E tests: . - -:::note warning -The SDK is in the process of creating its E2E tests, as defined in [ADR-59](https://docs.cosmos.network/main/build/architecture/adr-059-test-scopes). This page will eventually be updated with better examples. -::: ## Learn More