Outline long term QA goals #2427
Assignees
Labels
scope: testing
Code review, testing, making sure the code is following the specification.
type: feature-request
New feature or request improvement
Comments
MSalopek
added
type: feature-request
p-offtermatt
changed the title
|
Maybe convert this to ADR so there's at least one deliverable attached to this task. |
12 tasks
|
Superceded and closed by: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This issue documents the QA process of gaia and interchain-security repositories.
It takes into account the fact that we are testing distributed systems and attempts to create an overview of the current state and explores future improvements.
The purpose of the QA process is catching bugs as early as possible. In an ideal development workflow a bug should never reach production.
A bug found in the specification stage is a lot cheaper to resolve than a bug discovered in production (or even in testnet).
Ideally, all bugs should be found during the CI execution, and we hope that no bugs will ever even reach the testnet (although nothing can replace actual system stress test under load interacting with users).
During development and testnet operation the most commonly found types of bugs are listed below. Such bugs can be discovered earlier with better tooling. Some of these bugs can induce increases in block times, chain halts or introduce an attack surface which is difficult to remove if other systems have started depending on that behaviour.
Current state of testing
Unit testing
Unit testing is employed mostly for testing single-module functionality. It is the first step in testing and often the most practical. While highly important, unit tests often test a single piece of code and don't test relationships between different moving parts, this makes them less valuable when dealing with multi-module interactions.
Unittests often employ mocks to abstract parts of the system that are not under test. Mocks are not equivalent to actual models and should not be treated as such.
Out of all the approaches used, unit testing has the most tools available and the coverage can simply be displayed as % of code lines tested. Although this is a very nice and very easy to understand metric, it does not speak about the quality of the test coverage.
Since distributed systems testing is a lot more involved, our reliance on simple unit testing tools should be minimized due to lack of useful feedback. Of course, sometimes unittests are still necessary and helpful, but in some cases where unit tests are not helpful, we should use e2e or integration tests as appropriate rather than taking pains to add unit tests only to artificially increase coverage.
Integration testing
With integration testing we test the multi-module interactions while isolating them from the remainder of the system.
Integration tests can uncover bugs that are often missed by unit tests.
It is very difficult to gauge the actual test coverage imparted by integration tests and the available tooling is limited.
In gaia and interchain-security we employ the
ibc-go/testingframework to test interactions in-memory.At present, integration testing does not involve the consensus layer - it is only concerned with application level state and logic.
End-to-end testing
In our context end-to-end testing comprises of tests that use the actual application binaries in an isolated environment (e.g. docker container). During test execution the inputs are meant to simulate actual user interaction, either by submitting transactions/queries using the command line or using gRPC/REST APIs and checking for state changes after an action has been performed. With this testing strategy we also include the consensus layer in all of our runs. This is the closest we can get to testing user interactions without starting a full testnet.
End-to-end testing strategies vary between different teams and projects and we strive to unify our approach to the best of our ability (at least for ICS and gaia).
The available tooling does not give us significant (or relevant) line of code coverage information since most of the tools are geared towards analysing unittests and simple code branch evaluation.
We aim to adapt our best practices by learning from other similar systems and projects such as cosmos-sdk, ibc-go and cometbft.
General approach evaluation
MBT tooling evaluation
Requirements for MBT tooling
The issue we are trying to avoid is having the specification disconnected from the testing tooling. This gives us a situation where we have a specification that does not reflect the actual state of the system and vice-versa.
Since the specification is just text, it is rarely used and goes unmaintained after a while, resulting in consistency issues and misleading specifications.
It is desired that the specifications be written in a dedicated and executable specification language, as opposed to written entirely in a general purpose programming language or entirely in text.
The main issue of specifications in a general purpose programming language is that they are not abstract enough to enable efficient model description. Programming langs are first and foremost for implementing and not for describing systems. Models do not chiefly care about the intricacies of implementation, like performance or storage concerns, but instead they care about concisely and accurately describing a system (such as a finite state machine).
The main issue of specifications in pure text is that the spec is not executable, thus not useful for generating traces or in the context of verification. This leads to consistency issues, outdated and unmaintained specifications, and so on.
Goals
The text was updated successfully, but these errors were encountered: