Analysis to decide Comprehensive Test Suite for XCM features

[asiniscalchi]

As a dev I want a comprehensive test suite for XCM transfers using beset practices from other parachains

ACCEPTANCE:

• The team adds to the backlog the necessary tasks (if any) to extend the current test in order to accommodate XCM tests

[ccubu]

Parachain	Test Tool/Approach	Link
Darwinia	No tests	-
Polimec	XCM Emulator	Polimec Integration Tests
Astar	XCM Simulator + Chopsticks (E2E)	Astar XCM Simulator, Astar E2E Tests
Acala	XCM Simulator + Chopsticks (E2E)	Acala XCM Simulator, Acala E2E Tests
Centrifuge	Tests without tools	Centrifuge Integration Tests
HydraDX	XCM Emulator + Chopsticks (E2E)	HydraDX XCM Emulator, HydraDX Chopsticks E2E
Zeitgeist	XCM Emulator + Chopsticks	Zeitgeist XCM Emulator, Zeitgeist Chopsticks
Watr Network	Equivalent to Zombienet	Watr Parachain Integration Tests
Unique Network	Equivalent to Zombienet (JS)	Unique JS Tests
t3rn	Zombienet Tests	t3rn Zombienet Tests
Subsocial	No tests	-
Sora	XCM Simulator	Sora XCM Simulator Tests
Robonomics	Custom tool (equivalent to Zombienet)	Robonomics Prototype
Polkadex	Custom XCM Simulator	Polkadex XCM Simulator
Phala	XCM Simulator	Phala XCM Simulator
Pendulum	XCM Emulator	Pendulum Integration Tests
Nodle	XCM Simulator & Emulator	Nodle XCM Simulator
Neuroweb	No tests	-
Manta	XCM Simulator	Manta XCM Tests
Litentry	XCM Simulator	Litentry XCM Simulator
KILT	XCM Emulator + Chopsticks	KILT XCM Tests
InvArch	XCM Simulator	InvArch XCM Simulator
Integritee	Chopsticks	Integritee Chopsticks Tests
Hyperbridge	XCM Simulator	Hyperbridge XCM Integration Tests
Hashed	No tests	-
Frequency	E2E (equivalent to Zombienet)	Frequency Testnet Faucet
Crust	No tests	-
Composable	Had XCM Simulator (now commented out)	Composable XCM Simulator
Bitgreen	No tests	-
Bifrost	XCM Simulator	Bifrost XCM Tests
Aventus	No tests	-
Ajuna	Chopsticks (manual)	Ajuna Manual Chopsticks
Moonbeam	XCM Simulator + Chopsticks (manual)	Moonbeam XCM Simulator, Moonbeam Chopsticks

XCM Simulator: 10 networks (Astar, Acala, Sora, Polkadex, Phala, Manta, Litentry, InvArch, Bifrost, Moonbeam) mainly used as unit testing
Zombienet integration tests: 7 networks (t3rn, Robonomics, Watr, Frequency, KILT, Unique Network, Zeitgeist)
Chopsticks: 7 networks (Astar, Acala, HydraDX, Zeitgeist, KILT, Integritee, Ajuna, Moonbeam)
XCM Emulator: 4 networks (Polimec, HydraDX, Zeitgeist, Pendulum) used as integration/e2e testing but is starting to being deprecated by the author of the library and also the community is shifting to use chopsticks to setup e2e

So as a summary it seems that the most used tool to integrate tests into CI is xcm-simulator as unit tests. For integration tests the community is still divided but the most used is Zombienet followed by Chopsticks, although multiple networks use chopsticks manually to test. XCM Emulator is not really used anymore and is being deprecated in favor of Zombienet/Chopsticks integrations.