[JT-NM Tested] Unexpected inter-test dependencies

[garethsb]

Some reports from JT-NM Tested August 2022:

• IS-04-01 test_21 fails when we fail test 15 and 16 on failover between registries. pass when not testing 15 and 16
• why running some tests a second time make them pass, should we allow/forbid this?
• We had a case where IS-05-01, a device failed test_27, test_28, test 29, and test_30 but this also affected test_36 or test_37 but can pass those 2 last test if run independently.

[garethsb]

Fundamentally, devices are stateful, and the NMOS APIs are therefore stateful. Encountering situations where the results of previous tests affect subsequent ones is unsurprising, even if we do our best to minimize the interdependence.

IS-05-01 test_27-30 check scheduled activation. One suspects that one of the failed scheduled activations 'locked' one or more Sender or Receiver, which thus prevented bulk activation, perhaps correctly indicated by 423 Locked for the specific Sender/Receiver. If so, the error message for test_36/37 could at least indicate that. Following a general rule, I'm not in favour of the testing tool unlocking all Senders/Receivers before issuing the bulk activations, since this then tests two things rather than one.

IS-04-01 test_21 advertises a new Registration API, whereas test_15 and test_16 gather information during the same do_registry_basics_prereqs call that most of the other Registration API-related test cases do. If failover has failed, probably the DuT doesn't see the new advertisement in test_21 either. Perhaps this could be flagged in the error message too...

[peterbrightwell]

Architecture Review Group review: place on backlog