Allow omission of conformance class / conformance tests

[ronaldtse]

One major hesitation on standards adopting ModSpec is the requirement that every requirements class / requirement needs to be paired with their corresponding conformance class / conformance test.

Most of the time, the standards author will just "wing" the conformance part by making a generic conformance test, or just a rephrase of the original requirement, which means the result is not useful.

The new ModSpec can recognize that Requirements and Conformance can potentially be in separate types of documents.

In the ISO "management systems" world, they are:

• Standards -- provides requirements
• Audit manuals -- provides audit steps

If I can use management system standards as an example:

• ISO/IEC 27001 only states requirements
• Certification Bodies develop their own "Audit manuals" which are conformance tests
• Some standards like SS 584 (Singapore's cloud security standard) state both requirements ModSpec and audit steps (corresponding to ModSpec requirements and ModSpec conformance tests)

[cmheazel]

We want to avoid requirements which are not testable. Including the conformance tests should help enforce this discipline. Rather than give sloppy authors a get out of jail free card, we should emphasize the role that the conformance tests play in creating good standards. After all, a standard which is not testable is not much of a standard.

[ronaldtse]

After all, a standard which is not testable is not much of a standard.

@cmheazel I agree with this philosophy, that a standard providing both the requirement statements and conformance tests that validate those statements that would be ideal.

However there are 4 additional considerations:

1. If there are many requirements in a standard, it is reasonable to have the conformance tests in a separate part of the standard.
2. If there is a gap between the development of requirements and conformance tests, the requirements document may be published first.
3. If the group of setting requirements and the group for conformity assessment are different, then it is natural that they reside in different documents.
4. There are standards that only have requirements, and standards that only have conformance tests.

While philosophically it is clean to mandate that requirements must come with associated tests, it is not always desired or practical.

Instead, why not keep this requirement of bundling requirements and conformance tests only within OGC as a policy, and other organizations can use ModSpec with some flexibility?

[cmheazel]

Even in an Implementation Standard (Specification) the tests are rather abstract. The true work of writing conformance tests lies in creating the Executable Test Suite (ETS). The Abstract Test Suite provides sufficient information that a developer, who did not participate in the development of the standard, can still create a valid ETS for that standard. Witness GeoTIFF 1.1.

1. Conformance Tests are documented in the Abstract Test Suite, a separate annex in the Standard
2. If you are writing a requirement, then you had better be thinking about how to test it. Otherwise you are at high risk of creating an invalid, untestable, requirement. Note: this does not require the level of detail we see in an ETS.
3. How does the group writing requirements communicate their intent to the conformance group? The ATS provides that link.
4. There are incomprehensible standards, ambiguous standards, and unimplementable standards. We shouldn't encourage them.

Also, I find that in creating the ATS I always identify errors in the requirements. Writing the ATS is a good quality control practice.

[cmheazel]

Perhaps the ModSpec should distinguish between an abstract conformance test and an executable test. Analogous to the distinction between a test plan and a test procedure.

[cnreediii]

@cmheazel Sure. Could add ETS to the T&D section.

[ronaldtse]

@cmheazel I agree with the answers 1-4 as best practice. That said, it is not always possible.

Requirements are subject to interpretation and elaboration

Requirements are also of many forms.

This is an example from ISO 9001:2015, 5.2.1.

This is demonstrated in the two requirements below. The first requirement is unambiguous and self-contained, but the second requires a lot more context:

• "The <root> element shall contain at least one element <name>".
• "The quality policy shall be available and be maintained as documented information."

The second requirement here is not easily made "executable". There is a spectrum of requirements that are: easily executable, not easily executable, and also "not executable".

For the second requirement, without further breaking down the requirement and making it bite sized, it is impossible to develop any "executable test" because many words in there are subject to interpretation.

NOTE: Of course, it depends whether ModSpec is meant to handle things like this.

Conformance tests are also subject to interpretation and elaboration

We need to recognize that a conformance test has the following attributes:

1. Level of abstractness / implementation.
   • How readily can this test be executed?
   • This is a degree, since gradually removing pieces from an implementation gives you a more abstract test.
2. Level of conformance.
   • How deep does the test go?
   • The more comprehensive the checks the higher the confidence that the implementation complies with the standard.
3. Interpretation.
   • In IEC, they publish a special kind of document called the "ISH", "Interpretation sheet". It provides additional guidance on what the requirements mean from a standard.
   • This also indicates that a requirement can be subject to "interpretation", and depending on the "interpretation" the conformance test mechanisms will differ.

ISO/IEC certifications (management systems and lab)

The whole industry of management system standards, ISO 9001, 14001, 27001, 50001, etc only provide requirements. They are all published by ISO and IEC, but they do not dictate any tests.

The same goes for lab certifications, only requirements, no tests.

There are guidelines for conformity testing, such as ISO 19011, ISO 170xx, but they are not prescriptive tests.

The "conformance tests" in that case is managed by many different entities:

• Mutual recognition of compliance/conformance tests rests entirely separate organization called the IAF (International Accreditation Forum)
• Accreditation Bodies (ABs) (e.g. ANSI, SCC, UKAS) "certify/accredit" the Certification Bodies that issue certificates.
• Certification Bodies (CBs) (e.g. BSI, SGS, TUV) "certify" organizations or labs.
• The implementation is performed by the target entity.

The conformance test against for example, ISO 9001:2015, 5.2.1 as shown above, is performed by the CB:

• The ABs/IAF comes up with an interpretation for ISO 9001:2015, and provides guidance on how to audit ("test") 5.2.1.
• The CB comes up with an Audit Manual for ISO 9001 that includes an audit procedure for 5.2.1 implementing the ABs/IAF view.
• The CB's Audit Manual is checked and accredited by an AB annually.
• The CB checks the target entity on whether their implementation of 5.2.1 meets their audit criteria.

Different ABs interpret 5.2.1 differently, and their audit guidelines for CBs for 5.2.1 are slightly different. Every CB's interpretation on how to best check 5.2.1 will also be different, some are more lax, some are more rigid, some allow compensatory controls...

Conclusion / proposal

At this point we can see there is no "single, unambiguous, unquestionable interpretation" on what 5.2.1 means (there is general agreement, but not specifically into details), and of course, there cannot be any 100% agreement on what the conformance test for 5.2.1 is.

Yes, for technical standards, it is easier to come up with an unambiguous test for a requirement (even still, not all the time!). Yet there are standards that fall outside of this frame.

In OGC and ISO/TC 211 standards, I have seen too many "abstract to a point it is worthless" kind of ATS. They basically just regurgitate the contents of the requirements, and serve no purpose except sow confusion to the reader and place an unnecessary burden to the standard authors. I imagine that I'm not the only one experiencing this?

If ModSpec is meant to be a generic requirements model, we need to relax the "requirement" that "a requirement must have a conformance test". This requirement can be made an OGC policy so OGC always have good and testable standards? But we can let other organizations who adopt ModSpec decide for themselves?

[cnreediii]

@ronaldtse There may be a simple solution - I like simple solutions. On the call yesterday there was a suggestion - with agreement - that there will be a very short OGC centric companion document that 1.) states that the ModSpec is OGC Policy and 2.) could contain OGC specific guidance. Specific guidance could be an OGC profile of the ModSpec that contains the requirement that every requirement shall have a conformance test. We would then also need to redo some wording in the core ModSpec document to that effect.

As to ambiguous or not well done ATSs - that is a different issue and not a ModSpec problem.