Validation Suite Design

Components

Model

A model represents, abstractly, a particular physical system that is going to be the basis of a simulation run. It may define a set of global parameters that can be customized.

An example would be a model representing a single compartment RC-circuit for stability testing, with parameters describing the minimum and maximum δt to test.

Simulator model implementation

A script that implements a given model for a specific simulator, takes model parameters as arguments, and emits numerical simulation results in an accepted data representation format.

Per-model reference data generation

A script that creates the reference test data for analysis, if any is required. It takes model parameters as arguments. It might possibly also examine the simulator output in order to determine sample points or times or similar. Some reference data may be pre-generated, making the script implementation trivial.

Analysis

A model-specific or generic analysis script that compares simulator output with reference data and produces summary data.

Validation

A pass/fail criterion applied to the generated summary data.

Procedure

• For each simulator do:
  • For each model configuration (model + pre-defined model parameter set) with an implementation for this simulator do:
    1. Run implementation with model parameters.
    2. Generate analysis data for model and model parameters (this can be cached).
    3. Run analysis script.
    4. Run validation test on analysis summary data.

Possible layout

We could group configurations in a directory per model, with one script per model+parameter set. The script would take the name of the simulator to test as a parameter, and proceed as follows:

1. Does the corresponding simulator model implementation exist? (Should be an appropriately named script within the same directory, by some naming convention.) If not, exit.
2. Make an output directory for simulator and analysis results, following a naming convention that is a function of model name, model parameter set name, simulator name, and timestamp.
3. Run simulator script with parameters, passing it a standard filename in the output directory to use for output.
4. Run reference data generation script with model parameters [and possibly path to simulator output if required], again with a standard filename in the output directory.
5. Run analysis script with corresponding paths.
6. Run validation on analysis summary data to produce a pass/fail result, and a pass/fail record in the output directory.

With a standardized data representation format, the validation test should be able to be performed with a generic script, taking a path to the summary data and a simple boolean pass/fail condition.

Some analysis scripts may be generic (e.g. compute L∞ distance over named time series) and be held in a common script directory, while some others may be model specific.

Having one script (say in bash or another scripting language) per model/parameter-set looks like it will strike a balance between flexibility in the tests and limiting per-script boilerplate code.