baseline.md

Baseline:

Baseline requirements are the minimum set of requirements that all AI/ML systems should meet.

There are 28 questions that should be filled out by the development team/ML engineer for the application. The last question is specific to Generative AI applications only.

Questions highlighted in bold indicate those which require minor testing before the question can be answered.

CATEGORY	DESCRIPTION
Data	Have all features been tested for data leakage before being used as training data? The set of features developed for a model should not contain data unnecessary for predicting an outcome.
	Is the training data to the model verified for completeness? If there are variables we cannot collect due to dataset restrictions, are they documented? Ensure that there are no missing variables required for prediction or incomplete data with missing values. Missing values should be handled using proper statistical approaches. Furthermore, changes to the data must be reviewed before direct addition. This is important so that malicious values cannot be inserted into the training data.
	Have required permissions been obtained to use the training dataset? Does the training data have any copyright or legal restrictions?
	Is the entire dataset developed within the organization? If no, can you document the source of the third party dataset? If no, is the dataset open source? Are there restrictions to using the open source dataset (e.g. requiring owner approval)?
	Can the training dataset be recovered from a secondary location within the organization that is not your application? This secondary location can be another application or a third-party customer/vendor. If external to the company, the location must be verified to be secure.
	Is the training data maintained? (to correct labeling errors and attributes)
	Is the training dataset well-curated during pre-processing to not introduce any intentional or unintentional errors? For example, if pre-processing is done using code and not manually, the chances of errors being introduced are less.
	Is there a recovery plan in case training data is contaminated? This would ideally mean having a process to recover the training data and retrain a model in case it does not behave as intended.
	Has the model been tested with realistic data (similar to what is being predicted) and conditions before deployment? Are anomalies recorded?
Model	Is the model explainable? Though explainability is broad, for now, we mean documenting the (i) the purpose, (ii) model functionality (how the model works), including type of algorithm used, and, (iii) accuracy with test cases.
	In case the model is using confidential information (personal information or proprietary information), is there a warning/notification in place to limit unchecked predictions? For instance, if the AI/ML model is recommending a higher broadband plan, it should come with a warning that recommends consulting with a representative before making a change based on AI/ML output alone. If not, it can potentially have a wrong prediction in case the user received a manipulated output.
	Has the model been tested with different test datasets to emulate actual prediction conditions before production?
	Can your model code be deployed for secondary use? For example, can it be applied to a different use case/context or wrapped as a single package that can be hidden under another frontend.
	Are all third-party supply chain dependencies in code documented for security? This includes external packages used to import algorithms, online data repositories, or external code snippets.
	Did you test if someone without access to the source code, can learn information about the training data or algorithm?
	Did you test if the training data can be mislabeled without notifying the team?
	Do you document if the model behavior (including accuracy) change during migration from one environment to another? Do you have version control in place when the model is updated? For example, when the AI/ML application is moved from development to production, could there potentially be significant changes in the number of false positives or false negatives? Environment in this case can also mean changing platforms or the programming language used in the source code.
	Do you keep track of how the model predictions have changed over time? Do you document the cause if there is significant model deviation? This should also include documentation if the underlying model being used has also changed.
	Has the model been tested to check if it can be modified to prevent legitimate users from accessing? This is because attackers can modify a query to get a desired response from an AI/ML application. If this compromises the model's ability to classify accurately, it might classify legimitate user systems incorrectly and prevent them from access.
	Are changes to the model tracked and logged? Are access controls in place regulating who has access to change the model? Are these used to audit correctness? Is the audit performed either annually or on a risk-informed basis?
Artefact	Do you have a process in place for a human to correct any incorrect output from the application?
	Can the model output carry any sensitive information about a person? Can this output be saved outside the application? Are there access controls in place for who can access this sensitive data? Sensitive information can be either direct identifiers or inferences about a person, like their shopping preferences.
	Have you tested if the model output shows information beyond its intended goal/scope?
	Are consequences (unexpected user output, degradation of service, DoS, etc.) of failure modes of the output documented? A 'failure mode' happens when a model does not behave as expected. This can either be no output, wrong output, or problematic output (like usage of derogatory terms for instance).
System/Infrastructure	Does your application use separate hardware to support the AI/ML system? If yes, have you tested if an attacker can observe any information about the model? Has a penetration testing assessment been conducted on the hardware components of the model?
	Are there resources to maintain the model source code?
	Was any part of the AI/ML application created or modified by a team within the organization? If yes, do you have correct license and documentation in place for attribution? This can be important if the application can potentially be used to create a proxy model.
	Are there geolocation conditions (local laws, regulations, restrictions, etc.) that need to be met for multinational deployment, if applicable?
Generative AI
Model	Did you test if the model can make automated incorrect decisions based if incorrect input (prompt) is provided when it self learns? To prevent this, applications typically limit the scope of the data provided to the model as input to a specific purpose. There are guardrails built to reject unexpected data.