Merge branch 'devel' into switchport_tap_tool

.flake8

@@ -1,4 +1,6 @@
 [flake8]
+# Flake8 has been replaced with Ruff.
+# This file should be removed later, but kept here to avoid IDE extensions from messing up the code.
 ignore = E402,W503,W504,E741
 
 # Required for compatibility with Black

.github/workflows/offline-links-check.yml

@@ -20,10 +20,10 @@ jobs:
     timeout-minutes: 15
     steps:
       - uses: actions/checkout@v4
-      - name: 'Start docker-compose stack'
+      - name: 'Start docker compose stack'
         run: |
-          docker-compose -f development/docker-compose.yml up -d webdoc_avd
-          docker-compose -f development/docker-compose.yml ps
+          docker compose -f development/docker-compose.yml up -d webdoc_avd
+          docker compose -f development/docker-compose.yml ps
       - name: 'Test connectivity to mkdoc server'
         run: |
           bash -c 'while [[ "$(curl -s -o /dev/null -w ''%{http_code}'' 127.0.0.1:8000)" != "200" ]]; do sleep 5; done'
@@ -46,4 +46,4 @@ jobs:
             --timeout 30
       - name: 'Stop docker-compose stack'
         run: |
-          docker-compose -f development/docker-compose.yml down
+          docker compose -f development/docker-compose.yml down

.pre-commit-config.yaml

@@ -73,53 +73,31 @@ repos:
           - --comment-style
           - '<!--|  ~|  -->'
 
-  # - repo: https://github.com/pre-commit/mirrors-autopep8
-  #   rev: 'v1.5.6'  # Use the sha / tag you want to point at
-  #   hooks:
-  #   -   id: autopep8
-
-  - repo: https://github.com/pycqa/isort
-    rev: 5.13.2
-    hooks:
-      - id: isort
-        name: Check for changes when running isort on all python files.
-
-  - repo: https://github.com/psf/black
-    rev: 24.4.2
-    hooks:
-      - id: black
-        name: Check for changes when running Black on all python files.
-
-  - repo: https://github.com/pycqa/flake8
-    rev: 7.0.0
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    # Ruff version.
+    rev: v0.5.7
     hooks:
-      - id: flake8
-        name: Check for Flake8 errors on Python files.
-        # ignoring errors and selecting line length as per
-        # https://github.com/ansible/ansible/blob/devel/test/lib/ansible_test/_util/controller/sanity/pep8/current-ignore.txt
-        types: [python]
-        # The cv client api is generated from proto files, so it should not be linted.
-        exclude: ^python-avd/pyavd/_cv/api
+      # Run the linter.
+      - id: ruff
+        args:
+          - --fix
+      # Run the formatter.
+      - id: ruff-format
 
   - repo: https://github.com/pycqa/pylint
-    rev: "v3.1.1"
+    rev: "v3.2.6"
     hooks:
       - id: pylint # Use pylintrc file in repository
         name: Check for Linting errors on Python files outside the Ansible collection.
         description: This hook runs pylint.
         types: [python]
-        args:
-          # Suppress duplicate code for modules header
-          - -d duplicate-code
       - id: pylint # Use pylintrc file in Ansible collection
         name: Check for Linting errors on Python files in the Ansible collection.
         description: This hook runs pylint.
         types: [python]
         files: ansible_collections/arista/avd/
         args:
           - --rcfile=ansible_collections/arista/avd/pylintrc
-          # Suppress duplicate code for modules header
-          - -d duplicate-code
 
   - repo: https://github.com/adrienverge/yamllint.git
     rev: "v1.35.1"

ansible_collections/arista/avd/docs/contribution/eos_designs_internal_notes.md

@@ -43,8 +43,7 @@ The module is used in `arista.avd.eos_designs` to set facts for devices, which a
 
 ```yaml
   template_output: <true | false | default -> false>
-  conversion_mode: <"warning" | "info" | "debug" | "quiet" | "disabled" | default -> "debug">
-  validation_mode: <"error" | "warning" | "info" | "debug" | "disabled" | default -> "warning">
+  validation_mode: <"error" | "warning" | default -> "error">
   cprofile_file: <Filename for storing cprofile data used to debug performance issues>
 ```
 

ansible_collections/arista/avd/docs/contribution/input-variable-validation.md

@@ -28,75 +28,35 @@ The current implementation supports the automatic conversions listed below.
 | `bool`, `str` | `int` |
 | `int`, `str` | `bool` |
 | `bool`, `int` | `str` |
-| `dict`*, `list`** | `list` |
-
-\* If `primary_key` is set on the `list` schema, conversion from `dict`-of-`dicts` to `list`-of-`dicts` will insert the `primary_key`
-with the value of the outer dictionary key in each `dict`. If `primary_key` is *not* set on the `list` schema, only the input `dict`
-keys are returned as `list` items (any input `dict` values are lost).
-\*\* If `primary_key` is set on the `list` schema, conversion from `list` to `list`-of-`dicts` will insert the `primary_key` with the
-value of the input `list` items in each `dict`.
 
 An example of input variable conversion is `bgp_as`. `bgp_as` is expected as a string (`str`) since 32-bit AS numbers can be
-written with dot-notation such as `"65001.10000"`. Most deployments use 16-bit AS numbers, where the setting `bgp_as: 65001` will be interpreted as an integer in YAML, unless it is enclosed in quotes `bgp_as: "65001"`. In the schema for `bgp_as` the `convert_types` option is configured to `['int']` which means AVD Action Plugins will automatically convert
+written with dot-notation such as `"65001.10000"`. Most deployments use 16-bit AS numbers, where the setting `bgp_as: 65001`
+will be interpreted as an integer in YAML, unless it is enclosed in quotes `bgp_as: "65001"`. In the schema for `bgp_as` the
+`convert_types` option is configured to `['int']` which means AVD Action Plugins will automatically convert
 `bgp_as: 65001` to `bgp_as: "65001"`.
 
-Type conversion is also used for introducing changes to the data models without affecting existing deployments. For example,
-in AVD 4.0, the data models using "Dictionaries with wildcard keys" has been be changed to lists.
-Ex. the old data model:
-
-```yaml
-tenants:
-  mytenant:
-    vrfs:
-      myvrf:
-        ...
-```
-
-has been be changed to:
-
-```yaml
-tenants:
-  - name: mytenant
-    vrfs:
-      - name: myvrf
-        ...
-```
-
-The input data will be converted automatically and validated according to the new model to allow for a gracious introduction of this data-model changes in late 3.x versions.
-
-Type conversion is turned on by default.
-
-!!! CAUTION
-    Although conversion can be disabled, this is **not** recommended, and will most likely lead to issues with AVD or the generated
-    configuration. Data conversion also handles data deprecation and upgrades.
+Data conversion processing also handles deprecation warnings.
 
 ## Validation Options
 
 Schema validation is built in to the central Action plugins used in AVD. Each plugin runs variable type conversion first and then
 performs validation of the converted data.
 
-By default the data conversions are logged with `-v` and data validation issues will trigger warnings - not blocking further processing.
+By default any data validation issues will trigger errors - blocking further processing.
 This behavior can be adjusted by setting the variables described below.
 
-```yaml
-# Conversion Mode for AVD input data conversion | Optional
-# During conversion, messages will generated with information about the host(s) and key(s) which required conversion.
-# "disabled" means that conversion will not run - avoid this since conversion is also handling data deprecation and upgrade.
-# "error" will produce error messages and fail the task.
-# "warning" will produce warning messages.
-# "info" will produce regular log messages.
-# "debug" will produce hidden debug messages viewable with -v.
-# "quiet" will not produce any messages
-avd_data_conversion_mode: < "disabled" | "error" | "warning" | "info" | "debug" | "quiet" | default -> "debug" >
+!!! danger
+    The input variable `avd_data_validation_mode` now has a default value of `error`, and while it can be set to `warning`, this is highly discouraged.
+    All AVD code relies on the validation to ensure valid data, so the code assumes that the data is valid.
 
+    If the validation mode is set to `warning`, execution will continue with invalid data, which can lead to hard-to-read errors or incorrect behavior.
+
+```yaml
 # Validation Mode for AVD input data validation | Optional
 # During validation, messages will generated with information about the host(s) and key(s) which failed validation.
-# "disabled" means that validation will not run.
 # "error" will produce error messages and fail the task.
 # "warning" will produce warning messages.
-# "info" will produce regular log messages.
-# "debug" will produce hidden debug messages viewable with -v.
-avd_data_validation_mode: < "disabled" | "error" | "warning" | "info" | "debug" | default -> "warning" >
+avd_data_validation_mode: < "error" | "warning" | default -> "error" >
 ```
 
 ## Schema Details
@@ -125,7 +85,7 @@ This fragment will be merged with other fragments during development, to form th
 
 For reference, the full Role Schemas can be found here:
 
-- [`eos_designs` AVD Schema](https://github.com/aristanetworks/avd/tree/devel/python-avd/pyavd/eos_designs/schema/eos_designs.schema.yml)
+- [`eos_designs` AVD Schema](https://github.com/aristanetworks/avd/tree/devel/python-avd/pyavd/_eos_designs/schema/eos_designs.schema.yml)
 - [`eos_cli_config_gen` AVD Schema](https://github.com/aristanetworks/avd/tree/devel/python-avd/pyavd/_eos_cli_config_gen/schema/eos_cli_config_gen.schema.yml)
 
 The supported schema options depend on the type of variable that is described. The supported types are `int`, `bool`, `str`,
@@ -228,13 +188,11 @@ The meta-schema does not allow for other keys to be set in the schema.
 | Key | Type | Required | Default | Value Restrictions | Description |
 | ----| ---- | -------- | ------- | ------------------ | ----------- |
 | <samp>type</samp> | String | True | | Valid Values:<br>- `"list"` | Type of variable using the Python short names for each type.<br>`list` for List |
-| <samp>convert_types</samp> | List, items: String | | | | List of types to auto-convert from.<br>For `list` auto-conversion is supported from `dict` if `primary_key` is set on the list schema |
 | <samp>default</samp> | List | | | | Default value |
 | <samp>items</samp> | Dict | | | | Dictionary describing the schema of each list item. This is a recursive schema, so the value must conform to AVD Schema |
 | <samp>max_length</samp> | Integer | | | | Maximum length |
 | <samp>min_length</samp> | Integer | | | | Minimum length |
 | <samp>primary_key</samp> | String | | | Pattern: `^[a-z][a-z0-9_]*$` | Name of a primary key in a list of dictionaries.<br>The configured key is implicitly required and must have unique values between the list elements |
-| <samp>secondary_key</samp> | String | | | Pattern: `^[a-z][a-z0-9_]*$` | Name of a secondary key, which is used with `convert_types:['dict']` in case of values not being dictionaries |
 | <samp>unique_keys</samp> | List, items: String | | | Item Pattern: `^[a-z][a-z0-9_]*$` | Name of a key in a list of dictionaries.<br>The configured key must have unique values between the list elements.<br>This can also be a variable path using dot-notation like `parent_key.child_key` in case of nested lists of dictionaries. |
 | <samp>display_name</samp> | String | | | Regex Pattern: `"^[^\n]+$"` | Free text display name for forms and documentation (single line) |
 | <samp>description</samp> | String | | | Minimum Length: 1 | Free text description for forms and documentation (multi line) |

ansible_collections/arista/avd/docs/contribution/style-guide.md

@@ -10,22 +10,22 @@ This page lists guidelines for developing Python code or Jinja2 templates in the
 
 ## Python code style
 
-As AVD is an Ansible collection, we're required to follow guidelines from the official [Ansible documentation](https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_best_practices.html) for all Python code.
+For the Ansible collection `arista.avd.`, we are required to follow guidelines from the official [Ansible documentation](https://docs.ansible.com/ansible/latest/dev_guide/developing_modules_best_practices.html) for all Python code.
 
-Furthermore, the CI Pipeline (& pre-commit) for AVD enforces the following:
+For PyAVD and other areas we follow common good practices as enforced by Ruff and Pylint.
+
+The CI Pipeline (& pre-commit) for AVD enforces the following:
 
 - Maximum line length of 160
-- [Black](https://black.readthedocs.io/en/stable/index.html) version 24.3.0
-- [isort](https://pycqa.github.io/isort/) version 5.13.2
-- [Flake-8](https://flake8.pycqa.org/en/4.0.1/index.html) version 4.0.1
-- [pylint](https://pylint.pycqa.org/en/2.6/) version 2.6.0
+- [Ruff](https://docs.astral.sh/ruff/) version >=0.5.6
+- [pylint](https://pylint.readthedocs.io/en/stable/) version >=3.2.6
 
 Configurations for the above tools can be found in:
 
 - [pyproject.toml](https://github.com/aristanetworks/avd/blob/devel/pyproject.toml)
 - [.pre-commit-config.yaml](https://github.com/aristanetworks/avd/blob/devel/.pre-commit-config.yaml)
-- [.flake8](https://github.com/aristanetworks/avd/blob/devel/.flake8)
-- [pylintrc](https://github.com/aristanetworks/avd/blob/devel/pylintrc)
+- [pylintrc](https://github.com/aristanetworks/avd/blob/devel/ansible_collections/arista/avd/pylintrc) for the Ansible Collection
+- [pylintrc](https://github.com/aristanetworks/avd/blob/devel/pylintrc) for everything else
 
 ## Jinja2 Syntax and guidelines
 

ansible_collections/arista/avd/docs/plugins/Modules_and_action_plugins/eos_cli_config_gen.md

@@ -34,8 +34,7 @@ The `arista.avd.eos_cli_config_gen` module is an Ansible Action Plugin providing
 | <samp>generate_device_config</samp> | bool | optional | True |  | Flag to generate the device configuration. |
 | <samp>generate_device_doc</samp> | bool | optional | True |  | Flag to generate the device documentation. |
 | <samp>device_doc_toc</samp> | bool | optional | True |  | Flag to generate the table of content for the device documentation. |
-| <samp>conversion_mode</samp> | str | False | debug | Valid values:<br>- <code>error</code><br>- <code>warning</code><br>- <code>info</code><br>- <code>debug</code><br>- <code>quiet</code><br>- <code>disabled</code> | Run data conversion in either &#34;error&#34;, &#34;warning&#34;, &#34;info&#34;, &#34;debug&#34;, &#34;quiet&#34; or &#34;disabled&#34; mode.<br>Conversion will perform type conversion of input variables as defined in the schema.<br>Conversion is intended to help the user to identify minor issues with the input data, while still allowing the data to be validated.<br>During conversion, messages will be generated with information about the host(s) and key(s) which required conversion.<br>conversion_mode:disabled means that conversion will not run.<br>conversion_mode:error will produce error messages and fail the task.<br>conversion_mode:warning will produce warning messages.<br>conversion_mode:info will produce regular log messages.<br>conversion_mode:debug will produce hidden messages viewable with -v.<br>conversion_mode:quiet will not produce any messages. |
-| <samp>validation_mode</samp> | str | False | warning | Valid values:<br>- <code>error</code><br>- <code>warning</code><br>- <code>info</code><br>- <code>debug</code><br>- <code>disabled</code> | Run validation in either &#34;error&#34;, &#34;warning&#34;, &#34;info&#34;, &#34;debug&#34; or &#34;disabled&#34; mode.<br>Validation will validate the input variables according to the schema.<br>During validation, messages will be generated with information about the host(s) and key(s) which failed validation.<br>validation_mode:disabled means that validation will not run.<br>validation_mode:error will produce error messages and fail the task.<br>validation_mode:warning will produce warning messages.<br>validation_mode:info will produce regular log messages.<br>validation_mode:debug will produce hidden messages viewable with -v. |
+| <samp>validation_mode</samp> | str | False | error | Valid values:<br>- <code>error</code><br>- <code>warning</code> | Run validation in either &#34;error&#34; or &#34;warning&#34; mode.<br>Validation will validate the input variables according to the schema.<br>During validation, messages will be generated with information about the host(s) and key(s) which failed validation.<br>validation_mode:error will produce error messages and fail the task.<br>validation_mode:warning will produce warning messages. |
 | <samp>cprofile_file</samp> | str | False | None |  | Filename for storing cprofile data used to debug performance issues.<br>Running cprofile will slow down performance in it self, so only set this while troubleshooting. |
 
 ## Examples