feat: add python ci

[JaeAeich]

Add python ci

Summary by Sourcery

Introduce multiple CI workflows for Python projects, including setup, testing, code quality checks, vulnerability assessments, and package publishing to PyPI. These workflows utilize Poetry for dependency management and caching, and integrate tools like Bandit, mypy, ruff, and typos for various checks.

CI:

• Add a CI workflow for setting up Python and Poetry, including dependency management and caching.
• Introduce a documentation check workflow to ensure API documentation is up to date.
• Implement an integration test workflow that sets up the environment, runs tests, and uploads coverage reports.
• Add a unit test workflow to run unit tests and upload coverage reports.
• Create a code vulnerability test workflow using Bandit to check for code vulnerabilities.
• Introduce a type check workflow to ensure type safety and correctness using mypy.
• Add a workflow to publish Python packages to PyPI upon release.
• Implement a dependency vulnerability test workflow using Safety to check for vulnerabilities in dependencies.
• Add a code formatting check workflow to ensure code adheres to style guidelines using ruff.
• Introduce a linting workflow to ensure code quality and adherence to standards using ruff.
• Add a spell check workflow to identify spelling errors in the codebase using typos.

[sourcery-ai]

Reviewer's Guide by Sourcery

This pull request introduces a comprehensive set of Python CI workflows and actions, focusing on setting up the environment, running various code quality checks, tests, and publishing to PyPI. The changes are implemented by adding multiple YAML configuration files for different GitHub Actions, utilizing the Poetry package manager for Python dependency management.

File-Level Changes

Change	Details	Files
Implement Poetry setup action for consistent Python environment configuration	Configure system, Python, and Poetry versions Set up caching for pipx and Poetry dependencies Install Poetry and project dependencies Generate hash for required dependencies to optimize caching	actions/python/setup/poetry/action.yaml
Add documentation check action to verify API documentation is up to date	Generate API docs using sphinx-apidoc Compare current docs with latest generated docs Fail CI if documentation is out of date	actions/python/docs/autogenerated-docs-check/action.yaml
Implement integration and unit test actions with coverage reporting	Set up test environment using Poetry Run tests using pytest with coverage Upload coverage reports to Codecov	actions/python/code-test/integration-test/action.yaml actions/python/code-test/unit-test/action.yaml
Add code and dependency vulnerability check actions	Implement Bandit for code vulnerability scanning Use Safety for checking dependency vulnerabilities	actions/python/vulnerability/code-vulnerability/action.yaml actions/python/vulnerability/dependency-vulnerability/action.yaml
Implement code quality check actions	Add type checking using mypy Implement code formatting check using Ruff Add linting action using Ruff Implement spell checking using Typos	actions/python/code-quality/type-check/action.yaml actions/python/code-quality/format/action.yaml actions/python/code-quality/lint/action.yaml actions/python/code-quality/spell-check/action.yaml
Add PyPI publishing action for releasing Python packages	Set up environment for package publishing Implement Poetry-based package building and publishing to PyPI	actions/python/release/pypi/action.yaml

---

Tips

• Trigger a new Sourcery review by commenting @sourcery-ai review on the pull request.
• Continue your discussion with Sourcery by replying directly to review comments.
• You can change your review settings at any time by accessing your dashboard:
  • Enable or disable the Sourcery-generated pull request summary or reviewer's guide;
  • Change the review language;
• You can always contact us if you have any questions or feedback.

[sourcery-ai]

Hey @JaeAeich - I've reviewed your changes - here's some feedback:

Overall Comments:

• Consider pinning the external action to a specific version or commit hash instead of using @main to ensure stability and avoid unexpected changes.
• Enhance the pull request description to provide more context and details about the changes and their purpose.

Here's what I looked at during the review

• 🟢 General issues: all looks good
• 🟢 Security: all looks good
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.}

[JaeAeich]

TODO: remove before merge and add this as another PR.

[JaeAeich]

@sourcery-ai review

[sourcery-ai]

Hey @JaeAeich - I've reviewed your changes and they look great!

Here's what I looked at during the review

• 🟡 General issues: 2 issues found
• 🟡 Security: 1 issue found
• 🟢 Testing: all looks good
• 🟢 Complexity: all looks good
• 🟢 Documentation: all looks good

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

_{Help me be more useful! Please click 👍 or 👎 on each comment to tell me if it was helpful.}

[sourcery-ai]

suggestion: Using shasum for doc comparison may be overly sensitive

Consider using a more robust method for comparing documentation that isn't as sensitive to minor formatting changes.

        diff -rq /tmp/docs/source/pages/ ${{ inputs.auto_doc_dir }}/ | grep -v "Only in" > /tmp/docs_diff.txt
        if [ -s /tmp/docs_diff.txt ]; then
          echo "Differences found in documentation:"
          cat /tmp/docs_diff.txt
        else
          echo "No significant differences found in documentation."
        fi

[JaeAeich]

I want to remove os as a parameter here because this gets confusing in our case, note we are using custom poetry setup that I have written with ubuntu as base image. Is hasn't been tested on other OS liek OSX or windows. This input might cause errors due to incompatibility. Also I think in pythons case this just adds a level of complexity that should be abstracted from user.

[uniqueg]

Agreed - we only support Linux, at least for the foreseeable future.

I think this applies to other actions as well.

[JaeAeich]

TODO: Make this an optional fields too, grep and awk name of the project from pyproject and search if that dir exists, if so take that as file_path otherwise check if user has provided a file_path, else err.

We should aim to make all the fields non required, or atleast the input should not depend on project, ie codecov_token will be secret.CODECOV_TOKEN regardless of the project hence this will be easier to template. Essentially if we can ideally make all the deafults such that their is no values passed that are related to project, we can template all the CI work in one single repo.

[JaeAeich]

TODO: Remember to add below code in example ci, below code stops previously running ci if a new commit is pushed, saving ci time.

concurrency:
  group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
  cancel-in-progress: true

[JaeAeich]

Move this to common, this way we will have only one way to spell check the repo.

Currently spell checking is linked as dep from pyproj but I think it would make more sense to add spellchecking as pre-commit check (pre commit are being used for non code checks) which will make alot of sense.

If we do plan on going this route, then maybe it would be a good ide to remove typos from cookicutter as we should not have multiple spellcheckers.

[uniqueg]

All makes sense to me, including removing the spell checking from CC

[JaeAeich]

maybe this should just be named integration as parent dir is already called code-test.

[uniqueg]

Agreed

[uniqueg]

All makes sense to me, including removing the spell checking from CC

[uniqueg]

Agreed

[uniqueg]

Agreed - we only support Linux, at least for the foreseeable future.

I think this applies to other actions as well.

[uniqueg]

Is there a way that we could get around having to define this in all actions?

[JaeAeich]

if you mean poetry options, then no, but if you see the defaults are already set to what we use in cookiecutter, so if some one would want to use this action with cookicutter, they wont have to pass them either way.

[JaeAeich]

Well.... technically yes, but I am assuming people from outside the org might also use this action with CC and their own separate groups of deps, if we don't take that into consideration, yes we can definitely remove it.

[JaeAeich]

@uniqueg Im testing this CI on my private repo, and another good feature that I think would be nice to implement would be to add debug steps for CI. So basically for each CI that fails, comment on the PR, what went wrong or, how to improve it, or who to tag and ask for help. Check out this example.

I propose, for each CI we document the most common way to fix it. Lets say the format CI fails, the bot will comment, please run make fl, if you are using windows install poetry, and run this command etc etc, the debug steps. This will be on Doccumantion hub obv, and the PR link it.

Then we also let the PR creator know who to tag, this we fetch from a list of mainteiners from somewhere (a gitst perhaphs), this way we can have updated list of maintainer for a particular language or porject, maybe, lets say I have a gist

uniqueg

I provide gist link to the actions, and it fetches, does data sanitisation, and suggest the PR creator to tag the mainter if in need.

Thoughts??

PS: Yess Im testing on windows as well, imma make it compatible even if we don;t use it.