Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate pyproject.toml to poetry for inspection #135

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

glatterf42
Copy link
Member

@glatterf42 glatterf42 commented Oct 20, 2023

Closes #129.

As pointed out by @macflo8, the sdmx dependency version needs to be bumped. And all of the below can be discarded by dropping the second and third commit, if we don't like it. However:
While working on this, I found myself thinking that I'd like to manage dependencies with poetry as we already do in ixmp4. With poetry, we could quite easily update more dependencies as well:

$ poetry show -o

shows all packages in the venv that can be updated and whether this update is considered safe;

$ poetry show -o -T

shows all of these packages that we explicitly specify in our pyproject.toml. In this case, the output of the latter is

pyam-iamc                     1.9.0    2.0.0     Analysis & visualization of integrated-assessment sc...
sphinx                        6.2.1    7.2.6     Python documentation generator
sphinxcontrib-bibtex          2.5.0    2.6.1     Sphinx extension for BibTeX style citations.

With pip, I would now specify pyam-iamc >= 2.0.0 in the dependencies to force the new version. In contrast, poetry can manage pyproject.toml for us and keeps us from making that mistake:

$ poetry add pyam-iamc@latest
Using version ^2.0.0 for pyam-iamc

Updating dependencies
Resolving dependencies... (0.0s)

The current project's Python requirement (>=3.8,<3.12) is not compatible with some of the required packages Python requirement:
  - pyam-iamc requires Python <3.12,>=3.10, so it will not be satisfied for Python >=3.8,<3.10

Because no versions of pyam-iamc match >2.0.0,<3.0.0
 and pyam-iamc (2.0.0) requires Python <3.12,>=3.10, pyam-iamc is forbidden.
So, because message-ix-models depends on pyam-iamc (^2.0.0), version solving failed.

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
    
    For pyam-iamc, a possible solution would be to set the `python` property to ">=3.10,<3.12"

So we immediately know that we will only be able to use pyam-iamc 2.0.0 once we move our python requirement to be at least 3.10.

Poetry has some other nice features such as a focus on environment isolation. If poetry detects it is running in a venv, it will use that one, but if it isn't running in one, it will create and manage one for you. For ixmp4, we use the config option poetry config virtualenvs.in-project true to have these venvs installed in the project rather than globally.

On the user side, nothing should change by including this version of pyproject.toml and poetry.lock. We can still run

$ pip install message-ix-models[docs,report,tests]

or equivalently, when using poetry,

# To not remember all extra-names
$ poetry install --all-extras

# To specify only some extras
$ poetry install --extras "docs tests"

However, it might be possible to improve other things as well: maybe we can use the automatically created poetry.lock as an input file for pinned dependencies for the docs builds that is automatically kept up-to-date when we update dependencies via poetry.

To install message-ix-models using poetry, you can follow these steps from within your local clone of this branch:

# Install Poetry, minimum version >=1.2 required
$ curl -sSL https://install.python-poetry.org | python -

# You may have to reinitialize your shell at this point.
$ source ~/.bashrc

# Activate in-project virtualenvs
$ poetry config virtualenvs.in-project true

# Add dynamic versioning plugin 
$ poetry self add "poetry-dynamic-versioning[plugin]"

# Install dependencies
$ poetry install --all-extras

Once this is done, you can access the venv from the directory by running poetry shell, which will spawn a new shell where you can run mix-models and everything else as usual. However, you can also use poetry run <command> to run cli commands without creating a new shell.

Please note that I dropped some classifiers since poetry is adding python and license classifiers automatically.

How to review

  • Read the diff and note that the CI checks all pass.
  • Try installing the project via poetry and see how you like it :)
  • Ensure that changes/additions are self-documenting, i.e. that another
    developer (someone like the reviewer) will be able to understand what the code
    does in the future. (only if we decide to keep poetry)

PR checklist

  • Continuous integration checks all ✅
  • [ ] Add or expand tests; coverage checks both ✅ Not adding functionality.
  • [ ] Add, expand, or update documentation. Only if we keep poetry.
  • [ ] Update doc/whatsnew. Only if we keep poetry.

@glatterf42 glatterf42 added bug Something isn't working enh New features or functionality labels Oct 20, 2023
@glatterf42 glatterf42 self-assigned this Oct 20, 2023
@glatterf42
Copy link
Member Author

We could definitely use poetry to install our projects for readthedocs.
Looking also at poetry's docs for the install command (that explain the consequences of committing poetry.lock), it seems to me that this would indeed allow us to remove doc/requirements.in and doc/requirements.txt.
With poetry.lock present, poetry install will install the versions pinned in the file, so there is no need to keep additional pins around that we have to update manually every once in a while. Of course, we still have to keep the dependencies in pyproject.toml up-to-date, but poetry can make this easier and at least we wouldn't have to update dependencies in another place.

@glatterf42 glatterf42 changed the title Update sdmx dependency & migrate pyproject.toml to poetry for inspection Migrate pyproject.toml to poetry for inspection Feb 1, 2024
@glatterf42
Copy link
Member Author

Dropped bumping sdmx since this has been done on main already.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enh New features or functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Update sdmx version dependency
1 participant