Skip to content

Commit

Permalink
Refactor to use ixmp4 (#766)
Browse files Browse the repository at this point in the history
Co-authored-by: Philip Hackstock <[email protected]>
  • Loading branch information
danielhuppmann and phackstock authored Aug 23, 2023
1 parent e70a0b7 commit 525ede9
Show file tree
Hide file tree
Showing 16 changed files with 137 additions and 112 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/nightly.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# This workflow installs the package on Python 3.8, runs the tests and builds the docs
# This workflow installs the package, runs the tests and builds the docs
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: nightly
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/pytest-legacy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: '3.8'
python-version: '3.10'

- name: Install specific out-dated version of dependencies
# Update the package requirements when changing minimum dependency versions
# Please also add a section "Dependency changes" to the release notes
run: pip install pandas==1.2.0 numpy==1.19.0 matplotlib==3.5.0 iam-units==2020.4.21 xlrd==2.0 pint==0.13
run: pip install pandas==2.0.0 numpy==1.23.0 matplotlib==3.6.0 iam-units==2020.4.21 xlrd==2.0.1 pint==0.13

- name: Install other dependencies and package
run: pip install .[tests,optional_plotting,optional_io_formats,tutorials]
Expand Down
3 changes: 0 additions & 3 deletions .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,10 +18,7 @@ jobs:
- ubuntu-latest
- windows-latest
python-version:
- '3.11'
- '3.10'
- '3.9'
- '3.8'

fail-fast: false

Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ pyam: analysis & visualization <br /> of integrated-assessment and macro-energy

<!-- replace python version by dynamic reference to pypi once Python versions are configured there -->
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![python](https://img.shields.io/badge/python-≥3.8,<3.12-blue?logo=python&logoColor=white)](https://github.com/IAMconsortium/pyam)
[![python](https://img.shields.io/badge/python-≥3.10,<3.12-blue?logo=python&logoColor=white)](https://github.com/IAMconsortium/pyam)
[![pytest](https://github.com/IAMconsortium/pyam/actions/workflows/pytest.yml/badge.svg)](https://github.com/IAMconsortium/pyam/actions/workflows/pytest.yml)
[![ReadTheDocs](https://readthedocs.org/projects/pyam-iamc/badge/?version=latest)](https://pyam-iamc.readthedocs.io/en/latest/?badge=latest)
[![codecov](https://codecov.io/gh/IAMconsortium/pyam/branch/main/graph/badge.svg)](https://codecov.io/gh/IAMconsortium/pyam)
Expand Down
14 changes: 14 additions & 0 deletions RELEASE_NOTES.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,21 @@
# Next Release

The next release must bump the major version number.
Reactivate tests for Python 3.11 once ixmp4 0.3 is released.

## Dependency changes

Support for Python 3.7-3.9 was removed due to an incompatible dependency.

PR [#766](https://github.com/IAMconsortium/pyam/pull/766) added the **ixmp4** package
for better integration with the IIASA scenario database infrastructure.

## API changes

Credentials to access the IIASA scenario database infrastructure should now be managed
using the **ixmp4** package
(see [here](https://pyam-iamc.readthedocs.io/en/stable/api/iiasa.html)).

The column *exclude* of the `meta` indicators was moved to a new attribute `exclude`.
All validation methods are refactored such that the argument `exclude_on_fail` changes
this new attribute (see PR [#759](https://github.com/IAMconsortium/pyam/pull/759)).
Expand All @@ -19,8 +31,10 @@ instead of `pyam.to_list()`.

## Individual updates


- [#772](https://github.com/IAMconsortium/pyam/pull/772) Show all missing rows for `require_data()`
- [#771](https://github.com/IAMconsortium/pyam/pull/771) Refactor to start a separate validation module
- [#766](https://github.com/IAMconsortium/pyam/pull/766) Use **ixmp4** for credentials to access a Scenario Explorer database
- [#764](https://github.com/IAMconsortium/pyam/pull/764) Clean-up exposing internal methods and attributes
- [#763](https://github.com/IAMconsortium/pyam/pull/763) Implement a fix against carrying over unused levels when initializing from an indexed pandas object
- [#759](https://github.com/IAMconsortium/pyam/pull/759) Excise "exclude" column from meta and add a own attribute
Expand Down
54 changes: 44 additions & 10 deletions docs/api/iiasa.rst
Original file line number Diff line number Diff line change
@@ -1,15 +1,51 @@
.. currentmodule:: pyam.iiasa

The **Connection** class
========================
Databases hosted by IIASA
=========================

IIASA's ixmp Scenario Explorer infrastructure implements a RestAPI
to directly query the database server connected to an explorer instance.
See https://software.ene.iiasa.ac.at/ixmp-server for more information.
The |pyam| package allows to directly query the scenario databases hosted by the
IIASA Energy, Climate and Environment program (ECE), commonly known as
the *Scenario Explorer* infrastructure. It is developed and maintained
by the ECE `Scenario Services and Scientific Software team`_.

.. _`Scenario Services and Scientific Software team` : https://software.ece.iiasa.ac.at
You do not have to provide username/password credentials to connect to any public
database instance using |pyam|. However, to connect to project-internal databases,
you have to create an account at the IIASA-ECE *Manager Service*
(https://manager.ece.iiasa.ac.at). Please contact the respective project coordinator
for permission to access a project-internal database.

To store the credentials on your machine so that |pyam| can use it to query a database,
we depend on the Python package |ixmp4|. You only have to do this once
(unless you change your password).

The credentials will be valid for connecting to *Scenario Apps* based on |ixmp4|
as well as for (legacy) *Scenario Explorer* database backends (see below).

In a console, run the following:

.. code-block:: console
ixmp4 login <username>
You will be prompted to enter your password.

.. warning::

Your username and password will be saved locally in plain-text for future use!

*Scenario Apps* instances
-------------------------

Coming soon...

*Scenario Explorer* instances
-----------------------------

The *Scenario Explorer* infrastructure developed by the Scenario Services and Scientific
Software team was developed and used for projects from 2018 until 2023.

The |pyam| package uses this interface to read timeseries data as well as
categorization and quantitative indicators.
The data is returned as an :class:`IamDataFrame`.
See `this tutorial <../tutorials/iiasa_dbs.html>`_ for more information.

.. autoclass:: Connection
Expand All @@ -20,5 +56,3 @@ See `this tutorial <../tutorials/iiasa_dbs.html>`_ for more information.

.. autofunction:: lazy_read_iiasa
:noindex:

.. autofunction:: set_config
11 changes: 8 additions & 3 deletions docs/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -357,13 +357,18 @@
.. |pyam| replace:: :class:`pyam`
.. |ixmp4| raw:: html
<a href="https://docs.ece.iiasa.ac.at/ixmp4/">
<code class="xref py py-class docutils literal notranslate"><span class="pre">
ixmp4</span></code></a>
.. |br| raw:: html
<br>
<br>
.. |datapackage.Package.docs| raw:: html
<a href="https://github.com/frictionlessdata/datapackage-py#package">read
the docs</a>
<a href="https://github.com/frictionlessdata/datapackage-py#package">read the docs</a>
"""
2 changes: 1 addition & 1 deletion docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ Release v\ |version|.
.. |black| image:: https://img.shields.io/badge/code%20style-black-000000.svg
:target: https://github.com/psf/black

.. |python| image:: https://img.shields.io/badge/python-≥3.8,<3.12-blue?logo=python&logoColor=white
.. |python| image:: https://img.shields.io/badge/python-≥3.10,<3.12-blue?logo=python&logoColor=white
:target: https://github.com/IAMconsortium/pyam

.. |pytest| image:: https://github.com/IAMconsortium/pyam/actions/workflows/pytest.yml/badge.svg
Expand Down
Binary file modified docs/logos/pyam-header.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/logos/pyam-social-media.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
17 changes: 14 additions & 3 deletions docs/tutorials/iiasa_dbs.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -47,12 +47,23 @@
"metadata": {},
"source": [
"If you have credentials to connect to a non-public or restricted Scenario Explorer instance,\n",
"you can store this information by running the following command in a separate Python console:\n",
"you can store this information by running the following command in a console:\n",
"\n",
"```\n",
"import pyam\n",
"pyam.iiasa.set_config(<username>, <password>)\n",
"ixmp4 login <username>\n",
"\n",
"```\n",
"\n",
"You will be prompted to enter your password.\n",
"\n",
"<div class=\"alert alert-warning\">\n",
"\n",
"Your username and password will be saved locally in plain-text for future use!\n",
"\n",
"</div>\n",
"\n",
"\n",
"\n",
"When initializing a new **Connection** instance, **pyam** will automatically search for the configuration in a known location."
]
},
Expand Down
96 changes: 39 additions & 57 deletions pyam/iiasa.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,10 @@
is_list_like,
)
from pyam.logging import deprecation_warning
import ixmp4
from ixmp4.conf import settings
from ixmp4.conf.auth import ManagerAuth


logger = logging.getLogger(__name__)
# set requests-logger to WARNING only
Expand All @@ -36,28 +40,22 @@
""".replace(
"\n", ""
)
IXMP4_LOGIN = "Please run `ixmp4 login <username>` in a console"

# path to local configuration settings
DEFAULT_IIASA_CREDS = Path("~").expanduser() / ".local" / "pyam" / "iiasa.yaml"

JWT_DECODE_ARGS = {"verify_signature": False, "verify_exp": True}


def set_config(user, password, file=None):
"""Save username and password for the IIASA API connection to a file"""
file = Path(file) if file is not None else DEFAULT_IIASA_CREDS
if not file.parent.exists():
file.parent.mkdir(parents=True)

with open(file, mode="w") as f:
logger.info(f"Setting IIASA-connection configuration file: {file}")
yaml.dump(dict(username=user, password=password), f, sort_keys=False)
raise DeprecationWarning(f"This method is deprecated. {IXMP4_LOGIN}.")


def _read_config(file):
"""Read username and password for IIASA API connection from file"""
with open(file, "r") as stream:
return yaml.safe_load(stream)
creds = yaml.safe_load(stream)

return ManagerAuth(**creds, url=settings.manager_url)


def _check_response(r, msg="Error connecting to IIASA database", error=RuntimeError):
Expand All @@ -73,74 +71,58 @@ def __init__(self, creds: str = None, auth_url: str = _AUTH_URL):
----------
creds : pathlib.Path or str, optional
Path to a file with authentication credentials
auth_url : str, optionl
auth_url : str, optional
Url of the authentication service
"""
self.client = httpx.Client(base_url=auth_url, timeout=10.0, http2=True)
self.access_token, self.refresh_token = None, None

if creds is None:
if DEFAULT_IIASA_CREDS.exists():
self.creds = _read_config(DEFAULT_IIASA_CREDS)
deprecation_warning(
f"{IXMP4_LOGIN} and manually delete the file '{DEFAULT_IIASA_CREDS}'.",
"Using a pyam-credentials file",
)
self.auth = _read_config(DEFAULT_IIASA_CREDS)
else:
self.creds = None
self.auth = ixmp4.conf.settings.default_auth
elif isinstance(creds, Path) or is_str(creds):
self.creds = _read_config(creds)
self.auth = _read_config(creds)
else:
raise DeprecationWarning(
"Passing credentials as clear-text is not allowed. "
"Please use `pyam.iiasa.set_config(<user>, <password>)` instead!"
f"{IXMP4_LOGIN} instead."
)

# if no creds, get anonymous token
# TODO: explicit token for anonymous login will not be necessary for ixmp-server
if self.creds is None:
r = self.client.get("/legacy/anonym/")
if r.status_code >= 400:
raise ValueError("Unknown API error: " + r.text)
self.user = None
self.access_token = r.json()
# explicit token for anonymous login is not necessary for ixmp4 platforms
# but is required for legacy Scenario Explorer databases
if self.auth.user.username == "@anonymous":
self._get_anonymous_token()

# else get user-token
else:
self.user = self.creds["username"]
self.obtain_jwt()

def __call__(self):
try:
# raises jwt.ExpiredSignatureError if token is expired
jwt.decode(self.access_token, options=JWT_DECODE_ARGS)

except jwt.ExpiredSignatureError:
self.refresh_jwt()
self.user = self.auth.user.username
self.access_token = self.auth.access_token

return {"Authorization": "Bearer " + self.access_token}

def obtain_jwt(self):
r = self.client.post("/v1/token/obtain/", json=self.creds)
if r.status_code == 401:
raise ValueError(
"Credentials not valid to connect to https://manager.ece.iiasa.ac.at."
)
elif r.status_code >= 400:
def _get_anonymous_token(self):
r = self.client.get("/legacy/anonym/")
if r.status_code >= 400:
raise ValueError("Unknown API error: " + r.text)
self.user, self.access_token = None, r.json()

_json = r.json()
self.access_token = _json["access"]
self.refresh_token = _json["refresh"]

def refresh_jwt(self):
def __call__(self):
try:
# raises jwt.ExpiredSignatureError if token is expired
jwt.decode(self.refresh_token, options=JWT_DECODE_ARGS)
r = self.client.post(
"/v1/token/refresh/", json={"refresh": self.refresh_token}
jwt.decode(
self.access_token,
options={"verify_signature": False, "verify_exp": True},
)
if r.status_code >= 400:
raise ValueError("Unknown API error: " + r.text)
self.access_token = r.json()["access"]
except jwt.ExpiredSignatureError:
self.obtain_jwt()
if self.auth.user.username == "@anonymous":
self._get_anonymous_token()
else:
self.auth.refresh_or_reobtain_jwt()
self.access_token = self.auth.access_token

return {"Authorization": "Bearer " + self.access_token}


class Connection(object):
Expand Down
1 change: 1 addition & 0 deletions pyam/logging.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ def adjust_log_level(logger="pyam", level="ERROR"):

def deprecation_warning(msg, item="This method", stacklevel=3):
"""Write deprecation warning to log"""
warnings.simplefilter("always", DeprecationWarning)
message = f"{item} is deprecated and will be removed in future versions. {msg}"
warnings.warn(message, DeprecationWarning, stacklevel=stacklevel)

Expand Down
10 changes: 5 additions & 5 deletions setup.cfg
Original file line number Diff line number Diff line change
Expand Up @@ -19,25 +19,25 @@ classifiers =
[options]
packages = pyam
include_package_data = True
python_requires = >=3.8, <3.12
python_requires = >=3.10, <3.12

# NOTE TO DEVS
# If you change a minimum version below, please explicitly implement the change
# in our minimum-reqs test in the file ./.github/workflows/pytest-dependency.yml
# Please also add a section "Dependency changes" to the release notes
install_requires =
iam-units >= 2020.4.21
numpy >= 1.19.0, < 1.24
ixmp4 >= 0.2.0
numpy >= 1.23.0, < 1.24
requests
pyjwt
httpx[http2]
openpyxl
# remove exception in test_io.py when changing requirement to pandas ≥ 1.5
pandas >= 1.2.0
pandas >= 2.0.0
scipy
pint >= 0.13
PyYAML
matplotlib >= 3.5.0, < 3.7.1
matplotlib >= 3.6.0, < 3.7.1
seaborn
six
setuptools >= 41
Expand Down
Loading

0 comments on commit 525ede9

Please sign in to comment.