Add data providers for SSP data

.gitattributes

@@ -1,6 +1,6 @@
 # Reduce the number of merge conflicts
 doc/whatsnew.rst merge=union
 # Git LFS
-*.csv.gz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
 *.xlsx filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -38,20 +38,21 @@ pip-log.txt
 pip-delete-this-directory.txt
 
 # Unit test / coverage reports
-htmlcov/
-.tox/
-.nox/
 .benchmarks
+.cache
 .coverage
 .coverage.*
-.cache
-nosetests.xml
-coverage.xml
-*.cover
-*.py,cover
 .hypothesis/
+.nox/
 .pytest_cache/
 .ruff_cache
+.tox/
+*.cover
+*.py,cover
+coverage.xml
+htmlcov/
+nosetests.xml
+/prof/
 
 # Translations
 *.mo

.pre-commit-config.yaml

@@ -10,7 +10,7 @@ repos:
     language: python
     entry: bash -c ". ${PRE_COMMIT_MYPY_VENV:-/dev/null}/bin/activate 2>/dev/null; mypy $0 $@"
     additional_dependencies:
-    - "mypy <1.5"
+    - mypy
     - pytest
     - sdmx1
     - types-PyYAML

MANIFEST.in

@@ -1 +1,2 @@
 prune message_ix_models/data/test/MESSAGEix-GLOBIOM_1.1_R11_no-policy_baseline
+prune message_ix_models/data/test/ssp

doc/api/project.rst

@@ -1,7 +1,11 @@
 .. currentmodule:: message_ix_models.project
 
-Specific research projects (:mod:`.project`)
-********************************************
+Specific research projects (:mod:`~message_ix_models.project`)
+**************************************************************
+
+.. contents::
+   :local:
+   :backlinks: none
 
 .. automodule:: message_ix_models.project
    :members:
@@ -12,5 +16,41 @@ Specific research projects (:mod:`.project`)
 Shared Socioeconomic Pathways (:mod:`.project.ssp`)
 ===================================================
 
+Structure
+---------
+
+The enumerations :obj:`SSP_2017` and :obj:`SSP_2024` contain one member from the corresponding SDMX code lists.
+These can be used to uniquely identify both an SSP narrative *and* the set in which it occurs, in applications where this distinction is meaningful:
+
+.. code-block:: py
+
+   >>> from message_ix_models.project.ssp import SSP_2017, SSP_2024
+   >>> x = SSP_2017["2"]
+   >>> y = SSP_2024["2"]
+   >>> str(y)
+   "ICONICS:SSP(2024).2"
+   >>> x == y
+   False
+
 .. automodule:: message_ix_models.project.ssp
    :members:
+
+Data
+----
+
+.. automodule:: message_ix_models.project.ssp.data
+   :members:
+
+   Although free, neither the 2017 or 2024 SSP data can be downloaded automatically.
+   Both sources require that users first submit personal information to register before being able to retrieve the data.
+   :mod:`message_ix_models` does not circumvent this requirement.
+   Thus:
+
+   - A copy of the data are stored in :mod:`message_data`.
+   - :mod:`message_ix_models` contains only a ‘fuzzed’ version of the data (same structure, random values) for testing purposes.
+
+   .. todo:: Allow users without access to :mod:`message_data` to read a local copy of this data from a :attr:`.Config.local_data` subdirectory.
+
+   .. autosummary::
+      SSPOriginal
+      SSPUpdate

doc/api/report/index.rst

@@ -25,7 +25,7 @@ Not public:
 Introduction
 ============
 
-See :doc:`the discussion in the MESSAGEix docs <message_ix:reporting>` about the stack.
+See :doc:`the discussion in the MESSAGEix docs <message-ix:reporting>` about the stack.
 In short, :mod:`message_ix` must not contain reporting code that references ``coal_ppl``, because not every model built on the MESSAGE framework will have a technology with this name.
 Any reporting specific to ``coal_ppl`` must be in :mod:`message_ix_models`, since all models in the MESSAGEix-GLOBIOM family will have this technology.
 
@@ -95,21 +95,35 @@ API reference
 
    .. autosummary::
 
+      Config
       prepare_reporter
       register
       report
 
+.. currentmodule:: message_ix_models.report.plot
+
+Plots
+-----
+
+.. automodule:: message_ix_models.report.plot
+   :members:
+
+.. currentmodule:: message_ix_models.report.computations
+
 Operators
 ---------
 
-.. currentmodule:: message_ix_models.report.computations
 .. automodule:: message_ix_models.report.computations
    :members:
 
    :mod:`message_ix_models.report.computations` provides the following:
 
    .. autosummary::
 
+      codelist_to_groups
+      compound_growth
+      exogenous_data
+      filter_ts
       from_url
       get_ts
       gwp_factors
@@ -158,7 +172,7 @@ Utilities
 
 
 Command-line interface
-----------------------
+======================
 
 .. currentmodule:: message_ix_models.report.cli
 .. automodule:: message_ix_models.report.cli
@@ -192,3 +206,11 @@ Command-line interface
      -o, --output PATH     Write output to file instead of console.
      --from-file FILE      Report multiple Scenarios listed in FILE.
      --help                Show this message and exit.
+
+Testing
+=======
+
+
+.. currentmodule:: message_ix_models.report.sim
+.. automodule:: message_ix_models.report.sim
+   :members:

doc/api/tools.rst

@@ -24,17 +24,34 @@ Exogenous data (:mod:`.tools.exo_data`)
 
 .. automodule:: message_ix_models.tools.exo_data
    :members:
-   :exclude-members: ExoDataSource
+   :exclude-members: ExoDataSource, prepare_computer
 
    .. autosummary::
 
       MEASURES
       SOURCES
       DemoSource
       ExoDataSource
+      iamc_like_data_for_query
       prepare_computer
       register_source
 
+.. autofunction:: prepare_computer
+
+   The first returned key, like ``{measure}:n-y``, triggers the following computations:
+
+   1. Load data by invoking a :class:`ExoDataSource`.
+   2. Aggregate on the |n| (node) dimension according to :attr:`.Config.regions`.
+   3. Interpolate on the |y| (year) dimension according to :attr:`.Config.years`.
+
+   Additional key(s) include:
+
+   - ``{measure}:n-y:y0 indexed``: same as ``{measure}:n-y``, indexed to values as of |y0| (the first model year).
+
+   See particular data source classes, like :class:`.SSPOriginal`, for particular examples of usage.
+
+   .. todo:: Extend to also prepare to compute values indexed to a particular |n|.
+
 .. autoclass:: ExoDataSource
    :members:
    :special-members: __init__, __call__

doc/api/util.rst

@@ -57,7 +57,6 @@ Commonly used:
 
 .. autodata:: message_ix_models.util.cache.SKIP_CACHE
 
-
 :mod:`.util.click`
 ==================
 
@@ -66,6 +65,11 @@ Commonly used:
 .. automodule:: message_ix_models.util.click
    :members:
 
+   :data:`PARAMS` contains, among others:
+
+   - :program:`--urls-from-file=…` Path to a file containing scenario URLs, one per line.
+     These are parsed and stored on :attr:`.Config.scenarios`.
+
 :mod:`.util.config`
 ===================
 
@@ -181,6 +185,13 @@ Commonly used:
 .. automodule:: message_ix_models.util.pooch
    :members:
 
+:mod:`.util.pycountry`
+======================
+
+.. currentmodule:: message_ix_models.util.pycountry
+
+.. automodule:: message_ix_models.util.pycountry
+   :members:
 
 :mod:`.util.scenarioinfo`
 =========================

doc/conf.py

@@ -53,9 +53,16 @@
 }
 
 rst_prolog = """
+.. role:: py(code)
+   :language: python
+
 .. |Code| replace:: :class:`~sdmx.model.common.Code`
 .. |Platform| replace:: :class:`~ixmp.Platform`
 .. |Scenario| replace:: :class:`~message_ix.Scenario`
+
+.. |n| replace:: :math:`n`
+.. |y| replace:: :math:`y`
+.. |y0| replace:: :math:`y_0`
 """
 
 

doc/whatsnew.rst

@@ -1,8 +1,25 @@
 What's new
 **********
 
-.. Next release
-.. ============
+Next release
+============
+
+- New providers for exogenous data from the :class:`.SSPOriginal` and :class:`.SSPUpdate` (:pull:`125`) sources.
+- Improved :class:`.ScenarioInfo` (:pull:`125`):
+
+  - New attributes :attr:`~.ScenarioInfo.model`, :attr:`~.ScenarioInfo.scenario`, :attr:`~.ScenarioInfo.version`, and (settable) :attr:`~.ScenarioInfo.url`; class method :meth:`~.ScenarioInfo.from_url` to allow storing |Scenario| identifiers on ScenarioInfo objects.
+  - New property :attr:`~.ScenarioInfo.path`, giving a valid path name for scenario-specific file I/O.
+
+- Improvements to :mod:`~message_ix_models.report` (:pull:`125`):
+
+  - New :class:`.report.Config` class collecting recognized settings for the module.
+  - :py:`context["report"]` always exists as an instance of :class:`.report.Config`.
+  - New submodule :class:`.report.plot` with base class and 5 plots of time-series data stored on Scenarios.
+  - Submodule :class:`.report.sim` provides :func:`add_simulated_solution` for testing reporting configuration.
+  - New operator :func:`.filter_ts`.
+
+- New reusable command-line option :program:`--urls-from-file` in :mod:`.util.click` (:pull:`125`).
+- Add `pyarrow <https://pypi.org/project/pyarrow/>`_ to dependencies (:pull:`125`).
 
 v2023.9.12
 ==========