Merge pull request #218 from iiasa/update_steel_rebase_v2

Update MESSAGEix-Materials with new steel technologies

doc/main.bib

@@ -1711,3 +1711,12 @@ @article{popp_2017_SSPlanduse
   volume = {42},
   year = {2017}
 }
+
+@article{unlu_2024_materials,
+  author = {Gamze Ünlü and Florian Maczek and Jihoon Min and Stefan Frank and Fridolin Glatter and Paul Natsuo Kishimoto and Jan Streeck and Nina Eisenmenger and Volker Krey and Dominik Wiedenhofer},
+  doi = {https://doi.org/10.5194/egusphere-2023-3035},
+  journal = {EGUsphere [preprint]},
+  title = {MESSAGEix-Materials v1.0.0: Representation of Material Flows and Stocks in an Integrated Assessment Model},
+  url = {https://egusphere.copernicus.org/preprints/2024/egusphere-2023-3035/},
+  year = {2024}
+}

doc/material/index.rst

@@ -18,12 +18,21 @@ Code reference
 .. automodule:: message_ix_models.data.material
   :members:
 
+.. note::
+   See also :pull:`130`/the archived branch `materials-migrate <https://github.com/iiasa/message-ix-models/tree/migrate-materials>`_ for a distinct version of :mod:`.material`.
+   That earlier PR was superseded by :pull:`188`, but contains the 1.0.0 version of MESSAGEix-Materials, which was used for the first submission of :cite:`unlu_2024_materials`. The model structure is almost identical to the default model that was added by :pull:`188`.
+   Compared to :pull:`188` this version differs particularly in the following areas:
+
+   - Older base year calibration of "other industries" using outdated IEA EWEB data.
+   - Material demands computed in R through ``rpy2``, instead of Python implementation.
+   - Less accurate regional allocation/aggregation of base year demands for cement and steel.
+   - No use of :mod:`.tools.costs`.
+
 Data preparation
 ----------------
 
 These scripts are used to prepare and read the data into the model.
-They can be turned on and off individually under `DATA_FUNCTIONS` in `__init__.py`.
-For example, the buildings script (`data_buildings.py`) is only used when the buildings model outputs are given explicitly without linking the CHILLED/STURM model through a soft link.
+They can be turned on and off individually under ``DATA_FUNCTIONS`` in :mod:`__init__.py`.
 
 .. automodule:: message_ix_models.model.material.data_aluminum
   :members:
@@ -40,16 +49,13 @@ For example, the buildings script (`data_buildings.py`) is only used when the bu
 .. automodule:: message_ix_models.model.material.data_power_sector
   :members:
 
-.. automodule:: message_ix_models.model.material.data_buildings
-  :members:
-
 .. automodule:: message_ix_models.model.material.data_generic
   :members:
 
 .. automodule:: message_ix_models.model.material.data_ammonia_new
   :members:
 
-.. automodule:: message_ix_models.model.material.data_methanol_new
+.. automodule:: message_ix_models.model.material.data_methanol
   :members:
 
 Build and Solve the model from CLI
@@ -63,12 +69,12 @@ Use ``mix-models materials-ix {SSP} build`` to add the material implementation o
 
     mix-models \
         --url="ixmp://ixmp_dev/MESSAGEix-GLOBIOM 1.1-R12/baseline_DEFAULT#21" \
-        --local-data "./data" material-ix SSP2 build --tag test
+        --local-data "./data" material-ix SSP2 build --tag test --nodes R12
 
-The output scenario name will be baseline_DEFAULT_test. An additional tag `--tag` can be used to add an additional suffix to the new scenario name.
-The mode option `--mode` has two different inputs 'by_url' (by default) or 'by_copy'.
-The first one uses the provided url to add the materials implementation on top of the scenario from the url.
-This is the default option. The latter is used to create a 2 degree mitigation scenario with materials by copying carbon prices to the scenario that is specified by `--scenario_name`::
+The output scenario name will be baseline_DEFAULT_test. An additional tag ``--tag`` can be used to add an additional suffix to the new scenario name.
+The mode option ``--mode`` has two different inputs 'by_url' (by default) or 'by_copy'.
+The first one uses the provided ``--url`` to add the materials implementation on top of the scenario from the url.
+This is the default option. The latter is used to create a 2 degree mitigation scenario with materials by copying carbon prices to the scenario that is specified by ``--scenario_name``::
 
     mix-models --url="ixmp://ixmp_dev/MESSAGEix-Materials/scenario_name" material-ix \
      build --tag test --mode by_copy
@@ -78,14 +84,22 @@ This command line only builds the scenario but does not solve it. To solve the s
     mix-models --url="ixmp://ixmp_dev/MESSAGEix-Materials/scenario_name" material-ix \
      SSP2 solve --add_calibration False --add_macro False
 
-The solve command has the `--add_calibration` option to add MACRO calibration to a baseline scenario. `--add_macro` option solves the scenario with MACRO.
-Both options are False by default.To first calibrate the scenario and then solve that scenario with MACRO both options should be set to True.
+The solve command has the ``--add_calibration`` option to add MACRO calibration to a baseline scenario with a valid calibration file specified with ``--macro-file``.
+The ``--add_macro`` option determines whether the scenario should be solved with MESSAGE or MESSAGE-MACRO.
+MESSAGEix-Materials provides one calibration file that is only compatible with scenarios with first model year 2025 and the common model structure of a MESSAGEix-GLOBIOM scenario.
+To first calibrate the scenario and then solve that scenario with MACRO both options should be set to :any`True`.
+
+It is also possible to shift the first model year and solve a clone with shifted years with ``--shift_model_year``.
+If ``--shift_model_year`` is set together with the macro options the model year will be shifted before the MACRO calibration.
+
+All three options are :any:`False` by default.
+
 
 Reporting
 =========
 
 The reporting generates specific variables related to materials, mainly Production and Final Energy.
-The resulting reporting file is generated under message_ix_models\\data\\material\\reporting_output with the name “New_Reporting_MESSAGEix-Materials_scenario_name.xlsx”.
+The resulting reporting file is generated under :file:`message_ix_models/data/material/reporting_output` with the name “New_Reporting_MESSAGEix-Materials_scenario_name.xlsx”.
 More detailed variables related to the whole energy system and emissions are not included in this reporting.
 
 Reporting is executed by the following command::
@@ -106,35 +120,46 @@ Binary/raw data files
 
 The code relies on the following input files, stored in :file:`data/material/`:
 
+**Cement**
+
 :file:`CEMENT.BvR2010.xlsx`
    Historical cement demand data
 
-:file:`STEEL_database_2012.xlsx`
-  Historical steel demand data
+:file:`Global_cement_MESSAGE.xlsx`
+  Techno-economic parametrization data for cement sector combined (R12)
 
-:file:`Global_steel_cement_MESSAGE.xlsx`
-  Techno-economic parametrization data for steel and cement sector combined (R12)
+:file:`demand_cement.yaml`
+  Base year demand data
 
-:file:`demand_aluminum.xlsx`
-  Historical aluminum demand data
+**Aluminum**
 
 :file:`demand_aluminum.xlsx`
   Historical aluminum demand data
 
+:file:`aluminum_trade_data.csv`
+  Data retrieved from IAI MFA model and used for trade calibration
+
 :file:`aluminum_techno_economic.xlsx`
   Techno-economic parametrization data for aluminum sector
 
-:file:`generic_furnace_boiler_techno_economic.xlsx`
-  Techno-economic parametrization data for generic furnace technologies
+:file:`demand_aluminum.yaml`
+  Base year aluminum demand data
 
-:file:`iamc_db ENGAGE baseline GDP PPP.xlsx`
-  SSP GDP projection used for material demand projections
+**Iron and Steel**
 
-:file:`MESSAGEix-Materials_final_energy_industry.xlsx`
-  Final energy values to calibrate base year values for industries
+:file:`STEEL_database_2012.xlsx`
+  Historical steel demand data
 
-:file:`residual_industry_2019.xlsx`
-  Final energy values to calculate the residual industry demand
+:file:`Global_steel_MESSAGE.xlsx`
+  Techno-economic parametrization data for steel sector combined (R12)
+
+:file:`worldsteel_steel_trade.xlsx`
+  Historical data from `worldsteel Association <https://worldsteel.org/data)>`_
+
+:file:`demand_steel.yaml`
+  Base year steel demand data
+
+**Chemicals**
 
 :file:`nh3_fertilizer_demand.xlsx`
   Nitrogen fertilizer demand
@@ -163,6 +188,29 @@ The code relies on the following input files, stored in :file:`data/material/`:
 :file:`steam_cracking_hist_new_cap.csv`
   Steam cracker historical capacities in R12 regions
 
+:file:`ammonia/demand_NH3.yaml`
+   Ammonia demand in each R12 region in year 2020
+
+:file:`petrochemicals/demand_HVC.yaml`
+   HVC demand in each R12 region in year 2020
+
+:file:`methanol/demand_methanol.yaml`
+   Methanol demand in each R12 region in year 2020
+
+:file:`/methanol/results_material_SHAPE_comm.csv`
+   Commercial sector data from MESSAGEix-Buidings SHAPE scenario used to get wood demands from buildings to estimate resin demands
+
+:file:`/methanol/results_material_SHAPE_resid.csv`
+   Residential sector data from MESSAGEix-Buidings SHAPE scenario used to get wood demands from buildings to estimate resin demands
+
+:file:`/methanol/results_material_SSP2_comm.csv`
+   Commercial sector data from MESSAGEix-Buidings SSP2 scenario used to get wood demands from buildings to estimate resin demands
+
+:file:`/methanol/results_material_SSP2_resid.csv`
+   Residential sector data from MESSAGEix-Buidings SSP2 scenario used to get wood demands from buildings to estimate resin demands
+
+**Power sector**
+
 :file:`NTNU_LCA_coefficients.xlsx`
    Material intensity (and other) coefficients for power plants based on lifecycle assessment (LCA) data from the THEMIS database, compiled in the `ADVANCE project <http://www.fp7-advance.eu/?q=content/environmental-impacts-module>`_.
 
@@ -175,6 +223,32 @@ The code relies on the following input files, stored in :file:`data/material/`:
 :file:`LCA_commodity_mapping.xlsx`
    Commodity mapping (for materials) of global 11-regional MESSAGEix-GLOBIOM model to commodities of THEMIS LCA dataset.
 
+
+**Buildings**
+
+:file:`buildings/LED_LED_report_IAMC_sensitivity_R12.csv`
+   Data from MESSAGEix-Buidings LED scenarios used to get steel/cement/aluminum demands from buildings
+
+:file:`buildings/report_IRP_SSP2_BL_comm_R12.csv`
+   Commerical sector data from MESSAGEix-Buidings used to get steel/cement/aluminum demands from buildings
+
+:file:`buildings/report_IRP_SSP2_BL_resid_R12.csv`
+   Residential sector data from MESSAGEix-Buidings used to get steel/cement/aluminum demands from buildings
+
+**Other**
+
+:file:`generic_furnace_boiler_techno_economic.xlsx`
+  Techno-economic parametrization data for generic furnace technologies
+
+:file:`iamc_db ENGAGE baseline GDP PPP.xlsx`
+  SSP GDP projection used for material demand projections
+
+:file:`MESSAGEix-Materials_final_energy_industry.xlsx`
+  Final energy values to calibrate base year values for industries
+
+:file:`residual_industry_2019.xlsx`
+  Final energy values to calculate the residual industry demand
+
 :file:`SSP_UE_dyn_input.xlsx`
    Calibration file for industry end-use energy demands
 
@@ -190,36 +264,29 @@ The code relies on the following input files, stored in :file:`data/material/`:
 :file:`iea_mappings/industry.csv`
    Mapping of MESSAGEix-GLOBIOM industry technologies to IEA EWEB products of industry sector flows not covered by MESSAGEix-Materials
 
-:file:`ammonia/demand_NH3.yaml`
-   Ammonia demand in each R12 region in year 2020
-
 :file:`other/mer_to_ppp_default.csv`
    Default conversion factors for GDP MER to PPP used in MESSAGEix-GLOBIOM SSP2 scenarios.
    Used to create demand projections for steel/cement/aluminum/chemicals if GDP_PPP data is not in scenario
 
-:file:`buildings/LED_LED_report_IAMC_sensitivity_R12.csv`
-   Data from MESSAGEix-Buidings LED scenarios used to get steel/cement/aluminum demands from buildings
 
-:file:`buildings/report_IRP_SSP2_BL_comm_R12.csv`
-   Commerical sector data from MESSAGEix-Buidings used to get steel/cement/aluminum demands from buildings
 
-:file:`buildings/report_IRP_SSP2_BL_resid_R12.csv`
-   Residential sector data from MESSAGEix-Buidings used to get steel/cement/aluminum demands from buildings
+:file:`material/set.yaml`
+----------------------------
 
-:file:`/methanol/results_material_SHAPE_comm.csv`
-   Commercial sector data from MESSAGEix-Buidings SHAPE scenario used to get wood demands from buildings to estimate resin demands
+.. literalinclude:: ../../message_ix_models/data/material/set.yaml
+   :language: yaml
 
-:file:`/methanol/results_material_SHAPE_resid.csv`
-   Residential sector data from MESSAGEix-Buidings SHAPE scenario used to get wood demands from buildings to estimate resin demands
 
-:file:`/methanol/results_material_SSP2_comm.csv`
-   Commercial sector data from MESSAGEix-Buidings SSP2 scenario used to get wood demands from buildings to estimate resin demands
+Release notes
+=============
 
-:file:`/methanol/results_material_SSP2_resid.csv`
-   Residential sector data from MESSAGEix-Buidings SSP2 scenario used to get wood demands from buildings to estimate resin demands
 
-:file:`material/set.yaml`
-----------------------------
+This is the list of changes to MESSAGEix-Materials between each release.
 
-.. literalinclude:: ../../message_ix_models/data/material/set.yaml
-   :language: yaml
+
+**Version 1.1.0**
+
+.. toctree::
+   :maxdepth: 2
+
+   v1.1.0

doc/material/v1.1.0.rst

@@ -0,0 +1,80 @@
+Version 1.1.0 (August 27, 2024)
+-------------------------------
+
+This is a release of MESSAGEix-Materials and includes model additions and enhancements.
+There are also a few of changes in the command line interface.
+
+Model changes
+~~~~~~~~~~~~~
+
+**Iron & steel sector**
+
+This release introduces the following low-emission options:
+
+- Direct reduction of iron with hydrogen
+- Direct reduction of iron with natural gas
+- Direct reduction of iron with natural gas and CCS
+- Blast furnace with CCS
+- Blast furnace using charcoal
+- Basic oxygen furnace with top gas recirculation
+
+Global trade of finished steel has been calibrated according to wordsteel Association.
+
+Shared input data file :file:`Global_steel_cement_MESSAGE.xlsx` has been separated for steel and cement data.
+
+**Non metallic minerals sector**
+
+Shared input data file has been separated for steel and cement data.
+
+CCS addon reformulated to prohibit wet CCS addon on dry clinker kiln and vice versa.
+
+**Non ferrous metals sector**
+
+Global trade of aluminum has been calibrated according to IAI MFA model.
+
+**Power sector**
+
+Material intensity of hydro power plants updated (reference?)
+
+**Demand generator**
+
+GDP and population projections are read from scenarios by default by retrieving the parametrization of ``bound_activity_up`` for the technologies ``GDP``, ``GDP_PPP`` and ``Population``.
+If the data cannot be retrieved from the scenario parameters, default projections from an exogenous data file are used.
+
+**Other**
+
+MESSAGEix-Materials can be built on the :ref:`model-bare` now.
+
+Convenience function :func:`data_util.calculate_ini_new_cap` to calculate ``initial_new_capacity_up/lo`` based on demand parameter.
+
+MACRO calibration is now available for non-IIASA users who did not have access to the previously required MACRO calibration file.
+See :ref:`CLI Changes`
+
+An option has been added to shift the first model year of a scenario to a desired year before solving.
+See :ref:`CLI Changes`
+
+
+CLI changes
+~~~~~~~~~~~
+
+The :func:`build_scen` command uses the ``mix-models`` CLI parameter ``--nodes`` now.
+Currently ``R12`` is the only supported option.
+
+Scenarios can now be calibrated with MACRO by setting ``--add_calibration=True`` and setting ``--macro-file`` to an existing filename located in :file:`data/material/macro/`.
+
+Solving a scenario with a different first model year is possible with e.g.::
+
+   mix-models --url="ixmp://ixmp_dev/MESSAGEix-Materials/scenario_name" material-ix \
+     SSP2 solve --shift_model_year=2025
+
+Deprecations
+~~~~~~~~~~~~~
+The ``material-ix`` commands ``add-buildings-ts``, ``report-2`` and ``create-bare`` have been removed.
+
+New features
+~~~~~~~~~~~~
+
+**Bug Fixes**
+
+
+**Compatibility**

doc/whatsnew.rst

@@ -17,6 +17,9 @@ Next release
   - Replace solar and wind technologies with new ones (:pull:`206`).
   - Reorganize input files and incorporate `first_year.csv` data into `tech_map.csv` (:pull:`221`).
   - Reconfigure use and implementation of technology variants/modules to be more agnostic (:pull:`221`).
+- Improve and extend :doc:`/material/index` (:pull:`218`).
+
+  - Release of MESSAGEix-Materials 1.1.0 (:doc:`/material/v1.1.0`).
 
 v2024.8.6
 =========