NOx unit handling

[rgieseke]

With the following snippet i get a warning for the NOx unit "'NOx' has a unit of NO2 * kt / year, which is not compatible with an emission rate":

import primap2 as pm2
import pandas as pd

df = pd.DataFrame(
    [["World", "NOx", "kt NO2 / year", 10], ["World", "CO2", "kt C / year", 10]],
    columns=["area", "entity", "unit", "2020"],
)
df

coords_terminologies = {"area": "Test"}
coords_cols = {"area": "area", "entity": "entity", "unit": "unit"}
coords_defaults = {"source": "Test"}
df_if = pm2.pm2io.convert_wide_dataframe_if(
    df,
    coords_terminologies=coords_terminologies,
    coords_cols=coords_cols,
    coords_defaults=coords_defaults,
)
pm2.pm2io.from_interchange_format(df_if)

2022-02-03 12:25:42.935 | DEBUG    | primap2.pm2io._interchange_format:from_interchange_format:321 - Expected array shapes: [[1, 1, 2], [1, 1, 2]], resulting in size 4.
2022-02-03 12:25:42.957 | WARNING  | primap2._data_format:ensure_entity_and_units_valid:365 - 'NOx' has a unit of NO2 * kt / year, which is not compatible with an emission rate.

The df looks like

  area (Test) entity           unit  2020 source
0       World    NOx  kt NO2 / year  10.0   Test
1       World    CO2    kt C / year  10.0   Test

I don't get the warning if I set the entity to NO2.

Is this some definition missing from openscm-units? Or shouldn't this be possible as NOx is ambiguous (NO as well NO2)?
CEDS uses NOx with kt NO2 as unit.

The checking code is here: https://github.com/pik-primap/primap2/blob/main/primap2/_data_format.py#L355

[mikapfl]

Hm, the warning gets confused because unit and entity are different.

Multiple levels of answers:

• It is wrong to include the entity in the unit (that ship has sailed in the climate policy community long ago)
• If we do, then yes, the entity in the unit should match the entity. So, philosophically, I think the better unit for NOx emissions is "kt NOx / year", because NOx is something else than NO2 or NO. And these units shouldn't convert into each other. That's how it is implemented in openscm-units:

In [1]: import openscm_units

In [2]: u = openscm_units.unit_registry

In [3]: 1 * u.g * u.NOx
Out[3]: 1 <Unit('NOx * gram')>

In [4]: (1 * u.g * u.NOx).to(u.g * u.NO2)
DimensionalityError: Cannot convert from 'NOx * gram' ([NOx] * [mass]) to 'NO2 * gram' ([mass] * [nitrogen])

openscm-units provides the NOx_conversions context to convert NOx to NO2 if you want to:

In [6]: with u.context("NOx_conversions"):
   ...:     a = (1 * u.g * u.NOx).to(u.g * u.NO2)

In [7]: a
Out[7]: 1.0 <Unit('NO2 * gram')>

but you have to do it explicitly

So, by extension, we also handle this the same in primap2 at the moment: NOx is its own thing, if you insist, you can convert it, but then you should still align the unit and the entity.

[rgieseke]

Thanks for the reply!
Maybe the error message could be improved to denote that entity and unit don't match.
Not sure how hard that woud be, as for CO2 (ktC) or N2O it's possible to have a mis-match from entity to unit i think.

This seems to be what CEDS does for at least some sector: https://github.com/JGCRI/CEDS/blob/da316e2244b0ff24cfeae396ed596a3e95ab4328/code/module-H/H1.1a.Aggregate_NH3_NOx_for_N2O_7BC_ext.R#L64

[mikapfl]

The unit "C" is compatible with the unit "CO2", so ensure_valid will be fine with it:

In [6]: import primap2

In [7]: import primap2.tests

In [9]: ds = primap2.tests.examples.minimal_ds()

In [22]: dv = ds["CO2"].pr.dequantify()

In [23]: dv.attrs["units"] = "C * Gg / year"

In [24]: ds["CO2"] = dv.pr.quantify()

In [25]: ds["CO2"].pint.units
Out[25]: <Unit('C * gigagram / year')>

In [26]: ds.pr.ensure_valid()

But yeah, the wording of the error message could be enhanced, true. Maybe something along the lines 'NOx' has a unit of NO2 * kt / year, which is not compatible with an emission rate of NOx.?

This error also covers the case that you e.g. have a unit of NOx kt, without specifying the timespan (so an emission / a mass, not an emission rate), and it is not entirely trivial to cover all cases with specialized error messages.

[rgieseke]

In [9]: ds = primap2.tests.examples.minimal_ds()

Useful, good to know!

'NOx' has a unit of NO2 * kt / year, which is not compatible with an emission rate of NOx.

Yes, that's better. The thing is it still works if you only convert e.g. from kt to Mt, right?

import primap2 as pm2
import pandas as pd

df = pd.DataFrame(
    [["World", "NOx", "kt NO2 / year", 10], ["World", "CO2", "kt C / year", 10]],
    columns=["area", "entity", "unit", "2020"],
)
df

coords_terminologies = {"area": "Test"}
coords_cols = {"area": "area", "entity": "entity", "unit": "unit"}
coords_defaults = {"source": "Test"}
df_if = pm2.pm2io.convert_wide_dataframe_if(
    df,
    coords_terminologies=coords_terminologies,
    coords_cols=coords_cols,
    coords_defaults=coords_defaults,
)
ds = pm2.pm2io.from_interchange_format(df_if)

ds["NOx"].squeeze().pint.to("t NO2 / year")

[rgieseke]

If we do, then yes, the entity in the unit should match the entity. So, philosophically, I think the better unit for NOx emissions is "kt NOx / year", because NOx is something else than NO2 or NO. And these units shouldn't convert into each other. That's how it is implemented in openscm-units:

Thinking about it more, i wonder whether it's more a general question how far one wants to keep data in their 'original' format.
CEDS uses NOx with kt NO2/year and certain implicit assumptions.
Changing either adds a further layer on top of this.

It is wrong to include the entity in the unit (that ship has sailed in the climate policy community long ago)

Maybe it's good to have both :-)

[mikapfl]

Thinking about it more, i wonder whether it's more a general question how far one wants to keep data in their 'original' format.
CEDS uses NOx with kt NO2/year and certain implicit assumptions.
Changing either adds a further layer on top of this.

So far, we have erred on the side of homogenization in PRIMAP. So, we value homogeneity of the datasets in our database over keeping the original format. But that is of course a policy choice for each database and application, not something which can be decided once and for all.

[rgieseke]

Yeah, and i guess it makes sense that way. In RCMIP NOx in kt NOx/year was used as well. I think it's just becoming tricky if you start making available or sharing datasets in primap2 format (discussed this a bit with @JGuetschow). Then ideally it needs to be documented in some way if such changes are made. In this case i'm sure NOx/year as a unit is better.