ICOS CO2 Reader

[lewisblake]

Designed to close #862

[codecov]

Codecov Report

Attention: 4 lines in your changes are missing coverage. Please review.

Comparison is base (2de33c6) 80.41% compared to head (12a45eb) 80.42%.

Additional details and impacted files

@@             Coverage Diff              @@
##           main-dev     #888      +/-   ##
============================================
+ Coverage     80.41%   80.42%   +0.01%     
============================================
  Files           101      102       +1     
  Lines         17433    17467      +34     
============================================
+ Hits          14018    14048      +30     
- Misses         3415     3419       +4     

Flag	Coverage Δ
unittests	80.42% <88.88%> (+0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
pyaerocom/aeroval/fairmode_stats.py	92.10% <100.00%> (ø)
pyaerocom/config.py	83.56% <100.00%> (+0.03%)	⬆️
pyaerocom/plugins/icos/reader.py	87.87% <87.87%> (ø)

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[lewisblake]

For this PR we need to

• Download the data, aggregate it to yearly files (e.g., using cdo), place the aggregated files in lustre/storeB/project/aerocom/aerocom1/AEROCOM_OBSDATA/ICOS/aggregated/
  • Python package to download and load into memory: https://icos-carbon-portal.github.io/pylib/
  • Or more traditional download and archive approach: https://github.com/ICOS-Carbon-Portal/data
  • CAMS server
• Choose sample files (I have some locally) for testing, write tests, and place test files on the server
• Run a few Aeroval experiments as tests (obs only, with IFS output)

[lewisblake]

ICOS Carbon Portal SPARQL Client query to get CO2 data

prefix cpmeta: <http://meta.icos-cp.eu/ontologies/cpmeta/>
prefix prov: <http://www.w3.org/ns/prov#>
prefix xsd: <http://www.w3.org/2001/XMLSchema#>
select ?dobj ?hasNextVersion ?spec ?fileName ?size ?submTime ?timeStart ?timeEnd
where {
	VALUES ?spec {<http://meta.icos-cp.eu/resources/cpmeta/atcCo2L2DataObject>}
	?dobj cpmeta:hasObjectSpec ?spec .
	BIND(EXISTS{[] cpmeta:isNextVersionOf ?dobj} AS ?hasNextVersion)
	VALUES ?station {<http://meta.icos-cp.eu/resources/stations/AS_ZSF> <http://meta.icos-cp.eu/resources/stations/AS_HEL> <http://meta.icos-cp.eu/resources/stations/AS_PAL> <http://meta.icos-cp.eu/resources/stations/AS_TRN> <http://meta.icos-cp.eu/resources/stations/AS_GAT> <http://meta.icos-cp.eu/resources/stations/AS_HPB> <http://meta.icos-cp.eu/resources/stations/AS_OXK> <http://meta.icos-cp.eu/resources/stations/AS_IPR> <http://meta.icos-cp.eu/resources/stations/AS_WES> <http://meta.icos-cp.eu/resources/stations/AS_RUN> <http://meta.icos-cp.eu/resources/stations/AS_STE> <http://meta.icos-cp.eu/resources/stations/AS_OPE> <http://meta.icos-cp.eu/resources/stations/AS_CBW> <http://meta.icos-cp.eu/resources/stations/AS_CMN> <http://meta.icos-cp.eu/resources/stations/AS_PUI> <http://meta.icos-cp.eu/resources/stations/AS_BIR> <http://meta.icos-cp.eu/resources/stations/AS_UTO> <http://meta.icos-cp.eu/resources/stations/AS_SNO> <http://meta.icos-cp.eu/resources/stations/AS_IZO> <http://meta.icos-cp.eu/resources/stations/AS_KIT> <http://meta.icos-cp.eu/resources/stations/AS_PRS> <http://meta.icos-cp.eu/resources/stations/AS_SMR> <http://meta.icos-cp.eu/resources/stations/AS_RGL> <http://meta.icos-cp.eu/resources/stations/AS_SAC> <http://meta.icos-cp.eu/resources/stations/AS_ZEP> <http://meta.icos-cp.eu/resources/stations/AS_JUE> <http://meta.icos-cp.eu/resources/stations/AS_LMP> <http://meta.icos-cp.eu/resources/stations/AS_TOH> <http://meta.icos-cp.eu/resources/stations/AS_KRE> <http://meta.icos-cp.eu/resources/stations/AS_SVB> <http://meta.icos-cp.eu/resources/stations/AS_LUT> <http://meta.icos-cp.eu/resources/stations/AS_HTM> <http://meta.icos-cp.eu/resources/stations/AS_SSL> <http://meta.icos-cp.eu/resources/stations/AS_JFJ> <http://meta.icos-cp.eu/resources/stations/AS_PUY> <http://meta.icos-cp.eu/resources/stations/AS_WAO> <http://meta.icos-cp.eu/resources/stations/AS_NOR> <http://meta.icos-cp.eu/resources/stations/AS_LIN>}
			?dobj cpmeta:wasAcquiredBy/prov:wasAssociatedWith ?station .
	?dobj cpmeta:hasSizeInBytes ?size .
?dobj cpmeta:hasName ?fileName .
?dobj cpmeta:wasSubmittedBy/prov:endedAtTime ?submTime .
?dobj cpmeta:hasStartTime | (cpmeta:wasAcquiredBy / prov:startedAtTime) ?timeStart .
?dobj cpmeta:hasEndTime | (cpmeta:wasAcquiredBy / prov:endedAtTime) ?timeEnd .
	FILTER NOT EXISTS {[] cpmeta:isNextVersionOf ?dobj}
	{
		{FILTER NOT EXISTS {?dobj cpmeta:hasVariableName ?varName}}
		UNION
		{
			?dobj cpmeta:hasVariableName ?varName
			FILTER (?varName = "co2")
		}
	}
		{
			{VALUES ?spec {<http://meta.icos-cp.eu/resources/cpmeta/atcCo2L2DataObject>}}
			UNION
			{VALUES ?keyword {"CO2"^^xsd:string}
	?dobj cpmeta:hasKeyword ?keyword}
		}
}
order by desc(?submTime)

[lewisblake]

We should discuss on Monday the amount of testing needed in this PR since it is a derived class of ReadMEP with no additional functionality.

[avaldebe]

only small changes related to the test fixtures