try to fix airnow files with strange file encodings

[jgriesfeller]

Another approach:
determine file encoding beforehand and provide pandas with that

[codecov]

Codecov Report

Attention: 82 lines in your changes are missing coverage. Please review.

Comparison is base (47a645c) 79.45% compared to head (b50be24) 79.09%.

Additional details and impacted files

@@             Coverage Diff              @@
##           main-dev     #895      +/-   ##
============================================
- Coverage     79.45%   79.09%   -0.36%     
============================================
  Files           103      103              
  Lines         17821    17910      +89     
============================================
+ Hits          14159    14166       +7     
- Misses         3662     3744      +82     

Flag	Coverage Δ
unittests	79.09% <31.09%> (-0.36%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
pyaerocom/io/read_airnow.py	66.13% <31.09%> (-32.02%)	⬇️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jgriesfeller]

This is till not the final solution because it takes far too much RAM reading the entire dataset since the file reader reads all variables and keeps all that data in RAM. On the other hand the variables are read by aeroval variable by variable so the multi variable reading ability of the current reader is entirely useless.
This needs to be optimised so that only one variable is read at a time.
As it is now for reading reading vmro3 60GB of RAM are not enough.

[jgriesfeller]

Just for documentation: Not even 75GB is enough to read vmro3

[jgriesfeller]

Just as comment: The earlier years (rep 2017) will be added as utf-8 encoded files. Producing them is external work.

[lewisblake]

🎃