refactor cost read-in and cost manipulation - moving all cost related codes to one place, improve documentation

[cchrisgong]

Purpose of this PR

refactor cost read in and manipulation part of the core/datainput.gms, so all of the code are close together in one place

Type of change

(Make sure to delete from the Type-of-change list the items not relevant to your PR)

• Bug fix
• [ X] Refactoring
• New feature
• Minor change (default scenarios show only small differences)
• Fundamental change
• This change requires a documentation update

Checklist:

• My code follows the coding etiquette
• I performed a self-review of my own code
• I explained my changes within the PR, particularly in hard-to-understand areas
• I checked that the in-code documentation is up-to-date
• I adjusted the reporting in remind2 where it was needed
• I adjusted forbiddenColumnNames in readCheckScenarioConfig.R in case the PR leads to deprecated switches
• All automated model tests pass (FAIL 0 in the output of make test)
• The changelog CHANGELOG.md has been updated correctly

Further information (optional):

• Test runs are here:
• before (develop /trunk): /p/tmp/chengong/remind_costrefac/remind/output/testOneRegi_develop_2024-02-19_15.34.40
• after refactoring: /p/tmp/chengong/remind_costrefac/remind/output/testOneRegi_cost_2024-02-19_15.30.19
• Comparison of results (what changes by this PR?):

[robertpietzcker]

please make sure there are never more than 3 asterisks behind each other

[robertpietzcker]

does energy service capital still exist in the latest REMIND version, after transport|complex and buildings|putty were deleted?

[cchrisgong]

there are sill some

@robertpietzcker

[robertpietzcker]

this is only about costs, right? so maybe specify "technology cost data"?

[robertpietzcker]

I think it looks ok, but with this amount of code shifted, there could be errors.

I would propose you merge this to prevent merge conflicts, but do additional tests, namely compare all the parameters that might be involved (pm_data, p_inco0, pm_inco0_t, xxx)

I guess the easiest way would be to write them into a file in both the old and the new version, and do a quick winmerge/diff to see if they are exactly the same.
Please do this for both global and regionalized tech costs, and SSP1, SSP2 and SSP5

Then add a link to the folder where the old/new parameter results are saved

It would be painful if one of them changes without us being aware :-)

[robertpietzcker]

before we forget about this - did you do the comparison of the parameters before/after for the different model configurations?

[cchrisgong]

I did a test one regi in test/scenconfig_quick.csv, I believe it is ssp2? And compared endogenous parameters of capital cost. I'm submitting a patch though on some small things.

Sorry for the big change, but I think it shouldn't affect models

[robertpietzcker]

No, I was talking about my comment in my review:
"but do additional tests, namely compare all the parameters that might be involved (pm_data, p_inco0, pm_inco0_t, xxx)

I guess the easiest way would be to write them into a file in both the old and the new version, and do a quick winmerge/diff to see if they are exactly the same.
Please do this for both global and regionalized tech costs, and SSP1, SSP2 and SSP5

Then add a link to the folder where the old/new parameter results are saved"

so not a "are the results the same in a testOneRegi", but comparing the full parameters after datainput. if you do it in a file (simply write into a file with "put"), it will take 1 click on winmerge to check if they are identical; do this for all the setups (global tech costs; regional tech costs; SSP1; SSP2, SSP5) and then we know everything is 100% correct and we never need to worry if something indeed might have changed.

this is much clearer than doing a REMIND run, because the model is not deterministic and so it will be more challenging to decide "are these minimal differences just a slightly different optimum, or is something actually different in the model"

[cchrisgong]

I see, thanks, yes, I think I'll do that till Friday if that's okay. I'll submit a short patch PR just for the obvious things. I haven't used put before, I'm also in Linux system. So it will take me a day or two to get there, I hope that's okay

[robertpietzcker]

sure!
on linux, diff will work as well (or better?) than winmerge :-)

and the put utility is quite straight forward, if I remember correctly.

Thanks!