double FE|Transport|Freight variables in reporting

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

Variables

FE|Transport|Freight            
FE|Transport|Freight|Electricity
FE|Transport|Freight|Gases      
FE|Transport|Freight|Hydrogen   
FE|Transport|Freight|Liquids    

are reported twice in reporting, with different values, breaking further analysis.
This is present at least since 2021-10-09, but after 2021-10-02

$ for f in $( find /p/projects/remind/modeltests/output/ -name REMIND_generic_SSP2-AMT-Bas
e_withoutPlus.mif | sort ); do echo "$f"; awk -F ';' 'NR > 1 { print $3, $4 }' "$f" | sort | uniq -c | grep -m 1 "^\s*2"; done | tail
/p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-08-14_00.00.22/REMIND_generic_SSP2-AMT-Base_withoutPlus.mif
/p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-08-21_00.00.23/REMIND_generic_SSP2-AMT-Base_withoutPlus.mif
/p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-08-28_00.00.51/REMIND_generic_SSP2-AMT-Base_withoutPlus.mif
/p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-09-04_00.01.27/REMIND_generic_SSP2-AMT-Base_withoutPlus.mif
/p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-09-11_11.09.03/REMIND_generic_SSP2-AMT-Base_withoutPlus.mif
/p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-09-18_00.00.22/REMIND_generic_SSP2-AMT-Base_withoutPlus.mif
/p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-09-25_00.00.23/REMIND_generic_SSP2-AMT-Base_withoutPlus.mif
/p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-10-02_00.00.21/REMIND_generic_SSP2-AMT-Base_withoutPlus.mif
/p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-10-09_00.00.24/REMIND_generic_SSP2-AMT-Base_withoutPlus.mif
      2 CAZ FE|Transport|Freight

which points towards #116.

[fschreyer]

Ok, I think this is an issue which comes from the EDGE-T reporting and not from the standard reporting which is run by convGDX2MIF. The only variable existing in the standard reporting is FE|Transport|Freight|+|Liquids which will become FE|Transport|Freight in the mif without plus. So, I think @Loisel would need to take over.

[Loisel]

This is present at least since 2021-10-09, but after 2021-10-02

Are you sure about this @0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q ? When looking at the libraries, the duplicated variables should be there for 8 months already. Maybe some mechanism previously selected one of them for the output.

[Loisel]

Here is the line where @MariannaR most likely introduced the duplicates https://github.com/pik-piam/remind2/blame/master/R/reportEDGETransport.R#L87

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

This is present at least since 2021-10-09, but after 2021-10-02

Are you sure about this @0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q ? When looking at the libraries, the duplicated variables should be there for 8 months already. Maybe some mechanism previously selected one of them for the output.

I'm sure about SSP2-AMT-Base_2021-10-09_00.00.24 having duplicated variables in the reporting and SSP2-AMT-Base_2021-10-02_00.00.21 not having it.

And I'm reasonably sure that only remind2 had relevant changes in the meantime:

$ diff -y <( sed -n '/^==== installed package versions =====/,/^=====/ { /^ \?[0-9]\+/ p }' /p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-10-02_00.00.21/log.txt ) <( sed -n '/^==== installed package versions =====/,/^=====/ { /^ \?[0-9]\+/ p }' /p/projects/remind/modeltests/output/SSP2-AMT-Base_2021-10-09_00.00.24/log.txt )
 1 data.table    1.14.0  					1 data.table    1.14.0  
 2 devtools      2.4.2   					2 devtools      2.4.2   
 3 dplyr         1.0.7   					3 dplyr         1.0.7   
 4 edgeTrpLib    0.1.48  					4 edgeTrpLib    0.1.48  
 5 flexdashboard 0.5.2   					5 flexdashboard 0.5.2   
 6 gdx           1.51.4  					6 gdx           1.51.4  
 7 gdxdt         0.1.0   					7 gdxdt         0.1.0   
 8 gdxrrw        1.0.2   					8 gdxrrw        1.0.2   
 9 ggplot2       3.3.5   					9 ggplot2       3.3.5   
10 gtools        3.9.2   					10 gtools        3.9.2   
11 lucode        3.21.0  					11 lucode        3.21.0  
12 luplot        3.51.2  					12 luplot        3.51.2  
13 luscale       2.23.2  					13 luscale       2.23.2  
14 magclass      6.0.9   					14 magclass      6.0.9   
15 magpie        0.2266.1					15 magpie        0.2266.1
16 mip           0.131.4 				     |	16 mip           0.132.3 
17 mrremind      0.80.0  				     |	17 mrremind      0.82.0  
18 mrvalidation  2.30.3  				     |	18 mrvalidation  2.30.4  
19 optparse      1.6.6   					19 optparse      1.6.6   
20 plotly        4.9.4.1 					20 plotly        4.9.4.1 
21 remind        36.185.2					21 remind        36.185.2
22 remind2       1.46.0  				     |	22 remind2       1.46.1  
23 rlang         0.4.11  					23 rlang         0.4.11  
24 rmndt         0.5.4   					24 rmndt         0.5.4   
25 tidyverse     1.3.1   					25 tidyverse     1.3.1   
26 data.table    1.14.0  					26 data.table    1.14.0  
27 dplyr         1.0.6   					27 dplyr         1.0.6   
28 ggplot2       3.3.3   					28 ggplot2       3.3.3   
29 gtools        3.8.2   					29 gtools        3.8.2   
30 methods       3.6.3   					30 methods       3.6.3   
31 parallel      3.6.3   					31 parallel      3.6.3   
32 rlang         0.4.11  					32 rlang         0.4.11  
33 tidyverse     1.3.1   					33 tidyverse     1.3.1   
34 tools         3.6.3   					34 tools         3.6.3   

And I'm sure again that between those two runs, there was only one request merged into remind2:

$ git log --format="%C(yellow)%h %C(cyan)%an %Creset%s" 49abbfdf^..578d2196
578d219 Felix Schreyer update library version
7e23c93 Felix Schreyer add EU27 region to edge transport reporting
3fe68ed Felix Schreyer bugfix in steel production reporting
2a31aca fschreyer Merge pull request #115 from fschreyer/industry_CCS
89d34bc Felix Schreyer correct some code descriptions
49abbfd Felix Schreyer update library version

So maybe bisecting 2a31aca to 578d219 will shed light on this.