Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lower memory footprint for "fill the demSeOth with zeroes so that oneRegi tests work" #639

Merged
merged 2 commits into from
Aug 8, 2024

Conversation

lecfab
Copy link
Contributor

@lecfab lecfab commented Aug 8, 2024

Following @orichters' comment in #637, this PR reduces the memory footprint: v_demSeOth is now filled with zeroes up to a size 19x21x3 (the previous version filled it to 24000 instead of 3).

@lecfab lecfab requested a review from orichters August 8, 2024 08:48
@lecfab lecfab merged commit b6ae623 into pik-piam:master Aug 8, 2024
2 checks passed
@lecfab lecfab deleted the fill-zeroes-lowMemory branch August 8, 2024 09:06
v_demSeOth <- readGDX(gdx, c("v_demSeOth","vm_demSeOth"), field = "l", restore_zeros = T)[, y, ] * pm_conv_TWa_EJ
# SE demand of specific energy system technologies (ensure that all regions have a value)
v_demSeOth <- readGDX(gdx, c("v_demSeOth", "vm_demSeOth"), field = "l", restore_zeros = FALSE)[, y, ] * pm_conv_TWa_EJ
matrixRegionsYears <- new.magpie(cells_and_regions = getRegions(dataoc), years = y, fill = 0, sets = getSets(v_demSeOth, fulldim = FALSE))
Copy link
Contributor

@fbenke-pik fbenke-pik Aug 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it is necessary to create a dummy magclass object just for the matching here. Usually, the reference ref passed to matchDim is the magclass object you want to bind v_demSeOth to (i.e. which causes mbind to crash).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey,
I tried to matchDim v_demSeOth with tmp1, but the dimension names don't match. That's why i used an intermediate named object.
What would be a more fitting solution?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Falk's solution here: #644

@lecfab
Copy link
Contributor Author

lecfab commented Aug 12, 2024

@fbenke-pik For reproducibility, the error happens with the gdx /p/tmp/fabricel/powerRealizations/output/testOneRegi/fulldata.gdx:

Error in mbind(tmp1, setNames(dimSums(mselect(v_demSeOth, all_enty = "seh2"),  : 
  Cannot handle objects! Spatial as well as data dimensions differ!
  Differences from first mbind() input:
  spatial:
      missing: `LAM`, `OAS`, `SSA`, `NEU`, ...
       having: `EUR`
  data:
      missing: `SE (EJ/yr)`, `SE|Biomass (EJ/yr)`, `SE|Electricity (EJ/yr)`, `SE|Electricity|Combined Heat and Power w/o CC (EJ/yr)`, ...
       adding: `SE|Input|Hydrogen|Other Energy System Consumption (EJ/yr)`

Error traceback is

  1. convGDX2MIF(gdx)
  2. convGDX2MIF.R#53: mbind(output, reportSE(gdx, regionSubsetList, t)[, t, ])
  3. reportSE(gdx, regionSubsetList, t)
  4. reportSE.R#401: mbind(tmp1, setNames(dimSums(mselect(v_demSeOth, all_enty = "seh2"), dim = 3), "SE|Input|Hydrogen|Other Energy System Consumption (EJ/yr)"))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants