-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
memory and mbind problems in convGDX2MIF / reportFE #549
Comments
which returns:
|
I rather think of a one-sided function that restricts and expands
|
"Nicht mein Zirkus, nicht meine Affen." |
region <- c("AAA", "BBB", "CCC")
t <- paste0("y", c(2000, 2005, 2010))
name <- c("foo", "bar", "bazz")
A <- new.magpie(cells_and_regions = region, years = t, names = name, fill = 1,
sets = c("region", "t", "name"))
# spatial ----
## x smaller then ref ----
expect_equal(somebody_name_this_function(A[region[-2],,], A),
`mselect<-`(A, region = region[2], value = 0))
## x larger then ref ----
expect_equal(somebody_name_this_function(A, A[region[-2],,]),
A[region[-2],,])
# temporal ----
## x smaller then ref ----
expect_equal(somebody_name_this_function(A[,t[-2],], A),
`mselect<-`(A, t = t[2], value = 0))
## x larger then ref ----
expect_equal(somebody_name_this_function(x = A, ref = A[,t[-2],]),
A[,t[-2],])
# names ----
## x smaller then ref ----
expect_equal(somebody_name_this_function(A[,,name[-2]], A),
`mselect<-`(A, name = name[2], value = 0))
## x larger then ref ----
expect_equal(somebody_name_this_function(A, A[,,name[-2]]),
A[,,name[-2]]) |
I found a (somewhat) reliable way to assess the size of the problem.
So the returned object goes from 66 kb to 3 GB (with filtering for Peak memory demand of
|
That fits my experience that using --mem=16000 was sufficient. |
Will work on this next week (KW10) |
Keeping the use case of this helper in remind2 reporting in mind, I'd assume it should
|
@0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q could you point me to gdxes with irregularities that could make reportEmi and reportFE crash if I don't restore zeros? |
Nope.
|
hey, yeah, that's a good assumption, I eventually set |
But the use case in |
Thanks Simón. Right now I favour the following approach:
Let me know in case there are objections. If not, then nothing to do on your end, Simón. |
Hi Falk, that sounds good, thank you! |
Looks like this PR should solve the problems: Could solve the problems mostly by introducing new options to readGDX (pik-piam/gdx#8). As of now, I don't see a need for Validation:
|
Leaving this open, as it will probably become relevant in the future again. (see #524) |
The remindmodel/remind#1609 problem comes down to the
Probably some REMIND/GAMS carry-over. |
I also came across the issue of the non-existing SE/FE technologies today, when trying to do reporting for a Remind Base run, which failed due to the vm_demFeSector/vm_demFENonEnergySector mismatch. Falk's merge fortunately fixed that.
as there is no sefe mapping active here (and this equation is only active for Base runs). Maybe it's worth looking into this. |
Thanks Ricarda. |
The error that occured? That was in a |
|
No, you can run |
Well, it was a |
Closing, as this seems to be stable now, not causing any further issues. |
cd /p/projects/remind/modeltests/remind/output; grep oom-kill testOne*/log_output.txt
restore_zeros = FALSE
because we declare all variables over the general supersets (e.g.,tall
,all_enty
), but usually define them only over subsets (t
,entyFE
).restore_zeros = FALSE
reads in only data that is not zero, which is usually what we want. But if some axis of the data (e.g. all values in 2010, or everythingfehos
) happens to be entirely zero (usually that happens for specific gdxes only), that dimension of the resulting magpie object differs (e.g. c(2005, 2015, …) instead of c(2005, 2010, 2015, …)) and it is incompatible with all the rest of the objects we have. What would be needed is a function to expands a variable read in withrestore_zeros = FALSE
to the actual sets it should have. But that is a moving target, as it depends on all the other variables/parameters used in conjunction with it.magpie_expand
is not the solutionSome code to start with:
The question is how to be able to sum, mbind etc.
A
andB
.The problem is that the dimensions each variable is defined (not declared) over in the GAMS code (e.g. t instead of tall, se2fe(entySE,entyFE,te) instead of all all_enty/all_enty/all_te combinations) is not pre-defined, and the problem depends on which other variables the variables is combined with (in magpie operations) and what data those variables have.
The text was updated successfully, but these errors were encountered: