manipulating a disk.frame's structure

[Bryanrt-geophys]

Hello ZJ and all disk.frame guru's,

I am working to wrangle the outputs of a large amount of models, tidy them, and finally, visualize them. The functions I have written handle all model outputs and inputs as listed objects, therefore the more models the faster my computations fill up my RAM. I'm excited to learn your tool so I can try to fix my issue's.

In one of my functions I am reading a list of txt file's and turn them into a list of data frames. This was easy enough to change to a list of as.disk.frame() objects. However, downstream I would manipulate those what was once data frames. It doesn'tmy original handling of the data frames translates well to handling disk frames. Here is the first chunk of one of my functions I am running into issue with.

  new_pfl_data <- furrr::future_map2(pfl_data, col_ids, function(pfl, col){
    gc()
    purrr::map2(pfl[-1], col[-1],  ~ .x[c(2, .y)])
  }) %>%
    furrr::future_map(function(x){
      purrr::reduce(x, dplyr::left_join, by = "V2") %>%
        stats::setNames(c("Y", paste0("Z", 2:ncol(.))))
    })

pfl_data: now a list of disk.frames, was a list of tibbles
col_ids: list of numerics

As you can see I first parallel map through the disk.frames and the listed numerics, but I have removed the first column in the dataframe (pfl[-1]) and the first object in the numeric list (col[-1]). Since I changed pfl to a disk.frame, this operation does not seem to work. Also, do I need to swap out the furrr::map variations with disk.frame::map? Can you help me rework this function to flow with disk.frame?

Thanks,
Bryan

[xiaodaigh]

Can you make a simple MWE. Like fill pf_data col_ids.

disk.frame doesn't implement anything like pfl[-1]

[xiaodaigh]

I think you need to list down a simple example of what ur input is in data.frame format and what your expect output in data.frame.

Otherwise, I don't really get what u r trying to doing. I could spend time trying to figure it out but I would rather you present me with all the stuff, so it's as easy as possible to help you.

[Bryanrt-geophys]

Absolutely, sorry It's hard to share a REPREX. Essentially. I have many data frames for each time-step in a singular model. I want to isolate a unique column from each data frame. This is what col_ids is used for. In a prior function the unique columns are determined and populate col_ids. pfl_data contains a list of each data frame for each time-step in a singular model. Essentially, I am taking the first column(a spacial index column) and the unique column from every data frame and then left_join the data by the spacial index column to result in one dat frame per model containing a row with data from each time step. Does this make sense?

[Bryanrt-geophys]

I actually have a sample data set in a loosely formed package on my GitLab. If you pull my project there is an example file that goes through the workflow pointing to the small sample data set and passes it to each function I have made. This may help to see what I am doing. My problem is in the wrangle_pfl function. No worries though If you don't have time to take a look. I am also happy to show you the project if that would be faster.

https://gitlab.com/Bryanrt-geophys/tisc_viewr

[xiaodaigh]

can you make up the a simple data.frame for pf_data and col_ids? And show the desired output from running the above?

YOu can mock something up to illustrate the idea, u don't need the full data.

[xiaodaigh]

I would need you to copy the code here. Running the code will show the desired result. So nothing that requires me to clone a repo and dig around some code base.

[Bryanrt-geophys]

Sure thing. It's pretty late on the evening where I am but I'll get a fully runable reprex posted for you tomorrow morning first thing. Thank you.

…

On Fri, Apr 9, 2021, 23:37 evalparse ***@***.***> wrote: I would need you to copy the code here. Running the code will show the desired result. So nothing that requires me to clone a repo and dig around some code base. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#334 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASLJTRFVBDOMPBKTHSVRF5TTH7P2RANCNFSM42TZTGQA> .

[Bryanrt-geophys]

@xiaodaigh

I haven't learned how to share an RDS file cleanly from git but I think if you click on this link it will download the col_ids rds file for this REPREX and here is the pfl_data rds file.

# insert the quoted path (with the filename) to where you downloaded the rds file into the here() function
col_ids <- readr::read_rds(file = here::here())

# insert the quoted path (with the filename) to where you downloaded the rds file into the here() function
pfl_data <- readr::read_rds(file = here::here())

new_pfl_data <- furrr::future_map2(pfl_data, col_ids, function(pfl, col){
  purrr::map2(pfl[-1], col[-1],  ~ .x[c(2, .y)])
}) %>%
  furrr::future_map(function(x){
    purrr::reduce(x, dplyr::left_join, by = "V2") %>%
      stats::setNames(c("Y", paste0("Z", 2:ncol(.))))
  })

This example is just for one model. So as you can see both col_ids and pfl_data are listed lists. The first layer pertains to what model I am working with. As this REPREX only has data for one model the structure will always be pfl_data[[1]][[n]] or col_ids[[1]][[n]]. Each model has a pfl file per timestep. Each pfl file contains x-coordinates (col 1) y-coordinates(col-2) and z-coordinates for each layer of the ground modeled up to that timestep (col 3:n). Each model I work with can have from 20 - 90 of these files and essentially I am pulling two columns from each file (the y-coordinate column and a unique z-coordinate column dictated by the col_ids variable). However, becuase of the way the models output data, I need to remove the first listed pfl file from pfl and the first row from col as the rows in col relate to the number of files in pfl. This takes us up through the furrr::future_map2(). After this, I left_join() by the y-coordinate column and rename the columns.

Ideally, I would like to save the pfl files upstream with as.disk.frame(). Then I would be passing a list of disk.frame's in pfl_data. Do you think this workflow can be executed using disk.frame's? I'm pretty sure that applying your tool here will bypass the bottleneck in my RAM usage.

[xiaodaigh]

this work flow is too complicated to understand. I don't think reading thru ur description is the best way to understand what's going. So I think u should remove anything specific to ur problem like unique z-coordinate and boil it down to the essential thing you want to achieve without others having to understand your problem. Also don't make ppl download your file.

Creaate your file within the code like this

pfl_data = list(list(data.farme(v1=c(1,2,3))....) etc.

anyway, purrr::map2(pfl[-1], col[-1], ~ .x[c(2, .y)]) you can't pass it a list of disk.frame because of the line I just shown. Each of your dataset you want to subset by some different column where as a disk.fram you need to be applying the same column selection to each chunk of the dataframe.

in general, you are putting too much burden on the other party to try and answer your questions, I feel.

Anyway, disk.frame only support dplyr syntax so this wouldn't work anyway pfl[-1], you need to do something like pfl %>% select(....)

Good luck.

[Bryanrt-geophys]

Sorry I didn't explain it well but I will definitely start working on changing to dplyr syntax and see if I can figure it out from there.

…

On Sat, Apr 10, 2021, 18:29 evalparse ***@***.***> wrote: this work flow is too complicated to understand. I don't think reading thru ur description is the best way to understand what's going. So I'd u should refer anything specific to ur problem like *unique z-coordinate* and boil it down to thing that can be solved without understand any of those details anyway, purrr::map2(pfl[-1], col[-1], ~ .x[c(2, .y)]) you can't pass it a list of disk.frame because of the line I just shown. Each of your dataset you want to subset by some different column where as a disk.fram you need to be applying the same column selection to each chunk of the dataframe. in general, you are putting too much burden on the other party to try and answer your questions, I feel. Anyway, disk.frame only support dplyr syntax so this wouldn't work anyway pfl[-1], you need to do something like pfl %>% select(....) Good luck. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#334 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASLJTRBNPR6BDAI5SQJJ2LDTIDUOHANCNFSM42TZTGQA> .