F_ordered memmaps for the summary image calculation

[bashsyr]

caiman uses the giant C_framed file to calculate the summary image (for pnr and corr), I replaced it with the smaller F_framed ones and it gave me much better runtime (especially if you're fetching the file from the network) and it allows for file level increment with reduces runtime even more (went down on my setup from230s on frame_increment= 5 to 73s on the same frame increment)
F_ordered seems to work so much faster especially if you do file increment instead of frame increment.
and it's as easy as loading the F_ordered files with the "cm.load_movie_chain()" function, you can also do file increment instead of frame increment

[EricThomson]

Thanks for pointing this out. I thought I had looked into this and done some profiling! 😱 Will check it out again and try to reproduce what you are showing -- it absolutely makes sense you would get this, and I was surprised when I didn't find something similar. If you show code snippets that might help me reproduce (just to be clear: it's about calculating the correlation/pnr image on F versus C-mapped arrays, right? This is what I thought I had profiled but I may well be wrong, or have made a mistake).

[bashsyr]

Yes, I think what made the biggest difference was File incrementing instead of frame incrementing (for increment 1 there's "only" 100s difference).
Here's my code snip:

video = cm.load_movie_chain(vid_list[::file_increment])
video_for_calc = video
cn_filter, pnr = cm.summary_images.correlation_pnr(video_for_calc[::frame_increment], gSig=gsig, swap_dim=False)

At first I loaded the files separately and then I concadinate them, but I found load movie chain to work just as well
Here's the old approach

    file_increment = 5         # downsamppling parameter
    files_names = glob.glob(path + r'*F_frames*1000.mmap')
    files_names = natsort.natsorted(files_names)
    memmap_list = []    # a list of the individual videos as memmaps
    for index,name in enumerate(files_names):
        Yr, dims, T = cm.load_memmap(name, mode='r')
        memmap_list.append(Yr.T)
    gSig = (5, 5)
    images = np.concatenate(([item for item in memmap_list[::file_increment]]),axis = 0)
    images = images.reshape(len(images),dims[0],dims[1],order='F')
    cn_filter,pnr = cm.summary_images.correlation_pnr(images[::2], gSig=gSig[0], swap_dim=False)

Hope this helps :)

[kushalkolar]

That's the step size, you're subsampling which is why it's faster. Anyways that's the recommended way to get the corr pnr images for very large movies.

[bashsyr]

Yeah, I'm aware of that, but we always subsample (to factor of 5) as we have a rather long movie (40k+) frames.
But even without subsampling, it's still 100s faster in my case.

[EricThomson]

Thanks I will check this out (note I edited your post to format the code to make it easier for me to read I didn't make any other changes). I like the file increment trick! 😄 You said even with file_increment of 1 there is a significant speedup (or at least some speedup), so this is the direct comparison I would be most curious about and will look into.

It sounds like you have 40k frames, and when I profiled I did it with a small demo movie, so that could explain why I didn't see a difference.