[feature] Lomb-Scargle Periodograms for Multiband Data

[JulienPeloton]

Hello,

I’m amazed by the performance of the fastnifty method for Lomb-Scargle analysis, and I have started to use it everywhere for LSST analysis (with millions of periods to extract)! I was wondering if there are any plans to add this feature for the multiband version?

[lgarrison]

I'm glad you've found the package useful! For multi-band periodograms, I think it would be great to support, although I haven't looked into it before now. Skimming the Astropy docs & source code, it seems like there's a "fast" implementation and a "flexible" one. The fast one seems like it might "just" be a wrapper around single-band Lomb-Scargle (https://github.com/astropy/astropy/blob/main/astropy/timeseries/periodograms/lombscargle_multiband/implementations/mbfast_impl.py). In other words, you might be able to use the Astropy interface with sb_method="fastnifty" and get a speedup.

The code for the flexible method is more complicated! https://github.com/astropy/astropy/blob/main/astropy/timeseries/periodograms/lombscargle_multiband/implementations/mbflex_impl.py . I can't tell immediately if a NUFFT would help there.

Do you know if you need the fast or flexible method? If fast is acceptable, it would be good to know if the fastnifty backend works for you.

Finally, I'll note that an enhancement that would probably help for millions of periodograms is the ability to thread over multiple periodograms. Right now, we only support threading over periodograms with the same observation times, but we should be able to support different observation times, too. Multiprocessing can do this, but probably not as well as threads.

[JulienPeloton]

Thanks for the detailed answer!

In other words, you might be able to use the Astropy interface with sb_method="fastnifty" and get a speedup.

Indeed, good catch. It works out of the box with the default configuration, with a ~5x speed-up compare to astropy implementation.

Do you know if you need the fast or flexible method? If fast is acceptable, it would be good to know if the fastnifty backend works for you.

Fast is acceptable so far, the results I get are completely decent. However playing a bit with the parameters, I found that setting sb_method="fastnifty" does not allow to have the number of frequency terms (common and per-band) higher than 1:

model = LombScargleMultiband(
    time, mag, filters, error, nterms_base=2, nterms_band=1
)

frequency, power = model.autopower(method="fast", sb_method="fastnifty")
---------------------------------------------------------------------------   
ValueError: nterms != 1 only supported with 'chi2' or 'fastchi2' methods

https://github.com/astropy/astropy/blob/bf1103e5061d4a13c928afcaa66865154f5d9afe/astropy/timeseries/periodograms/lombscargle/implementations/main.py#L221-L226

but definitely not a showstopper at this stage of my work (I rather want speed than precision).

Finally, I'll note that an enhancement that would probably help for millions of periodograms is the ability to thread over multiple periodograms. Right now, we only support threading over periodograms with the same observation times, but we should be able to support different observation times, too. Multiprocessing can do this, but probably not as well as threads.

Currently I use naive distributed processing to parallelize the load (Spark) -- so each periodogram is treated independently from the others. But I agree threading over periodograms with different observation times would be great (as probably many steps are re-used internally) :-)

[lgarrison]

Great! I'm glad that's a partial solution, anyway. I'll keep this issue open to remind us about the flexible method. And I'm glad the package is playing nice with Spark.

Even single-band nifty-ls doesn't support nterms > 1 right now. I was looking at that a bit earlier, and it definitely seems like there's an opportunity for a speedup there. The Astropy implementation uses many trig_sum() calls, which finufft replaces: https://github.com/astropy/astropy/blob/bf1103e5061d4a13c928afcaa66865154f5d9afe/astropy/timeseries/periodograms/lombscargle/implementations/fastchi2_impl.py#L104. I'll open an issue to track that.

[JulienPeloton]

thanks! Let me know if you need beta testing, I would be happy to provide feedback :-)