Gridding/remapping glider data

[dhruvbalwada]

I have found gridding glider data (or any gridding/regridding) to be a huge can of worms.

The package currently has two features:

• 1D gridding (depth/pressure in the vertical). This seems to do a mix of bin averaging+interpolation. If the bin size contains a few data points then they are averaged, and if it doesn't have any then I believe linear interpolation is done (can someone confirm this?)
• 2D gridding. This uses a krigging approach, where the correlation structure and its parameters have to be manually set. I believe this code was translated over from some older MATLAB scripts that have been floating around oceanography since the mid 90s (correct @isgiddy?).

In practice, I never end up using either of these two options. What I currently do is simply use scipy's interpolation routines, and do a linear interpolation onto a uniform grid that I define (the function interp_pres_time in https://github.com/dhruvbalwada/glider-panel-demo/blob/main/data_preprocess_2.ipynb). I know this is crude, but it is simple.
There are a few reasons I don't use GT:
a) I am usually looking to grid in 2D. So instead of using the 1D approach from GT and then using something else from somewhere else doesn't seem useful.
b) I don't like krigging (at least the way it is implemented) because it tends to over smooth the fields.

I would like to make a wish list of things that GT should be able to achieve, and also hear from other people about what they usually do about gridding the glider data. It seems to me that everyone must be doing some gridding for the gliders since they are all writing papers using gridded fields, but no one ever talks about how it was done in their papers (at least from the few I have seen - please put in references if you know). I assume some ad-hoc practices have taken root in different groups, and I don't know if they are consistent or not. All this does matter for the analysis, for example when starting to compute things like gradients/spectra/geostrophic velocities etc.

Wishlist for gridding in GT:

• 1D gridding to depth, density or other tracers, and ability to easily switch back and forth (@jbusecke knows quite a bit about this).
  • Sorting. I know some times people will sort the density field before gridding to get rid of little overturns in the density field.
• 2D gridding using simple options, like scipy's interpolation routines with linear, cubic, pchip etc as options. I think for a lot of users this should be sufficient, and will provide a simple first pass. Also, a lot of these approaches stay more true to the data and don't do too much smoothing.
• 2D gridding using krigging with more options and description of algorithms. Preferably the ability to learn the parameters from the data itself as done in gaussian process regression (e.g. https://github.com/oceanhackweek/ohw20-proj-argo-mapping)

Since, a lot of you have more experience with these things please chime in about expanding or editing this wishlist.

[soerenthomsen]

Thanks Dhruv for raising these points. This will also matter for the uniform OceanGliders format and how to finally save final process/ quality controlled data i.e. as a 1D or 2D field! @vturpin can maybe say a few words about this.

I am not an expert on this but to be able to get back to the initial most close the real observation dataset is key as different scientific questions require different gridding / smoothing etc.

[jbusecke]

For the 1D case I think we can use the machinery I built over at xgcm. As mentioned in there, this might need some tweaking but we want to introduce xgcm to more observationally based data, so I'll have some time to work on it in the near future.

I think however, that enumerating the ways people want to do "annyhring gridding" with the data is das extremly useful before starting this work. Thanks for starting this @dhruvbalwada, and I am very keen to hear what folks over here are doing (and wishing for).

[bastienqueste]

Completely with you here Dhruv... After years of trying different things, and launching deep into optimal interpolations/krigging, I've come back to just binning data with either mean or median depending on data type and issues, and linearly interpolating across empty bins if necessary - either vertically, horizontally or ideally density space depending, once again on the variable and the question at hand.

For krigging/optimal interpolations to be well done, you need a robust model of dimensional variability. It generally works well when viewed from above, but it's very often used in oceanographic time-depth sections... Not only are the scales of variability orders of magnitude different along each dimension (that can be accounted for), but the time/horizontal scales of variability often change very significantly with depth (think size of features in surface mixed layer vs deeper layer) - this can also be accounted for but isn't implemented in any of the commonly used OI scripts floating around in the oceanographic community. Working in time-density can alleviate that problem, but you're left having to then regrid in time-depth to produce figures for papers.

Want to map Argo float data spatially to create climatologies of mixed layer depth: OI and krigging work great.
Want to make a glider section? Grid your data - interpolate if you must. Keep it simple.

[dhruvbalwada]

Good points. I have been digging into solving some of these issues with krigging for glider data, but don't have anything useful to share at the moment. Will keep the group updated if I find something useful.

One of the big problems has been to come up with a measure to compare different gridding methods, which is not simply visual. Do you happen to know of any good references that might help to compare methods?

Can you say a little more about what you mean by "robust model of dimensional variability"? Do you mean the choice of the correlation function, which is usually chosen to be a Gaussian kernel?

[bastienqueste]

Basically yeah, the type and coefficients of your correlation function... Ideally you'd base it on empirical variograms.

(comment moved to correct thread due to lazy phone reply)

[vturpin]

Hi all, To answer Soren's point - first one of the list... :

Thanks Dhruv for raising these points. This will also matter for the uniform OceanGliders format and how to finally save final process/ quality controlled data i.e. as a 1D or 2D field! @vturpin<https://m365.eu.vadesecure.com/safeproxy/v4?f=MkOshBDii9Lk9SODeEwUdYDV_fuY8r2xvPj6twVreBHC7nCex37l8V9PDSMKGtxK&i=caoQxIGzSVqt1jf6HAPfwfnWw4h6rsOIX718dahxUiqOHbz0GlwDE3XC-BTOh-DqKA-XNChK2n-G1MlAUqu2ww&k=PrMK&r=7z-AtxwMB_cldUfAQKARvIjvmiUd549jerPfVdhJ9wjP1fUP685gSQAHW3b2dcTk&s=4cfff5b0f58dee9c1ad91ff0f49d8fbd40250d693d2fc1ca8f0a6b3d8f50adeb&u=https%3A%2F%2Fgithub.com%2Fvturpin> can maybe say a few words about this.

I am not sure I understand exactly the issue, but here are elements I can provide (maybe useful). + OG1.0 format is a trajectory netcdf (1D). Quality controlled ancillary variable ("TEMP_QC" for example) are part of this format. Any gridded data set (2D) are products derived from the OG1.0. + Along track LATITUDE and LONGITUDE are mandatory. LAT and LON should then be interpolated from GPS locations. There is no unique technique for such interpolation. The only requirement is to have the interpolation technique "published" as a good practice under the Ocean Best Practice Repository (OBPS). Basically, the glider Tool documentation should describe the technic used for such interpolation and published on OBPS. Also, the official version of the format will be released 1st or 2d of April. It has been simplified a little bit from the previous version I shared with you. You can find it here : https://drive.google.com/file/d/1Ri0V_xJjVRDaKKin0sBFk_6u1Dpt7Xwa/view?usp=sharing It should not evolve much anymore. Best, V.

…

________________________________ From: Bastien Queste ***@***.***> Sent: Tuesday, March 30, 2021 9:05 AM To: GliderToolsCommunity/GliderTools ***@***.***> Cc: Turpin Victor (OceanOPS) ***@***.***>; Mention ***@***.***> Subject: Re: [GliderToolsCommunity/GliderTools] Gridding/remapping glider data (#67) Basically yeah, the type and coefficients of your correlation function... Ideally you'd base it on empirical variograms. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<https://m365.eu.vadesecure.com/safeproxy/v4?f=h5zl5B-vqwKO9M1Brq_T_GImUkcoxTWAEGsuo61NLbAbQOXaW4VWCAiAcx4kLMOW&i=hl0EjgGTCc_YX-lFWgnfkc3wA_Hm8l1k4Szz5LTFvagjtJtpvtbKuyCkzN4lbYVc4P74ZYD_l5GeI9h3nxzKcQ&k=249w&r=MN2m--7Eiu9tDHR3t9kb5nIzMNFLot-lrukfnWwGiBqgn6HMlq_DM9mPwDKKVxJw&s=44e1980b5c6240a2da5421e01edbc8620851af4997f919d5642e0085ee80e6cc&u=https%3A%2F%2Fgithub.com%2FGliderToolsCommunity%2FGliderTools%2Fdiscussions%2F67%23discussioncomment-546761>, or unsubscribe<https://m365.eu.vadesecure.com/safeproxy/v4?f=NKuk3dmhEtS9FEu68uDfuDZ-bP6Vl5urd97SVMI_dgymnsfbVHB53w2HUwIm4ErE&i=_1IYHo3xSnpcOgkY5UWIc09bs77D_y1S0yWV4_LtNmO8dh7-EV9SJYe15ErZ_h0ARUU5fxbKGjQiBkAefJwLjw&k=pZWT&r=EEN3b4yGQ5SQ5bXX5t8bHMa0kP4ddeJuQ0QADy9WDlO558F5ri0x-cwLVSKE2bEa&s=13dd0eba1c64ad6b599dd0e6fd79d9d6a79551de287bb964bcc4afc5f2ce2d5e&u=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAS7CRM4OKHI2QOEXYB2IKYTTGFZ5HANCNFSM4ZZ4ZY2A>.

________________________________ Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l'intention exclusive de ses destinataires et sont confidentiels. Si vous recevez ce message par erreur ou s'il ne vous est pas destiné, merci de le détruire ainsi que toute copie de votre système et d'en avertir immédiatement l'expéditeur. Toute lecture non autorisée, toute utilisation de ce message qui n'est pas conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite. L'Internet ne permettant pas d'assurer l'intégrité de ce message électronique susceptible d'altération, l’expéditeur (et ses filiales) décline(nt) toute responsabilité au titre de ce message dans l'hypothèse où il aurait été modifié ou falsifié. This message and any attachments (the "message") is intended solely for the intended recipient(s) and is confidential. If you receive this message in error, or are not the intended recipient(s), please delete it and any copies from your systems and immediately notify the sender. Any unauthorized view, use that does not comply with its purpose, dissemination or disclosure, either whole or partial, is prohibited. Since the internet cannot guarantee the integrity of this message which may not be reliable, the sender (and its subsidiaries) shall not be liable for the message if modified or falsified.

[soerenthomsen]

Thank you @vturpin for shoring this info! This is also relevant for the issues #49 and #50 @lukegre @jbusecke @dhruvbalwada