Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update documentation and warnings before 0.1 release #502

Open
wants to merge 32 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
915fd1c
Incremental commit on streamlined doc and user warnings
rhugonnet Apr 8, 2024
2085bdb
Merge remote-tracking branch 'upstream/main' into towards_0.1
rhugonnet Apr 28, 2024
526904c
Incremental commit on documentation
rhugonnet Apr 30, 2024
1968f37
Incremental commit on documentation
rhugonnet Apr 30, 2024
507007f
Incremental commit for doc
rhugonnet May 6, 2024
ea8fb99
Homogenize contributor
rhugonnet May 6, 2024
1260677
Incremental commit on doc
rhugonnet May 7, 2024
dc8469d
Merge remote-tracking branch 'upstream/main' into towards_0.1
rhugonnet May 13, 2024
e230033
Linting
rhugonnet May 13, 2024
fe1c9f0
Incremental commit
rhugonnet May 17, 2024
461d61e
Incremental commit
rhugonnet May 22, 2024
9d657ea
Merge remote-tracking branch 'upstream/main' into towards_0.1
rhugonnet May 23, 2024
5c862c6
Show more toc levels
rhugonnet Jun 14, 2024
84d517e
Merge remote-tracking branch 'upstream/main' into towards_0.1
rhugonnet Jun 15, 2024
22bb334
Add custom css rule for toggle buttons
rhugonnet Jun 16, 2024
1086e0c
Merge remote-tracking branch 'upstream/main' into towards_0.1
rhugonnet Sep 5, 2024
767c440
Incremental commit on doc
rhugonnet Sep 7, 2024
b508171
Incremental commit on doc
rhugonnet Sep 10, 2024
35cf9dd
Add table for coregistration methods
rhugonnet Sep 11, 2024
dfeab70
Merge remote-tracking branch 'upstream/main' into towards_0.1
rhugonnet Sep 11, 2024
254833a
Add timeout to terrain and finalize coreg
rhugonnet Sep 12, 2024
9f9b05d
Incremental commit on doc
rhugonnet Sep 19, 2024
32b6ecf
Incremental commit on doc
rhugonnet Oct 2, 2024
2893605
Finalize new uncertainty page
rhugonnet Oct 5, 2024
71718c0
Remove blockwise coreg example temporarily
rhugonnet Oct 5, 2024
caf2f33
Add pipeline info()
rhugonnet Oct 5, 2024
20b3c4f
Eriks comments
rhugonnet Oct 6, 2024
4633ac9
Merge remote-tracking branch 'upstream/main' into towards_0.1
rhugonnet Oct 6, 2024
0f29bd1
Linting
rhugonnet Oct 6, 2024
7e2dc97
Fixes for tests
rhugonnet Oct 6, 2024
4e4bfd0
Fix directional bias example
rhugonnet Oct 7, 2024
073f494
Incremental commit on doc
rhugonnet Oct 22, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion doc/source/_static/css/custom.css
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ div.cell details.hide > summary {

div.cell details[open].above-input div.cell_input {
border-top: None;
}
}
4 changes: 2 additions & 2 deletions doc/source/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -312,11 +312,11 @@ To build and pass your coregistration pipeline to {func}`~xdem.DEM.coregister_3d
## Uncertainty analysis

```{important}
Several uncertainty functionalities of xDEM are being implemented directly in SciKit-GStat for spatial statistics
Several uncertainty functionalities of xDEM are being implemented directly in SciKit-GStat for spatial statistics
(e.g., fitting a sum of variogram models, pairwise subsampling for grid data). This will allow to simplify several
function inputs and outputs, by relying on a single {func}`~skgstat.Variogram` object.

This will trigger API changes in future package versions.
This will trigger API changes in future package versions.
```

### Core routines for heteroscedasticity, spatial correlations, error propagation
Expand Down
12 changes: 6 additions & 6 deletions doc/source/biascorr.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ Several good practices help performing a successful bias correction:
- **Avoid using "fit" with a subsample size larger than 1,000,000:** Otherwise the optimizer will be extremely slow and might fail with a memory error; consider using "bin_and_fit" instead to reduce the data size before the optimization which still allows to utilize all the data,
- **Avoid using "fit" or "bin_and_fit" for more than 2 dimensions (input variables):** Fitting a parametric form in more than 2 dimensions is quite delicate, consider using "bin" or sequential 1D corrections instead,
- **Use at least 1000 bins for all dimensions, being mindful about dimension number:** Using a small bin size is generally too rough, but a large bin size will grow exponentially with the number of bias variables,
- **Use customized bin edges for data with large extreme values:** Passing simply a bin size will set the min/max of the data as the full binning range, which can be impractical (e.g., most curvatures lies between -2/2 but can take values of 10000+).
- **Use customized bin edges for data with large extreme values:** Passing simply a bin size will set the min/max of the data as the full binning range, which can be impractical (e.g., most curvatures lie between -2/2 but can take values of 10000+).

## Bias-correction methods

Expand Down Expand Up @@ -215,7 +215,7 @@ _ = ax[1].set_yticklabels([])
- **Cons:** Long optimization for a sum of sinusoids.

The default optimizer for directional biases fits a sum of sinusoids using 1 to 3 different frequencies and
keeps the best performing fit, which is useful for periodic along-track errors common to DEMs:
keeps the best performing fit, which is useful for periodic along-track errors common to DEMs.

```{code-cell} ipython3
:tags: [hide-cell]
Expand Down Expand Up @@ -260,7 +260,7 @@ _ = ax[1].set_yticklabels([])
```

For strip-like errors, performing an empirical correction using only a binning with `fit_or_bin="bin"` allows more
flexibility, but requires a larger amount of static surfaces:
flexibility, but requires a larger amount of static surfaces.

```{code-cell} ipython3
:tags: [hide-cell]
Expand Down Expand Up @@ -405,9 +405,9 @@ aspect, slope = ref_dem.get_terrain_attribute(["aspect", "slope"])

# Pass the variables to the fit_and_apply function matching the names declared above
corrected_dem = biascorr.fit_and_apply(
ref_dem,
tba_dem_nk,
inlier_mask=inlier_mask,
ref_dem,
tba_dem_nk,
inlier_mask=inlier_mask,
bias_vars={"aspect": aspect, "slope": slope, "elevation": ref_dem}
)
```
Expand Down
56 changes: 43 additions & 13 deletions doc/source/cheatsheet.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,15 +16,15 @@ kernelspec:

# Cheatsheet: How to correct... ?

In elevation data analysis, the problem generally starts with identifying what correction method to apply when
In elevation data analysis, the problem generally starts with identifying what correction method to apply when
observing a specific pattern of error in your own data.

Below, we summarize a cheatsheet that links what method is likely to correct a pattern of error you can visually
Below, we summarize a cheatsheet that links what method is likely to correct a pattern of error you can visually
identify on **a map of elevation differences with another elevation dataset (looking at static surfaces)**!

## Cheatsheet

The patterns of errors categories listed in this spreadsheet **are linked to visual examples further below** that
The patterns of errors categories listed in this spreadsheet **are linked to visual examples further below** that
you use to compare to your own elevation differences.

```{list-table}
Expand All @@ -37,7 +37,7 @@ you use to compare to your own elevation differences.
- Cause and correction
- Notes
* - {ref}`sharp-landforms`
- Positive and negative errors that are larger near high slopes and symmetric with opposite slope orientation, making landforms appear visually.
- Positive and negative errors that are larger near high slopes and symmetric with opposite slope orientation, making landforms appear visually.
- Likely horizontal shift due to geopositioning errors, use a {ref}`coregistration` such as {class}`~xdem.coreg.NuthKaab`.
- Even a tiny horizontal misalignment can be visually identified! To not confuse with {ref}`peak-cavity`.
* - {ref}`smooth-large-field`
Expand Down Expand Up @@ -74,8 +74,8 @@ pyplot.rcParams['savefig.dpi'] = 600
pyplot.rcParams['font.size'] = 9 # Default 10 is a bit too big for coregistration plots
```

It is often crucial to relate the location of your errors on static surfaces to the terrain distribution
(in particular, its slope and curvature), which can usually be infered visually from a hillshade.
It is often crucial to relate the location of your errors on static surfaces to the terrain distribution
(in particular, its slope and curvature), which can usually be inferred visually from a hillshade.


```{code-cell} ipython3
Expand All @@ -102,10 +102,15 @@ hs.plot(cmap="Greys_r", cbar_title="Hillshade")
(sharp-landforms)=
### Sharp landforms

Example of sharp landforms appearing with a horizontal shift due to geolocation errors. We here translate the DEM
Example of sharp landforms appearing with a horizontal shift due to geolocation errors. We here translate the DEM
horizontally by 1/10th of a pixel, for a pixel resolution of 20 m.

```{code-cell} ipython3
:tags: [hide-input]
:mystnb:
: code_prompt_show: "Show code to simulate horizontal shift errors"
: code_prompt_hide: "Hide code to simulate horizontal shift errors"

# Simulate a translation of 1/10th of a pixel
x_shift = 0.1
y_shift = 0.1
Expand All @@ -119,10 +124,15 @@ dh.plot(cmap='RdYlBu', vmin=-3, vmax=3, cbar_title="Elevation differences of\nho
(smooth-large-field)=
### Smooth-field offset

Example of smooth large offset field created by a wrong vertical CRS. We here show the difference due to the EGM96
geoid added on top of the ellipsoid.
Example of smooth large offset field created by a wrong vertical CRS. We here show the difference due to the EGM96
geoid added on top of the ellipsoid.

```{code-cell} ipython3
:tags: [hide-input]
:mystnb:
: code_prompt_show: "Show code to simulate vertical referencing errors"
: code_prompt_hide: "Hide code to simulate vertical referencing errors"

# Set current vertical CRS with a geoid
dem.set_vcrs("EGM96")
# Remove the geoid
Expand All @@ -139,6 +149,11 @@ dh.plot(cmap='RdYlBu', cbar_title="Elevation differences of\nvertical transform
Example of ramp created by a rotation due to camera errors. We here show just a slight rotation of 0.02 degrees.

```{code-cell} ipython3
:tags: [hide-input]
:mystnb:
: code_prompt_show: "Show code to simulate rotation errors"
: code_prompt_hide: "Hide code to simulate rotation errors"

# Apply a rotation of 0.02 degrees
rotation = np.deg2rad(0.02)
# Affine matrix for 3D transformation
Expand Down Expand Up @@ -167,6 +182,11 @@ dh.plot(cmap='RdYlBu', cbar_title="Elevation differences of\nrotation (m)")
Example of undulations resembling jitter errors. We here artificially create a sinusoidal signal at a certain angle.

```{code-cell} ipython3
:tags: [hide-input]
:mystnb:
: code_prompt_show: "Show code to simulate undulation errors"
: code_prompt_hide: "Hide code to simulate undulation errors"

# Get rotated coordinates along an angle
angle = -20
xx = gu.raster.get_xy_rotated(dem, along_track_angle=angle)[0]
Expand All @@ -186,11 +206,16 @@ synthetic_bias.plot(cmap='RdYlBu', vmin=-3, vmax=3, cbar_title="Elevation differ
(peak-cavity)=
### Peak cuts and cavity fills

Example of peak cutting and cavity filling errors. We here downsampled our DEM from 20 m to 100 m to simulate a lower
native resolution, then upsample it again to 20 m, to show the errors affect areas near high curvatures such as
Example of peak cutting and cavity filling errors. We here downsampled our DEM from 20 m to 100 m to simulate a lower
native resolution, then upsample it again to 20 m, to show the errors affect areas near high curvatures such as
peaks and cavities.

```{code-cell} ipython3
:tags: [hide-input]
:mystnb:
: code_prompt_show: "Show code to simulate resolution errors"
: code_prompt_hide: "Hide code to simulate resolution errors"

# Downsample DEM (bilinear)
dem_100m = dem.reproject(res=100)

Expand All @@ -202,16 +227,21 @@ dh.plot(cmap='RdYlBu', vmin=-40, vmax=40, cbar_title="Elevation differences of\n
(point-oscillation)=
### Point oscillating

An example of oscillating point errors created by wrong point-raster comparison by rasterization of the points,
An example of oscillating point errors created by wrong point-raster comparison by rasterization of the points,
which are especially large around steep slopes.

```{code-cell} ipython3
:tags: [hide-input]
:mystnb:
: code_prompt_show: "Show code to simulate point-raster comparison errors"
: code_prompt_hide: "Hide code to simulate point-raster comparison errors"

# Simulate swath coordinates of an elevation point cloud
x = np.linspace(dem.bounds.left, dem.bounds.right, 100)
y = np.linspace(dem.bounds.top - 5000, dem.bounds.bottom + 5000, 100)

# Interpolate DEM at these coordinates to build the point cloud
# (to approximate the real elevation at these coordinates,
# (to approximate the real elevation at these coordinates,
# which has negligible impact compared to rasterization)
z = dem.interp_points((x,y))
epc = gu.Vector(gpd.GeoDataFrame(geometry=gpd.points_from_xy(x=x, y=y, crs=dem.crs), data={"z": z}))
Expand Down
2 changes: 1 addition & 1 deletion doc/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -191,4 +191,4 @@ def setup(app):

html_css_files = [
"css/custom.css",
]
]
36 changes: 18 additions & 18 deletions doc/source/coregistration.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ kernelspec:
xDEM implements a wide range of **coregistration algorithms and pipelines for 3-dimensional alignment** from the
peer-reviewed literature often tailored specifically to elevation data, aiming at correcting systematic elevation errors.

Two categories of alignment are generally differentiated: 3D [affine transformations](https://en.wikipedia.org/wiki/Affine_transformation)
Two categories of alignment are generally differentiated: 3D [affine transformations](https://en.wikipedia.org/wiki/Affine_transformation)
described below, and other alignments that possibly rely on external variables, described in {ref}`biascorr`.

Affine transformations can include vertical and horizontal translations, rotations and reflections, and scalings.
Expand Down Expand Up @@ -74,9 +74,9 @@ tba_dem = xdem.DEM(xdem.examples.get_path("longyearbyen_tba_dem"))
aligned_dem = tba_dem.coregister_3d(ref_dem, my_coreg_pipeline)
```

Alternatively, the coregistration can be applied by calling {func}`~xdem.coreg.Coreg.fit_and_apply`, or sequentially
Alternatively, the coregistration can be applied by calling {func}`~xdem.coreg.Coreg.fit_and_apply`, or sequentially
calling the {func}`~xdem.coreg.Coreg.fit` and {func}`~xdem.coreg.Coreg.apply` steps,
which allows a broader variety of arguments at each step, and re-using the same transformation to several objects
which allows a broader variety of arguments at each step, and re-using the same transformation to several objects
(e.g., horizontal shift of both a stereo DEM and its ortho-image).

```{code-cell} ipython3
Expand Down Expand Up @@ -105,10 +105,10 @@ Often, an `inlier_mask` has to be passed to {func}`~xdem.coreg.Coreg.fit` to iso
- [Nuth and Kääb (2011)](https://doi.org/10.5194/tc-5-271-2011)
* - {ref}`icp`
- Translation and rotations
- [Besl and McKay (1992)](https://doi.org/10.1117/12.57955)
- [Besl and McKay (1992)](https://doi.org/10.1117/12.57955)
* - {ref}`vshift`
- Vertical translation
-
-
```

## Using a coregistration
Expand Down Expand Up @@ -141,10 +141,10 @@ See {ref}`biascorr` for more information on non-rigid transformations ("bias cor

### Accessing coregistration metadata

The metadata surrounding a coregistration method, which can be displayed by {func}`~xdem.coreg.Coreg.info`, is stored in
The metadata surrounding a coregistration method, which can be displayed by {func}`~xdem.coreg.Coreg.info`, is stored in
the {attr}`~xdem.coreg.Coreg.meta` nested dictionary.
This metadata is divided into **inputs** and **outputs**. Input metadata corresponds to what arguments are
used when initializing a {class}`~xdem.coreg.Coreg` object, while output metadata are created during the call to
This metadata is divided into **inputs** and **outputs**. Input metadata corresponds to what arguments are
used when initializing a {class}`~xdem.coreg.Coreg` object, while output metadata are created during the call to
{func}`~xdem.coreg.Coreg.fit`. Together, they allow to apply the transformation during the
{func}`~xdem.coreg.Coreg.apply` step of the coregistration.

Expand All @@ -157,13 +157,13 @@ For both **inputs** and **outputs**, four consistent categories of metadata are

**Note:** Some metadata, such as the parameters `fit_or_bin` and `fit_func` described below, are pre-defined for affine coregistration methods and cannot be modified. They only take user-specified value for {ref}`biascorr`.

**1. Randomization metadata (common to all)**:
**1. Randomization metadata (common to all)**:

- An input `subsample` to define the subsample size of valid data to use in all methods (recommended for performance),
- An input `random_state` to define the random seed for reproducibility of the subsampling (and potentially other random aspects such as optimizers),
- An output `subsample_final` that stores the final subsample size used, which can be smaller than requested depending on the amount of valid data intersecting the two elevation datasets.
- An input `subsample` to define the subsample size of valid data to use in all methods (recommended for performance),
- An input `random_state` to define the random seed for reproducibility of the subsampling (and potentially other random aspects such as optimizers),
- An output `subsample_final` that stores the final subsample size used, which can be smaller than requested depending on the amount of valid data intersecting the two elevation datasets.

**2. Fitting and binning metadata (common to nearly all methods)**:
**2. Fitting and binning metadata (common to nearly all methods)**:

- An input `fit_or_bin` to either fit a parametric model by passing **"fit"**, perform an empirical binning by passing **"bin"**, or to fit a parametric model to the binning with **"bin_and_fit" (only "fit" or "bin_and_fit" possible for affine methods)**,
- An input `fit_func` to pass any parametric function to fit to the bias **(pre-defined for affine methods)**,
Expand All @@ -175,7 +175,7 @@ For both **inputs** and **outputs**, four consistent categories of metadata are
- An output `fit_perr` that stores the error of optimized parameters (only for default `fit_optimizer`),
- An output `bin_dataframe` that stores the dataframe of binned statistics.

**3. Iteration metadata (common to all iterative methods)**:
**3. Iteration metadata (common to all iterative methods)**:

- An input `max_iterations` to define the maximum number of iterations at which to stop the method,
- An input `tolerance` to define the tolerance at which to stop iterations (tolerance unit defined in method description),
Expand All @@ -189,7 +189,7 @@ For both **inputs** and **outputs**, four consistent categories of metadata are
- Outputs `shift_x`, `shift_y` and `shift_z` that store the easting, northing and vertical offsets, respectively.

```{tip}
In xDEM, you can extract the translations and rotations of an affine matrix using {class}`xdem.coreg.AffineCoreg.to_translations` and
In xDEM, you can extract the translations and rotations of an affine matrix using {class}`xdem.coreg.AffineCoreg.to_translations` and
{class}`xdem.coreg.AffineCoreg.to_rotations`.

To further manipulate affine matrices, see the [documentation of pytransform3d](https://dfki-ric.github.io/pytransform3d/rotations.html).
Expand All @@ -199,7 +199,7 @@ To further manipulate affine matrices, see the [documentation of pytransform3d](

These metadata are only inputs specific to a given method, outlined in the method description.

For instance, for {class}`xdem.coreg.Deramp`, an input `poly_order` to define the polynomial order used for the fit, and
For instance, for {class}`xdem.coreg.Deramp`, an input `poly_order` to define the polynomial order used for the fit, and
for {class}`xdem.coreg.DirectionalBias`, an input `angle` to define the angle at which to do the directional correction.

## Coregistration methods
Expand Down Expand Up @@ -376,7 +376,7 @@ _ = ax[1].set_yticklabels([])

### The {class}`~xdem.coreg.CoregPipeline` object

Often, more than one coregistration approach is necessary to obtain the best results, and several need to be combined
Often, more than one coregistration approach is necessary to obtain the best results, and several need to be combined
sequentially. A {class}`~xdem.coreg.CoregPipeline` can be constructed for this:

```{code-cell} ipython3
Expand Down Expand Up @@ -434,4 +434,4 @@ Additionally, ICP tends to fail with large initial vertical differences, so a pr

```{code-cell} ipython3
pipeline = xdem.coreg.VerticalShift() + xdem.coreg.ICP() + xdem.coreg.NuthKaab()
```
```
10 changes: 5 additions & 5 deletions doc/source/ecosystem.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,11 +2,11 @@

# Ecosystem

rhugonnet marked this conversation as resolved.
Show resolved Hide resolved
xDEM is but a single tool among a large landscape of open tools for geospatial elevation analysis! Below is a list of
xDEM is but a single tool among a large landscape of open tools for geospatial elevation analysis! Below is a list of
other **tools that you might find useful to combine with xDEM**, in particular for retrieving elevation data or to perform complementary analysis.
rhugonnet marked this conversation as resolved.
Show resolved Hide resolved

```{seealso}
Tools listed below only relate to elevation data. To analyze georeferenced rasters, vectors and point cloud data,
Tools listed below only relate to elevation data. To analyze georeferenced rasters, vectors and point cloud data,
check out **xDEM's sister-package [GeoUtils](https://geoutils.readthedocs.io/)**.
```
## Python
Expand All @@ -24,10 +24,10 @@ Complementary Python tools to **analyze elevation data** are for instance:

## Julia

If you are working in Julia, the [Geomorphometry](https://github.com/Deltares/Geomorphometry.jl) package provides a
If you are working in Julia, the [Geomorphometry](https://github.com/Deltares/Geomorphometry.jl) package provides a
wide range of terrain analysis for elevation data.

## Other community resources

Whether to retrieve data among their wide range of open datasets, or to dive into their other resources, be sure to check out the
amazing [OpenTopography](https://opentopography.org/) and [OpenAltimetry](https://openaltimetry.earthdatacloud.nasa.gov/data/) efforts!
Whether to retrieve data among their wide range of open datasets, or to dive into their other resources, be sure to check out the
amazing [OpenTopography](https://opentopography.org/) and [OpenAltimetry](https://openaltimetry.earthdatacloud.nasa.gov/data/) efforts!
Loading
Loading