Add cellarea docs, allow Literate.jl tutorials in the doc pipeline, fix typos #800

asinghvi17 · 2024-10-13T13:27:07Z

Switch all references for reproject from GDAL to Proj, improve docstring
Add Proj to the docs project
Slightly clarify some doc examples
Add Literate.jl example pipeline
Add detailed doc page and exploration of cellarea

amend reference to GDAL

asinghvi17 · 2024-10-13T13:27:18Z

Supersedes #789

rafaqz

My main thought here is why doesn't the cellarea tutorial just go in the cellarea API docs? (With a link from toolbars)

Or in extended docs? This approach feels like a lot of text needs to be repeated and the overall complexity of the documentation is increased.

To me tutorials should be more about combining multiple methods into workflows.

(Or maybe when one method just has so many options to explain it needs a lot of examples, rasterization may fit this case)

rafaqz · 2024-10-13T13:38:44Z

src/methods/reproject.jl

@@ -2,15 +2,16 @@
 """
    reproject(obj; crs)

-Reproject the lookups of `obj` to a different crs. 
+Reproject the lookups (axes) of `obj` to a different crs. 


We try to maintain the distinction between axes and lookups as different things

I see what you're saying, the problem is that even people who've used rasters for a while have no clue what "lookups" are. There has to be some frame of reference. "Dimension values" is also super unclear - what does that mean?

Maybe we can rewrite the parentheses to be more clear? Would "axis values" work there?

FWIW, this is meant to be for people who have no clue what cellarea is or why you might want it. So you have to go from the ground up, more or less.

I get it, but we just can't use a common base function name that also works on a Raster to explain a DD function that's really a different thing.

It's clearly difficult to find words that describe lookups without semantic overloads (see AxisKeys.jl... keys are a different base method again) but it's important that we try.

I think axis values is helpful, but maybe values is also a pretty empty word. Maybe this needs to be systematic from DD up, we could workshop the language over there.

(FWIW lookup was intentionally chosen to avoid the overloads with base concepts that other packages have. The problem with avoiding overloads is you end up with a less common word)

Maybe we could say "x and y values" for now?

Then we're back to the problem of "what is a lookup" again?

Maybe we could say "location information". But I would prefer to just use lookups here and have a glossary where we would explain these differences, because otherwise we would have to duplicate this info everywhere we use lookups.

I would argue that we should duplicate this information everywhere we use lookups, at least in top-level, user accessible functions. I don't want to overestimate the curiosity of a new user v/s their unwillingness to click a link.

That's not to say that we shouldn't have a glossary with a more detailed explanation, but there should be something like:

[lookup values](link to DD lookup docs) (the positions of the cells in each dimension)

or something

Actually "lookup values (indicating the positions of each cell)" sounds pretty descriptive

lazarusA · 2024-10-13T13:44:24Z

your cellarea tutorial is not running the lines as intended (no output in most). Plus, is hard to find, I needed to look for it in the search box, maybe add a new section to the nav bar?

lazarusA · 2024-10-13T13:45:40Z

In yax, we have something like, although is not very much maintained 🫨 .

lazarusA · 2024-10-13T13:51:01Z

@rafaqz IMHO, API docs should not have extended docs, and be as clean as possible to the main description of input and output arguments. For developers it might be easier to see things there if they work as intended, another test, but documenting should have a low entry level, specially for people (me) that don't know how to use those functions 😄 .

rafaqz · 2024-10-13T13:53:51Z

Sure, I just meant using extended docs so you can add more examples for the online docs that don't swamp the repl. Not so you would actually look for them in the repl.

My main point is not to have cellarea docs twice, I don't want too much duplicate text to maintain.

asinghvi17 · 2024-10-13T14:04:08Z

Even extended docs are a little button you can click on online, I don't see an alternative to tutorials unfortunately. These also serve very different uses since the tutorials are to introduce someone to a function and what you can do with it, REPL docs are to remind you of what the function itself does and how to call it

rafaqz · 2024-10-13T16:51:39Z

rasterio also just uses the api docs for functions like this
https://rasterio.readthedocs.io/en/stable/cli.html#rasterize

And their tutorials are reserved for general things (though I'm not saying their docs are great)

From my perspective, contributors here and for DD have come and gone, and in some years there is a good chance I will have to maintain these docs. Minimizing that task across what are now very many packages is becoming central to maintaining my sanity.

If you make tutorials for some Rasters.jl methods, at some point we are committing to making and maintaining them for all methods.

I really want to try putting these tutorials in the regular API docs, and changing that when they are really too big and we are committed to fleshing out the docs for every other method here.

(possibly some more tutorial style stuff can go in the geocompx book too?)

rafaqz · 2024-10-13T17:12:27Z

Currently we have methods listed here:
https://rafaqz.github.io/Rasters.jl/stable/methods

They could be better organised in a menu somehow. But to me just improving those is better than adding additional versions of them all. Like cellarea isn't even listed there yet.

asinghvi17 · 2024-10-13T17:32:02Z

My issue with putting these in docstrings is twofold:

They don't run, so they can't be used as additional sanity checks. Output may change over time and the REPL examples in the docstrings would stay static. This is another source of maintenance burden.
The kind of examples I want to show (which are, fundamentally, full-workflow examples) are unsuitable to REPL examples because they aren't visual enough. In the REPL, I have no way to show a raster's values without completely overwhelming the user with information. A plot, on the other hand, is simple and immediately understandable.

Looking at the rasterio docs you sent - when will I ever actually run those commands? If I run them, how do I see the output? Those docs aren't the best, and a user will get disheartened seeing that they have to do things. For that matter, when will a user ever run the example commands in the REPL docs, if they don't know they are there? This is the missing source of information I want to correct here. People don't fundamentally care unless things are immediately in front of them, so they can see what it does and see the benefit.

That being said, I can see your point about implicit promises and maintaining stuff. How about this:

Move the cellarea tutorial to a new "Estimating spatial means" tutorial (and keeping the tutorial structure)
Move some of the explanatory text here to the cellarea docs

asinghvi17 · 2024-10-13T17:33:22Z

If the goal of the geocompx books is to replicate the R and Python versions directly then I'm not sure if we can sneak these examples in there, though there will definitely be more examples of these kinds of functions.

rafaqz · 2024-10-13T18:13:32Z

The Rasters API docs already have examples with plots! And they do actually run, as doctests :)

I like the split - estimating spatial means is a complete workflow task, perfect for a tutorial

It can link to the cellarea API docs for method details

rafaqz · 2024-10-13T18:16:27Z

I actually think all of the current API docs can be better and longer with more explanation and examples, I just haven't had time

lazarusA · 2024-10-13T19:11:00Z

docs/src/tutorials/methods/cellarea.jl

+
+    But fundamentally, this is all that `zonal` is doing under the hood - 
+    masking and cropping the raster to the geometry, and then computing the statistic.
+=#


from experience (a painful one), I can tell you that for long explanations using literate and .jl files is not ideal. I did that for a while in the yax docs, and eventually I went back to just markdown. For small scripts that just run is fine, like in beautifulmakie, but once you start doing bigger things I would say that markdown should be the way to go.

Hmm, why not? It's worked pretty well for me so far, but those were all in situations where my markdown content doesn't exceed ~150 lines. Curious what your issues were...

I do at some point want to integrate a Quarto like system into most of my docs, since that seems to be the best at this stuff.

I agree with @lazarusA, Literate.jl adds a layer of complexity that's just not worth it in some cases.

I would prefer not to have it here

felixcremer · 2024-10-13T23:54:06Z

docs/src/tutorials/methods/cellarea.jl

+# To construct a raster, we'll need to specify the x and y dimensions.  These are called "lookups" in Rasters.jl.
+using Rasters.Lookups
+# We can now construct the x and y lookups.  Here we'll use a start-at-one, step-by-five grid.
+# Note that we're specifying that the "sampling", i.e., what the coordinates actually mean, 


Suggested change

# Note that we're specifying that the "sampling", i.e., what the coordinates actually mean,

# Note that we're specifying the "sampling", i.e., what the coordinates actually mean,

I might take this section out, not sure yet.

felixcremer · 2024-10-13T23:57:16Z

docs/src/tutorials/methods/cellarea.jl

+# Now, let's compute the average precipitation per square meter across Chile.
+# First, we need to get the area of each cell in square meters.  We'll use the spherical method, since we're working with a geographic coordinate system.  This is the default.
+areas = cellarea(masked_precip)
+masked_areas = mask(areas; with = chile)


Why do we have to mask again?

#797, but this should be fixed once the cf branch is merged.

asinghvi17 · 2024-10-15T16:30:35Z

@rafaqz now I remember why I was against jldoctest blocks - they don't show up in the REPL at all! So they're fundamentally useless for in-REPL documentation. Unless there's some module level switch to change this (but I don't think there is...)

asinghvi17 · 2024-10-15T16:32:32Z

Ah I guess we're missing the julia> prompt (that's a super weird thing to require IMO)

KristofferC/OhMyREPL.jl#293 (comment)

rafaqz · 2024-10-15T16:40:42Z

Yeah for me e.g. rasterize has a full code example

asinghvi17 added 8 commits October 10, 2024 17:41

Explain what reproject does a bit more

d63518a

amend reference to GDAL

Add Proj.jl to docs project + simplify make.jl

3990a9c

Minor restructuring to GBIF workflow example to make it clearer

f06a98c

Add a header to array_operations.md

ef18b27

Merge remote-tracking branch 'origin/main' into moredocs

0c68d21

Add a brief tutorial for cellarea, with descriptions

c795d2c

Add a motivating example to cellarea

9e1e86c

Add a little note that this is exactly what zonal does

2b6f33b

asinghvi17 mentioned this pull request Oct 13, 2024

[WIP] More docs #789

Closed

5 tasks

asinghvi17 requested review from rafaqz and lazarusA and removed request for rafaqz October 13, 2024 13:32

rafaqz reviewed Oct 13, 2024

View reviewed changes

asinghvi17 added 3 commits October 13, 2024 07:04

Clean up the docs a bit

b57b633

Add tutorials to top bar

2aa2186

add NaturalEarth to docs project

06ddbc1

lazarusA reviewed Oct 13, 2024

View reviewed changes

felixcremer reviewed Oct 14, 2024

View reviewed changes

asinghvi17 added 5 commits October 13, 2024 18:37

remove Literate, make the tutorial a full-workflow example

2390aba

Fix title + plotting

1f844bd

Add a bit more text about what we're doing with cellarea

e7e3423

Add the examples from the old cellarea tutorial to the docstring

5c6c761

Remove tutorials/methods/cellarea.jl

bbf539a

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cellarea docs, allow Literate.jl tutorials in the doc pipeline, fix typos #800

Add cellarea docs, allow Literate.jl tutorials in the doc pipeline, fix typos #800

asinghvi17 commented Oct 13, 2024 •

edited

Loading

asinghvi17 commented Oct 13, 2024

rafaqz left a comment •

edited

Loading

rafaqz Oct 13, 2024

asinghvi17 Oct 13, 2024

asinghvi17 Oct 13, 2024

rafaqz Oct 13, 2024 •

edited

Loading

asinghvi17 Oct 13, 2024

asinghvi17 Oct 14, 2024

felixcremer Oct 14, 2024

asinghvi17 Oct 14, 2024

asinghvi17 Oct 14, 2024

asinghvi17 Oct 14, 2024

lazarusA commented Oct 13, 2024

lazarusA commented Oct 13, 2024

lazarusA commented Oct 13, 2024

rafaqz commented Oct 13, 2024 •

edited

Loading

asinghvi17 commented Oct 13, 2024 •

edited

Loading

rafaqz commented Oct 13, 2024 •

edited

Loading

rafaqz commented Oct 13, 2024

asinghvi17 commented Oct 13, 2024 •

edited

Loading

asinghvi17 commented Oct 13, 2024

rafaqz commented Oct 13, 2024 •

edited

Loading

rafaqz commented Oct 13, 2024

lazarusA Oct 13, 2024

asinghvi17 Oct 13, 2024

asinghvi17 Oct 13, 2024 •

edited

Loading

rafaqz Oct 13, 2024

felixcremer Oct 13, 2024

asinghvi17 Oct 14, 2024

felixcremer Oct 13, 2024

asinghvi17 Oct 14, 2024 •

edited

Loading

asinghvi17 commented Oct 15, 2024 •

edited

Loading

asinghvi17 commented Oct 15, 2024

rafaqz commented Oct 15, 2024

	# Note that we're specifying that the "sampling", i.e., what the coordinates actually mean,
	# Note that we're specifying the "sampling", i.e., what the coordinates actually mean,

Add cellarea docs, allow Literate.jl tutorials in the doc pipeline, fix typos #800

Are you sure you want to change the base?

Add cellarea docs, allow Literate.jl tutorials in the doc pipeline, fix typos #800

Conversation

asinghvi17 commented Oct 13, 2024 • edited Loading

asinghvi17 commented Oct 13, 2024

rafaqz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rafaqz Oct 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lazarusA commented Oct 13, 2024

lazarusA commented Oct 13, 2024

lazarusA commented Oct 13, 2024

rafaqz commented Oct 13, 2024 • edited Loading

asinghvi17 commented Oct 13, 2024 • edited Loading

rafaqz commented Oct 13, 2024 • edited Loading

rafaqz commented Oct 13, 2024

asinghvi17 commented Oct 13, 2024 • edited Loading

asinghvi17 commented Oct 13, 2024

rafaqz commented Oct 13, 2024 • edited Loading

rafaqz commented Oct 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asinghvi17 Oct 13, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

asinghvi17 Oct 14, 2024 • edited Loading

Choose a reason for hiding this comment

asinghvi17 commented Oct 15, 2024 • edited Loading

asinghvi17 commented Oct 15, 2024

rafaqz commented Oct 15, 2024

asinghvi17 commented Oct 13, 2024 •

edited

Loading

rafaqz left a comment •

edited

Loading

rafaqz Oct 13, 2024 •

edited

Loading

rafaqz commented Oct 13, 2024 •

edited

Loading

asinghvi17 commented Oct 13, 2024 •

edited

Loading

rafaqz commented Oct 13, 2024 •

edited

Loading

asinghvi17 commented Oct 13, 2024 •

edited

Loading

rafaqz commented Oct 13, 2024 •

edited

Loading

asinghvi17 Oct 13, 2024 •

edited

Loading

asinghvi17 Oct 14, 2024 •

edited

Loading

asinghvi17 commented Oct 15, 2024 •

edited

Loading