Modifiers - Private static STAC JSON files in private Azure Blob storage

[BnJam]

I am trying to get the pystac_client to work with a set of static JSON files sitting in a private Azure Blob storage Container. I realize I must implement a modifier to alter the HREF's in-place with a SAS token, however, I am struggling to get this to work properly.

Am I not seeing the documentation for writing modifiers? Is there a generic signing modifier that will take my SAS token and sign the relevant HREFs for me? All my STAC files pass validation checks and all the Catalog is ABSOLUTE_PUBLISHED.

If having a set of generic modifiers built-in to pystac-client is out of scope, would it be possible to have better documentation for implementing modifiers?

[gadomski]

The signature of modifier is Optional[Callable[[Modifiable], None]], where Modifiable = Union[pystac.Collection, pystac.Item, pystac.ItemCollection, dict]. For an advanced example of defining a modifer function, see sign_inplace in https://github.com/microsoft/planetary-computer-sdk-for-python/blob/7970d3bda89f4dbd052516cd1df52b00b658d855/planetary_computer/sas.py. A simpler example (untested) might look like:

def modifier(modifiable: Modifiable) -> None:
    if isinstance(modifiable, Catalog):
        for child in modifiable.get_children():
            child.set_self_href(_sign(child.get_self_href()))

def _sign(href: str) -> str:
    ...

Does that help?

[BnJam]

Thanks @gadomski ! That does help a lot :)

I did read through the planetarycopmuter.sign and it helped, however, I do believe we could use some better and more direct documentation on this topic.

I had ended up implementing a modified version of this example.
I used this modifier in the request_modifier call of pystac_client.Client.open.
Simplified version:

def sign_request(request: requests.Request) -> requests.Request:
    def append_tok(href):
        return f'{href}?{sas}'
    request.url = append_tok(request.url)
    return request

However, that was a bandaid to traverse the Catalog and children - still had some issues pulling the Assets, but I will give your (untested) approach a try!

[TomAugspurger]

Is there any interest in an os.walk style on STAC catalogs? I'm having trouble deciding if that's what would be helpful or not.

[gadomski]

Something different than https://pystac.readthedocs.io/en/stable/api/pystac.html?highlight=walk#pystac.Catalog.walk?

[TomAugspurger]

TIL. Neat.

…

On Mon, Feb 13, 2023 at 8:52 AM Pete Gadomski ***@***.***> wrote: Something different than https://pystac.readthedocs.io/en/stable/api/pystac.html?highlight=walk#pystac.Catalog.walk ? — Reply to this email directly, view it on GitHub <#421 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAKAOISBOWZ2DCOHAHZI27LWXJDCJANCNFSM6AAAAAAUPW4KBE> . You are receiving this because you commented.Message ID: ***@***.***>

[gadomski]

Moving this out of the v0.7 milestone, as there may be more auth work/docs coming from the PC side of things in the not-to-distant future.

[BnJam]

This is nice! I have come across more use cases in my projects and internal work for using static catalogs over served ones.
Is there interest in having a StaticClient to perform ItemSearch on? Understanding that it will not have the same
performance as an API Client, but, it's a place to start. Would using the Catalog.walk() to assess timestamps and geo bounds to return Items from something like StaticClient.search be a possibility?

Something different than https://pystac.readthedocs.io/en/stable/api/pystac.html?highlight=walk#pystac.Catalog.walk?

Naturally, this is just a suggestion, I look to your knowledge on the best approach for expanding access to static catalogs.

[jsignell]

Is there interest in having a StaticClient to perform ItemSearch on?

This idea of a static catalog search has been discussed a few times #425, #66 and always deemed out of scope. I did write up a little proof of concept (https://github.com/jsignell/stac-static) of search that implements the search API on top of an item collection loaded locally into a geopandas dataframe.