Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Only check if results are cloud hosted when there are results #839

Merged
merged 6 commits into from
Oct 17, 2024

Conversation

jhkennedy
Copy link
Collaborator

@jhkennedy jhkennedy commented Oct 3, 2024

Currently, if you search for a collection that doesn't exist because you have the wrong name for it, earthaccess will bonk pretty hard with an error message that doesn't describe the problem:

>>> import earthaccess
>>> earthaccess.login()
>>> results = earthaccess.search_data(
...     short_name='OPERA_L3_DSWX-HLS_V1_1.0',
...     bounding_box=(-95.19, 30.59, -94.99, 30.79),
...     temporal=('2024-04-30', '2024-05-31'),
... )
Traceback (most recent call last):
  File ".../mambaforge/envs/earthaccess/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-2-7ebcf77bff69>", line 5, in <module>
    results = earthaccess.search_data(
              ^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../Documents/Code/contributing/earthaccess/earthaccess/api.py", line 131, in search_data
    return query.get_all()
           ^^^^^^^^^^^^^^^
  File ".../mambaforge/envs/earthaccess/lib/python3.12/site-packages/cmr/queries.py", line 127, in get_all
    return self.get(self.hits())
           ^^^^^^^^^^^^^^^^^^^^^
  File ".../earthaccess/earthaccess/search.py", line 472, in get
    cloud = self._is_cloud_hosted(response[0])
                                  ~~~~~~~~^^^
IndexError: list index out of range

I would expect this instead:

>>> import earthaccess
>>> earthaccess.login()
>>> results = earthaccess.search_data(
...     short_name='OPERA_L3_DSWX-HLS_V1_1.0',
...     bounding_box=(-95.19, 30.59, -94.99, 30.79),
...     temporal=('2024-04-30', '2024-05-31'),
... )
>>> print(len(results))
0

This PR:

  • only checks if the data is cloud-hosted where there are more than no results returned, allowing earthaccess to return an empty list.
  • adds a header to the changelog since I can't ever remember which changelog styling we use off the top of my head ("Not Keep a Changelog... the other one...")
  • Add forgotten links for the v0.11.0 release to the Changelog

Background:

I was helping a user on Earthdata Forum troubleshoot why their STAC CMR queries weren't working and gave them a version of their script using earthaccess:
https://forum.earthdata.nasa.gov/viewtopic.php?t=5953&sid=578d74d50b040bc37ad8aa188610a90a

When I initially tried to convert the script, I had the short name wrong and encountered the error message above, which I found rather incomprehensible.

I can see having the wrong short name and getting no results as a common situation users could encounter.


As an aside, I had initially naively used the CMR collection name as the short name. What I didn't realize is that In CMR, short_name is not required to be unique, but short_name + version is, so CMR STAC uses short_name + version for the collection ID to provide a unique, human-readable collection ID.

This is the search I was trying to perform:

>>> import earthaccess
>>> earthaccess.login()
>>> results = earthaccess.search_data(
...     short_name='OPERA_L3_DSWX-HLS_V1',
...     bounding_box=(-95.19, 30.59, -94.99, 30.79),
...     temporal=('2024-04-30', '2024-05-31'),
... )
>>> print(len(results))
58

Pull Request (PR) draft checklist - click to expand
  • Please review our
    contributing documentation
    before getting started.
  • Ensure an issue exists representing the problem being solved in this PR.
  • Populate a descriptive title. For example, instead of "Updated README.md", use a
    title such as "Add testing details to the contributor section of the README".
    Example PRs: #763
  • Populate the body of the pull request with:
  • Update CHANGELOG.md with details about your change in a section titled
    ## Unreleased. If such a section does not exist, please create one. Follow
    Common Changelog for your additions.
    Example PRs: #763
  • Update the documentation and/or the README.md with details of changes to the
    earthaccess interface, if any. Consider new environment variables, function names,
    decorators, etc.

Click the "Ready for review" button at the bottom of the "Conversation" tab in GitHub
once these requirements are fulfilled. Don't worry if you see any test failures in
GitHub at this point!

Pull Request (PR) merge checklist - click to expand

Please do your best to complete these requirements! If you need help with any of these
requirements, you can ping the @nsidc/earthaccess-support team in a comment and we
will help you out!

  • Add unit tests for any new features.
  • Apply formatting and linting autofixes. You can add a GitHub comment in this Pull
    Request containing "pre-commit.ci autofix" to automate this.
  • Ensure all automated PR checks (seen at the bottom of the "conversation" tab) pass.
  • Get at least one approving review.

📚 Documentation preview 📚: https://earthaccess--839.org.readthedocs.build/en/839/

@codecov-commenter
Copy link

codecov-commenter commented Oct 3, 2024

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.35%. Comparing base (2f49c08) to head (34f4102).
Report is 62 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #839      +/-   ##
==========================================
- Coverage   73.88%   72.35%   -1.54%     
==========================================
  Files          31       31              
  Lines        2003     1993      -10     
==========================================
- Hits         1480     1442      -38     
- Misses        523      551      +28     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Collaborator

@chuckwondo chuckwondo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes #526, so I've suggested an update to the changelog, plus always assigning a value to cloud.

CHANGELOG.md Outdated Show resolved Hide resolved
earthaccess/search.py Outdated Show resolved Hide resolved
@chuckwondo
Copy link
Collaborator

@jhkennedy, how about a unit test as well?

@jhkennedy
Copy link
Collaborator Author

@chuckwondo yes, that would be a good idea, wouldn't it! 🤦🤣

It'll take me a little but I'll circle back around to that

@jhkennedy
Copy link
Collaborator Author

This fixes #526

🤔 now, how did I miss that issue?! From the conversation there, there's also a refactor we should do.

I prefer to merge this to get the quick fix out the door and circle back to the refactor later. I think that means closing #526 and opening a new issue for it, but leaving #526 open would also work.

earthaccess/search.py Outdated Show resolved Hide resolved
All notable changes to this project will be documented in this file.

The format is based on [Common Changelog](https://common-changelog.org/)
and this project uses [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💯 💯 💯

@mfisher87
Copy link
Collaborator

I prefer to merge this to get the quick fix out the door and circle back to the refactor later. I think that means closing #526 and opening a new issue for it

💯

@chuckwondo
Copy link
Collaborator

This fixes #526

🤔 now, how did I miss that issue?! From the conversation there, there's also a refactor we should do.

I prefer to merge this to get the quick fix out the door and circle back to the refactor later. I think that means closing #526 and opening a new issue for it, but leaving #526 open would also work.

Good point. I forgot about the refactoring bit. I think closing #526 with this PR makes sense, and opening a new issue for the refactoring bit would be my preference (and @mfisher87's preference too, if I'm interpreting his previous comment correctly).

Copy link
Collaborator

@chuckwondo chuckwondo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thanks for adding the ref to Common Changelog that was missing.

Copy link
Collaborator

@chuckwondo chuckwondo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops! I was too hasty in approving. How about a unit test?

@jhkennedy jhkennedy linked an issue Oct 7, 2024 that may be closed by this pull request
@jhkennedy
Copy link
Collaborator Author

Yep, unit test is on my todo list, but I likely won't get to it this week. Any maintainer should feel free to push a test to this branch if they want. Otherwise, I'll get to it next week.

Copy link

github-actions bot commented Oct 16, 2024

Binder 👈 Launch a binder notebook on this branch for commit 8389aa0

I will automatically update this comment whenever this PR is modified

Binder 👈 Launch a binder notebook on this branch for commit 34f4102

@jhkennedy
Copy link
Collaborator Author

Alright, I added a unit test and confirmed the test fails on the main branch but works on this branch.

@jhkennedy
Copy link
Collaborator Author

jhkennedy commented Oct 16, 2024

I also opened the follow-on refactor issue here: #844

@chuckwondo @mfisher87 @betolink feel, free to edit/expand/etc.

Copy link
Collaborator

@chuckwondo chuckwondo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Joe!

CHANGELOG.md Show resolved Hide resolved
@jhkennedy jhkennedy merged commit 9784e4c into nsidc:main Oct 17, 2024
13 checks passed
@jhkennedy jhkennedy deleted the dont-bonk branch October 17, 2024 16:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

DataGranules.get raises IndexError when Granules found = zero
4 participants