Skip to content

Commit

Permalink
More descriptive examples etc.
Browse files Browse the repository at this point in the history
  • Loading branch information
@eni-awowale
eni-awowale committed Sep 26, 2023
1 parent f5d84b0 commit 2ecb40e
Showing 1 changed file with 31 additions and 64 deletions.
95 changes: 31 additions & 64 deletions docs/how_to_publish_to_cmr_with_earthdata_varinfo.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,32 +5,10 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to use `earthdata-varinfo` to publish UMM-Var records to CMR"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"### Version 2.0.0"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"## How to use `earthdata-varinfo` to publish UMM-Var records to CMR\n",
"### Overview:\n",
"\n",
"This notebook demonstrates how to create and publish, Unified Metadata Model-Variable (UMM-Var) records to NASA's Common Metadata Repository (CMR) with, `earthdata-varinfo` >= 2.0.0. `earthdata-varinfo` utilizes [`python-cmr`](https://github.com/nasa/python_cmr) to query CMR for collection granules to download locally. The `VarInfoFromNetCDF4` class in `earhdata-varinfo` is used to create CMR compliant UMM-Var entries. Lastly, the `requests` library is used to publish UMM-Var records to a given CMR environment (`OPS`, `UAT`, and `SIT`)."
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"This notebook demonstrates how to create and publish, Unified Metadata Model-Variable (UMM-Var) records to NASA's Common Metadata Repository (CMR) with, `earthdata-varinfo` >= 2.0.0. `earthdata-varinfo` utilizes [`python-cmr`](https://github.com/nasa/python_cmr) to query CMR for collection granules to download locally. The `VarInfoFromNetCDF4` class in `earhdata-varinfo` is used to create CMR compliant UMM-Var entries. Lastly, the `requests` library is used to publish UMM-Var records to a given CMR environment (`OPS`, `UAT`, and `SIT`).\n",
"### Setting up your environment to run this notebook\n",
"\n",
"Create and activate your `pyenv` or conda environment, then:\n",
Expand All @@ -45,47 +23,29 @@
"git clone https://github.com/nasa/earthdata-varinfo\n",
"cd earthdata-varinfo\n",
"pip install -e .\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"### Other notebook requirements:\n",
"\n",
"Required packages for `earthdata-varinfo` should be installed as dependencies of PyPI. For local development, without a standard pip installation, third party requirements can be installed from the following files:\n",
"Required packages should be installed as dependencies of `earthdata-varinfo` when doing a pip install from PyPI. \n",
"For local development, without a standard pip installation, third party requirements can be installed from the following files:\n",
"\n",
"```\n",
"pip install -r requirements.txt -r dev-requirements.txt\n",
"pip install notebook\n",
"```"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"```\n",
"### Example usage:\n",
"\n",
"* [GLDAS_NOAH10_3H](https://cmr.uat.earthdata.nasa.gov/search/collections.umm_json?concept-id=C1256543837-EEDTEST)\n",
"* [M2I1NXASM](https://cmr.uat.earthdata.nasa.gov/search/collections.umm_json?concept-id=C1256535511-EEDTEST)"
]
},
{
"attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
"* [M2I1NXASM](https://cmr.uat.earthdata.nasa.gov/search/collections.umm_json?concept-id=C1256535511-EEDTEST)\n",
"\n",
"### Authorization:\n",
"\n",
"* Launchpad or Bearer tokens must be used in order query and publish to CMR.\n",
"* Bearer tokens are tokens that contain the header `Bearer` before the token\n",
" * For example: `Bearer tHiS iS mY tOKen`\n",
"* Launchpad tokens are tokens that do **NOT** contain any headers\n",
" * For example: `tHiS iS mY tOKen`\n",
"* Launchpad or EDL tokens must be used in order query and publish to CMR.\n",
"* Authorization headers for EDL tokens contain the header prefix `Bearer` before the token\n",
" * For example: `Bearer <EDL token>`\n",
"* Authorization headers for Launchpad tokens do **NOT** contain any prefixes in the header\n",
" * For example: `<Launchpad token>`\n",
"\n",
"To request a Launchpad Token visit:\n",
"* [Launchpad Authentication User's Guide](https://wiki.earthdata.nasa.gov/display/CMR/Launchpad+Authentication+User%27s+Guide)"
]
Expand All @@ -105,7 +65,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Update `auth_header` to include your Bearer (e.g. `'Bearer tHiS iS mY tOKen'`) or Launchpad token (e.g. `tHiS iS mY tOKen`)"
"Update `auth_header` to include your EDL token (e.g. `Bearer <EDL token>`) or Launchpad token (e.g. `<Launchpad token>`)"
]
},
{
Expand All @@ -114,7 +74,7 @@
"metadata": {},
"outputs": [],
"source": [
"auth_header = 'Header + Token or Only token'"
"auth_header = 'Bearer <EDL token> or <Launchpad token>'"
]
},
{
Expand Down Expand Up @@ -161,8 +121,12 @@
"\n",
"* Download the most recent granule for **GLDAS_NOAH10_3H**\n",
"* Generate the UMM-Var records for this granule\n",
"* Publish these records to CMR if `publish=True` and return a list of published variable concept-ids\n",
"(e.g. `['V1259971755-EEDTEST', 'V1259971757-EEDTEST' ...]`)"
"* Publish these records to CMR if `publish=True`. \n",
"* If `publish=True`, a list of ingested variable concept-ids or the error(s) from an unsucessful ingest is returned\n",
" * `['V1259971755-EEDTEST', 'V1259971757-EEDTEST', ...]` \n",
" * `['V1259971755-EEDTEST', '#: CMR error 1\\n #: CMR error 2', ...]`\n",
"* If `publish=False` (default) a list of UMM-Var entries is returned:\n",
" * `[...{'Name': 'lat', 'LongName': 'lat', ...}, {'Name': 'time', 'LongName': 'time', ...}...]`"
]
},
{
Expand All @@ -180,8 +144,8 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Publishing and creating UMM-Var entries for **M2I1NXASM** with:\n",
"\n",
"### Publishing and creating UMM-Var entries for **M2I1NXASM**:\n",
"This example is an alternative to using `generate_collection_umm_var`. It demonstrates the individual components of `generate_collection_umm_var` with:\n",
"* `varinfo.cmr_search`: queries CMR for a granule download link and downloads granules locally\n",
"* `VarInfoFromNetCDF4`: varinfo parent class that represents the contents of a granule\n",
"* `varinfo.umm_var`: contains functions for creating and publishing UMM-Var records to CMR\n",
Expand Down Expand Up @@ -230,7 +194,8 @@
"source": [
"Get the granule record and granule download URL with `get_granules` and `get_granule_link`\n",
"\n",
"* `get_granules`: queries `CMR_OPS` for a UMM-G record (granule record) given a collection or granule concept-id\n",
"* `get_granules`: queries `CMR_OPS` (default) for a UMM-G record (granule record) given a collection or granule concept-id\n",
" * you can query any CMR environment by adding `cmr_env=CMR_UAT` or `cmr_env=CMR_SIT`\n",
"* `get_granule_link`: parses the UMM-G record from `get_granules` for a data download URL"
]
},
Expand All @@ -241,7 +206,6 @@
"outputs": [],
"source": [
"granule_response = get_granules(concept_id=collection_concept_id_merra,\n",
" cmr_env=CMR_UAT,\n",
" auth_header=auth_header)\n",
"url = get_granule_link(granule_response)\n",
"url"
Expand Down Expand Up @@ -290,7 +254,9 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Retrieve a dictionary of UMM-Var JSON records."
"Retrieve a dictionary of UMM-Var JSON records\n",
"* Returns a nested dictionary of UMM-Var records with full variable paths as keys and their UMM-Var records as values\n",
"* e.g. `{'/lon': {'Name': 'lon', 'LongName': 'lon', ...}, '/lat': {'Name': 'lat', 'LongName': 'lat', ...}...}`"
]
},
{
Expand All @@ -308,7 +274,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"Publish all UMM-Var records for **M2I1NXASM** to CMR_UAT with `publish_all_umm_var`;\n",
"Publish all UMM-Var records for **M2I1NXASM** to CMR_UAT with `publish_all_umm_var`\n",
"* Returns a dictionary of variable names and variable concept-ids as key value pairs respectively.\n",
"* Example output: ```{'/lon': 'V1259972387-EEDTEST', '/lat': 'V1259972389-EEDTEST'...}```"
]
Expand All @@ -331,6 +297,7 @@
"metadata": {},
"source": [
"### Publish one UMM-Var record with the `var_info.get_variable()` object\n",
"This example is another alternative to using `generate_collection_umm_var`. In this example we use the granule we have already download locally (**M2I1NXASM**) to create and ingest a single UMM-Var record.\n",
"* Use `var_info.get_variable()` to retrieve the variable object from `var_info`\n",
"* Keys are the full variable paths (e.g. `'/TROPPV'`)"
]
Expand Down

0 comments on commit 2ecb40e

Please sign in to comment.