quickstarts/using-radiant-mlhub-models-api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='https://radiant-assets.s3-us-west-2.amazonaws.com/PrimaryRadiantMLHubLogo.png' alt='Radiant MLHub Logo' width='300'/>\n",
    "\n",
    "## Using the Radiant MLHub Models API\n",
    "\n",
    "In addition to training datasets, Radiant MLHub also gives access to machine learning models that generate predictions from EO data. You can learn more about the available models and how to query them in [this blog post](https://medium.com/radiant-earth-insights/geospatial-models-now-available-in-radiant-mlhub-a41eb795d7d7).\n",
    "\n",
    "This Jupyter notebook, which you may copy and adapt for any use, shows basic examples of how to use the API to find and use models. Full documentation for the API is available at [docs.mlhub.earth](docs.mlhub.earth).\n",
    "\n",
    "We'll show you how to set up your authentication, see the list of available models, and retrieve the individual models by ID. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Authentication\n",
    "\n",
    "Access to the Radiant MLHub API requires an API key. To get your API key, go to [mlhub.earth](https://mlhub.earth/) and click the \"Sign in / Register\" button in the top right to log in. If you have not used Radiant MLHub before, you will need to sign up and create a new account; otherwise, just sign in. Once you have signed in, click on your user avatar in the top right and select the \"Settings & API keys\" from the dropdown menu.\n",
    "\n",
    "In the **API Keys** section of this page, you will be able to create new API key(s). *Do not share* your API key with others as this may pose a security risk.\n",
    "\n",
    "Next, we will create a `MLHUB_API_KEY` variable that `pystac-client` will use later use to add our API key to all requests:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MLHub API Key:  ································································\n"
     ]
    }
   ],
   "source": [
    "import getpass\n",
    "\n",
    "MLHUB_API_KEY = getpass.getpass(prompt=\"MLHub API Key: \")\n",
    "MLHUB_ROOT_URL = \"https://api.radiant.earth/mlhub/v1\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we connect to the Radiant MLHub API using our API key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools as it\n",
    "import requests\n",
    "from pprint import pprint\n",
    "from urllib.parse import urljoin\n",
    "\n",
    "from pystac_client import Client\n",
    "from pystac import StacIO\n",
    "from pystac.extensions.scientific import ScientificExtension\n",
    "\n",
    "client = Client.open(\n",
    "    MLHUB_ROOT_URL, parameters={\"key\": MLHUB_API_KEY}, ignore_conformance=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### List models\n",
    "\n",
    "A **model** in the Radiant MLHub API is a [STAC Item](https://github.com/radiantearth/stac-spec/tree/master/item-spec/) that implements the [ML Model Extension](https://github.com/stac-extensions/ml-model). This extensions provides high-level metadata about the model, including a description of the architecture and training environment. It also includes links to the datasets on which the model was trained.\n",
    "\n",
    "We start by creating a `requests.Session` instance so that we can include the API key in all of our requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MLHubSession(requests.Session):\n",
    "    def __init__(self, *args, api_key=None, **kwargs):\n",
    "        super().__init__(*args, **kwargs)\n",
    "        self.params.update({\"key\": api_key})\n",
    "\n",
    "    def request(self, method, url, *args, **kwargs):\n",
    "        url_prefix = MLHUB_ROOT_URL.rstrip(\"/\") + \"/\"\n",
    "        url = urljoin(url_prefix, url)\n",
    "        return super().request(method, url, *args, **kwargs)\n",
    "\n",
    "\n",
    "session = MLHubSession(api_key=MLHUB_API_KEY)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we list the available models using the `/models` endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total Datasets: 1\n",
      "-----\n",
      "model-cyclone-wind-estimation-torchgeo-v1\n"
     ]
    }
   ],
   "source": [
    "response = session.get(\"/models\")\n",
    "models = response.json()\n",
    "\n",
    "models_limit = 30\n",
    "\n",
    "print(f\"Total Datasets: {len(models)}\")\n",
    "print(\"-----\")\n",
    "for model in it.islice(models, models_limit):\n",
    "    model_id = model[\"id\"]\n",
    "    print(f\"{model_id}\")\n",
    "if len(models) > models_limit:\n",
    "    print(\"...\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model metadata\n",
    "\n",
    "We can also use the STAC API Item Search capability to a fetch model metadata by ID. Let's fetch the Tropical Cyclone Model using its ID."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'assets': {'inferencing-checkpoint': {'href': 'https://api.radiant.earth/mlhub/v1/download/gAAAAABh1Lo2NQfk1Km_1oYLIx7_EdvyKXpHQZ0Hq2_Tj4gZppf4GGa2RFI9xXQP5x_aVRxXoYPqThgrZwgrVEtC6E_V9gl0-aVdxPY2rhg82Ahm2ap1OfmD2hT2IjM270TqQ_qYCu5httnFFRNaUKncz6BOKH7spCf8Agdp5-AnBtq-99LvDP2B_M_T5guUIK3h8pg1YzbIWXVTLAzkrFubZO1yRde9Hw==',\n",
      "                                       'roles': ['ml-model:checkpoint'],\n",
      "                                       'title': 'Final model checkpoint',\n",
      "                                       'type': 'application/octet-stream'},\n",
      "            'inferencing-compose': {'href': 'https://api.radiant.earth/mlhub/v1/download/gAAAAABh1Lo2SptgZTV6PvAyDE3FNfhyme0nsLIXt-CcGOTcUWfHOUp37prpshZh2U7bm0qR-JsnwkRCfjq2RfFilWX2uscNafq2c6fmy25CZYEdOghu-l0eQPeguKvRlzcdQ8eCZP4Y9PAr2httfkCaT8Be16YyaFtI2OErhc0mAWZ38WPRBenrmvuJXGQTrcMwDoR3ncigfX_Lp1iCONWYgcAaoCpOR1I5L49IM3q7MzR0wKk1XQRYMPZaiIYMWXnM5ziOtnUz',\n",
      "                                    'roles': ['ml-model:inference-runtime'],\n",
      "                                    'title': 'Model inferencing runtime',\n",
      "                                    'type': 'text/x-yaml; '\n",
      "                                            'application=compose'}},\n",
      " 'bbox': [-179.999, -89.999, 179.999, 89.999],\n",
      " 'collection': 'model-cyclone-wind-estimation-torchgeo',\n",
      " 'geometry': {'coordinates': [[[-179.999, -89.999],\n",
      "                               [179.999, -89.999],\n",
      "                               [179.999, 89.999],\n",
      "                               [-179.999, 89.999],\n",
      "                               [-179.999, -89.999]]],\n",
      "              'type': 'Polygon'},\n",
      " 'id': 'model-cyclone-wind-estimation-torchgeo-v1',\n",
      " 'links': [{'href': 'http://api.radiant.earth/mlhub/v1/collections/model-cyclone-wind-estimation-torchgeo',\n",
      "            'rel': 'collection',\n",
      "            'type': 'application/json'},\n",
      "           {'href': 'http://api.radiant.earth/mlhub/v1/collections/model-cyclone-wind-estimation-torchgeo',\n",
      "            'rel': 'parent',\n",
      "            'type': 'application/json'},\n",
      "           {'href': 'https://api.radiant.earth/mlhub/v1',\n",
      "            'rel': <RelType.ROOT: 'root'>,\n",
      "            'title': 'Radiant MLHub API',\n",
      "            'type': <MediaType.JSON: 'application/json'>},\n",
      "           {'href': 'http://api.radiant.earth/mlhub/v1/collections/model-cyclone-wind-estimation-torchgeo/items/model-cyclone-wind-estimation-torchgeo-v1',\n",
      "            'rel': 'self',\n",
      "            'type': 'application/geo+json'},\n",
      "           {'href': 'http://api.radiant.earth/mlhub/v1/collections/model-cyclone-wind-estimation-torchgeo/items/model-cyclone-wind-estimation-torchgeo-v1/tiles',\n",
      "            'rel': 'alternate',\n",
      "            'title': 'tiles',\n",
      "            'type': 'application/json'}],\n",
      " 'properties': {'datetime': None,\n",
      "                'end_datetime': '2019-12-12T23:59:59Z',\n",
      "                'license': 'MIT',\n",
      "                'ml-model:architecture': 'resnet18',\n",
      "                'ml-model:learning_approach': 'supervised',\n",
      "                'ml-model:prediction_type': 'regression',\n",
      "                'ml-model:type': 'ml-model',\n",
      "                'providers': [{'email': 'caleb.robinson@microsoft.com',\n",
      "                               'name': 'Microsoft AI for Good Research Lab '\n",
      "                                       '(Caleb Robinson)',\n",
      "                               'roles': ['producer'],\n",
      "                               'url': 'https://www.microsoft.com/en-us/ai/ai-for-good'}],\n",
      "                'sci:citation': 'Caleb Robinson. (2021). Tropical Cyclone Wind '\n",
      "                                'Estimation model (2.0). Zenodo. '\n",
      "                                'https://doi.org/10.5281/zenodo.5773331.',\n",
      "                'sci:doi': '10.5281/zenodo.5773331',\n",
      "                'sci:publications': [{'citation': 'Stewart, A., Robinson, C. '\n",
      "                                                  'and Corley, I., 2021. '\n",
      "                                                  'TorchGeo: deep learning '\n",
      "                                                  'with geospatial data. arXiv '\n",
      "                                                  'preprint arXiv:2111.08872, '\n",
      "                                                  '[online] (11). Available '\n",
      "                                                  'at: '\n",
      "                                                  '<https://arxiv.org/abs/2111.08872> '\n",
      "                                                  '[Date Accessed].'}],\n",
      "                'start_datetime': '2000-01-01T00:00:00Z'},\n",
      " 'stac_extensions': ['https://stac-extensions.github.io/ml-model/v1.0.0/schema.json',\n",
      "                     'https://stac-extensions.github.io/scientific/v1.0.0/schema.json'],\n",
      " 'stac_version': '1.0.0',\n",
      " 'type': 'Feature'}\n"
     ]
    }
   ],
   "source": [
    "results = client.search(ids=[\"model-cyclone-wind-estimation-torchgeo-v1\"])\n",
    "cyclone_model = next(results.get_items())\n",
    "pprint(cyclone_model.to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is a lot of metadata here, but let's walk through some of the key fields and what they mean."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Spatiotemporal Metadata\n",
    "\n",
    "The `geometry` and `bbox` fields represent geographic areas over which the model can be used. Similarly, the `start_datetime` and `end_datetime` properties represent the temporal range of data that be used by the model. The special value `9999-12-31T23:59:59Z` for the `end_datetime` property means that any date after the `start_datetime` will work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Bounding Box: [-179.999, -89.999, 179.999, 89.999]\n",
      "Geometry: {'type': 'Polygon', 'coordinates': [[[-179.999, -89.999], [179.999, -89.999], [179.999, 89.999], [-179.999, 89.999], [-179.999, -89.999]]]}\n",
      "Datetime Range: 2000-01-01T00:00:00Z/2019-12-12T23:59:59Z\n"
     ]
    }
   ],
   "source": [
    "print(f\"Bounding Box: {cyclone_model.bbox}\")\n",
    "print(f\"Geometry: {cyclone_model.geometry}\")\n",
    "\n",
    "start_datetime = cyclone_model.to_dict()[\"properties\"][\"start_datetime\"]\n",
    "end_datetime = cyclone_model.to_dict()[\"properties\"][\"end_datetime\"]\n",
    "end_datetime = end_datetime if end_datetime != \"9999-12-31T23:59:59Z\" else \"...\"\n",
    "\n",
    "print(f\"Datetime Range: {start_datetime}/{end_datetime}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Citation Information\n",
    "\n",
    "Citation and publication metadata related to the model will be described using the STAC [Scientific Citation Extension](https://github.com/stac-extensions/scientific). Let's first check if this Item implements the Scientific Citation Extension, and then print some of the citation metadata if it does."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Has citation information: True\n",
      "\n",
      "Citation: Caleb Robinson. (2021). Tropical Cyclone Wind Estimation model (2.0). Zenodo. https://doi.org/10.5281/zenodo.5773331.\n",
      "DOI: 10.5281/zenodo.5773331\n",
      "\n",
      "Publications\n",
      "------------\n",
      "- Citation: Stewart, A., Robinson, C. and Corley, I., 2021. TorchGeo: deep learning with geospatial data. arXiv preprint arXiv:2111.08872, [online] (11). Available at: <https://arxiv.org/abs/2111.08872> [Date Accessed].\n"
     ]
    }
   ],
   "source": [
    "implements_sci_ext = ScientificExtension.has_extension(cyclone_model)\n",
    "print(f\"Has citation information: {implements_sci_ext}\\n\")\n",
    "\n",
    "if implements_sci_ext:\n",
    "    sci_ext = ScientificExtension.ext(cyclone_model)\n",
    "    if sci_ext.citation is not None:\n",
    "        print(f\"Citation: {sci_ext.citation}\")\n",
    "    if sci_ext.doi is not None:\n",
    "        print(f\"DOI: {sci_ext.doi}\")\n",
    "    if sci_ext.publications is not None:\n",
    "        print(\"\\nPublications\\n------------\")\n",
    "        for publication in sci_ext.publications:\n",
    "            if publication.citation is not None:\n",
    "                print(f\"- Citation: {publication.citation}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Metadata\n",
    "\n",
    "Metadata related to the ML model itself will be described using the STAC [ML Model Extension](https://github.com/stac-extensions/ml-model). This extension includes properties describing model architecture and prediction type, and also defines type STAC Link and Asset types that are specific to ML models. Let's take a closer look at what each of these properties represents.\n",
    "\n",
    "*Note: PySTAC does not currently have built-in support for the ML Model Extension, so we will access these properties using the `items.properties` dictionary directly.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Architecture: resnet18\n",
      "Learning Approach: supervised\n",
      "Prediction Type: regression\n"
     ]
    }
   ],
   "source": [
    "architecture = cyclone_model.properties[\"ml-model:architecture\"]\n",
    "learning_approach = cyclone_model.properties[\"ml-model:learning_approach\"]\n",
    "prediction_type = cyclone_model.properties[\"ml-model:prediction_type\"]\n",
    "\n",
    "print(f\"Architecture: {architecture}\")\n",
    "print(f\"Learning Approach: {learning_approach}\")\n",
    "print(f\"Prediction Type: {prediction_type}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that this is based on a residual neural network (ResNet18), that it generates a regression output, and that it was trained using supervised learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Assets\n",
    "\n",
    "The ML Model Extension describes some STAC Asset types that are specific to ML models. In particular, the Tropical Cyclone Model provides Assets for a model checkpoint and a [Compose file](https://github.com/compose-spec/compose-spec/blob/master/spec.md), either of which may be used to run the model to generate inferences. These assets have roles of `ml-model:checkpoint` and `ml-model:inference-runtime`, respectively. Let's take a look at these Assets and then download the Compose file to see what that looks like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checkpoint Asset\n",
      "----------------\n",
      "{'href': 'https://api.radiant.earth/mlhub/v1/download/gAAAAABh1Lo2NQfk1Km_1oYLIx7_EdvyKXpHQZ0Hq2_Tj4gZppf4GGa2RFI9xXQP5x_aVRxXoYPqThgrZwgrVEtC6E_V9gl0-aVdxPY2rhg82Ahm2ap1OfmD2hT2IjM270TqQ_qYCu5httnFFRNaUKncz6BOKH7spCf8Agdp5-AnBtq-99LvDP2B_M_T5guUIK3h8pg1YzbIWXVTLAzkrFubZO1yRde9Hw==',\n",
      " 'roles': ['ml-model:checkpoint'],\n",
      " 'title': 'Final model checkpoint',\n",
      " 'type': 'application/octet-stream'}\n",
      "\n",
      "Inference Runtime Asset\n",
      "----------------\n",
      "{'href': 'https://api.radiant.earth/mlhub/v1/download/gAAAAABh1Lo2SptgZTV6PvAyDE3FNfhyme0nsLIXt-CcGOTcUWfHOUp37prpshZh2U7bm0qR-JsnwkRCfjq2RfFilWX2uscNafq2c6fmy25CZYEdOghu-l0eQPeguKvRlzcdQ8eCZP4Y9PAr2httfkCaT8Be16YyaFtI2OErhc0mAWZ38WPRBenrmvuJXGQTrcMwDoR3ncigfX_Lp1iCONWYgcAaoCpOR1I5L49IM3q7MzR0wKk1XQRYMPZaiIYMWXnM5ziOtnUz',\n",
      " 'roles': ['ml-model:inference-runtime'],\n",
      " 'title': 'Model inferencing runtime',\n",
      " 'type': 'text/x-yaml; application=compose'}\n"
     ]
    }
   ],
   "source": [
    "checkpoint_asset = next(\n",
    "    asset\n",
    "    for asset_key, asset in cyclone_model.assets.items()\n",
    "    if \"ml-model:checkpoint\" in asset.roles\n",
    ")\n",
    "\n",
    "print(\"Checkpoint Asset\\n----------------\")\n",
    "pprint(checkpoint_asset.to_dict())\n",
    "\n",
    "inference_runtime_asset = next(\n",
    "    asset\n",
    "    for asset_key, asset in cyclone_model.assets.items()\n",
    "    if \"ml-model:inference-runtime\" in asset.roles\n",
    ")\n",
    "\n",
    "print()\n",
    "\n",
    "print(\"Inference Runtime Asset\\n----------------\")\n",
    "pprint(inference_runtime_asset.to_dict())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll read the Compose file and examine its contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "services:\n",
      "  inference:\n",
      "    image: radiantearth/cyclone-model-torchgeo:1\n",
      "    volumes:\n",
      "      - \"${INPUT_DATA}:/var/data/input\"\n",
      "      - \"${OUTPUT_DATA}:/var/data/output\"\n",
      "\n"
     ]
    }
   ],
   "source": [
    "stac_io = StacIO.default()\n",
    "compose_file_contents = stac_io.read_text(inference_runtime_asset.href)\n",
    "print(compose_file_contents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find more details about running models using Compose files in the [\"Inference/Training Runtimes\"](https://github.com/stac-extensions/ml-model#inferencetraining-runtimes) section of the ML Model Extension documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next Steps\n",
    "\n",
    "This tutorial was a quick introduction to working with the Radiant MLHub Models API in a notebook. For more, see:\n",
    "\n",
    "- [Reading Data from the STAC API](./reading-stac.ipynb)\n",
    "- [Using the Radiant MLHub API](./using-radiant-mlhub-api.ipynb)\n",
    "- [How to use the Radiant MLHub API to browse and download the LandCoverNet dataset](../tutorials/radiant-mlhub-landcovernet.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}