python-explore-apis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Amanda Devine  \n",
    "21 October 2019  \n",
    "WS61 Biodiversity Informatics 101: Preparing for Biodiversity_Next and Beyond    \n",
    "https://github.com/amdevine/bdn-python-apis\n",
    "\n",
    "\n",
    "Retrieving data from Web APIs using the Python Requests library\n",
    "===================================================="
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This Jupyter notebook walks through the process of using the Python Requests library, in conjunction with the Pandas library, to download data from website REST API services.\n",
    "\n",
    "As an example dataset, this notebook uses data on the US National Parks available via the United States National Park Service Data API."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Definitions\n",
    "------------------\n",
    "\n",
    "- **API**: Application Programming Interface. A special page on a website that provides structured data for other programs and applications.\n",
    "    + **REST**: Newer API format, easily accessed via URL, returns data in a variety of formats including JSON \n",
    "    + **SOAP**: Older API format, more complex, accepts queries and returns data in XML only\n",
    "\n",
    "\n",
    "- **GET Request**: An HTTP command to retrieve code and data from a website. GET requests can be made in a variety of ways; the Requests library offers a very easy way to make GET requests from Python.\n",
    "\n",
    "\n",
    "- **JSON**: JavaScript Object Notation. A common format of structuring data, analogous to a Python dictionary.\n",
    "\n",
    "\n",
    "- **Base URL**: The website URL for all API data. A variety of endpoints can be added to the base URL.\n",
    "    > NPS Base URL: `https://developer.nps.gov/api/v1`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- **Endpoint**: The specific URL where the API page can be found. Each website might have multiple endpoints that return different kinds of data. \n",
    "    > Parks Endpoint: `https://developer.nps.gov/api/v1/parks`\n",
    "\n",
    "\n",
    "- **Parameter**: An additional criterion that is added to the endpoint to filter data returned. Parameters are usually added to the endpoint with a `?` character, and are in the format of `field=value`. Multiple parameters can be added to an endpoint, separated by a `&`.\n",
    "    > parkCode, stateCode, and limit parameters: `https://developer.nps.gov/api/v1/parks?parkCode=yell&stateCode=WY&limit=5`\n",
    "    \n",
    "\n",
    "- **API Key**: A string of characters assigned by the website to identify the user requesting data via the API. For many API services, an API Key is required when making a request.\n",
    "    > National Parks API Key: `https://developer.nps.gov/api/v1/parks?api_key=1mdaBewB37R0kUA2ZtfA6URe7PeUsig6jLQmSXyx` (not a real key!)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "NPS Data API\n",
    "-----------------\n",
    "\n",
    "The National Park Service (NPS) Data API is a service provided by the United States National Park Service to supply data about natural areas managed by the National Park Service. Data available through the API include \n",
    "- park information\n",
    "- campground information\n",
    "- alerts, events, news, and educational resources.\n",
    "\n",
    "All users of the NPS Data API are required to have an API Key. Keys can be obtained here: https://www.nps.gov/subjects/developer/get-started.htm\n",
    "\n",
    "API documentation is available here: https://www.nps.gov/subjects/developer/api-documentation.htm  \n",
    "This documentation shows a list of possible endpoints. Clicking on an endpoint shows the parameters that can be supplied to that endpoint to filter results, as well as an example of the JSON data that will be returned from a GET request.\n",
    "\n",
    "The NPS Data API GitHub Repository contains examples of using Python and/or PHP to retrieve data. It is available here: https://github.com/nationalparkservice/nps-api-samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Python `requests` library\n",
    "-----------------------------\n",
    "\n",
    "The Python Requests library provides a simple way to query and retrieve JSON data from API services. It is a wrapper that calls other Python libraries such as `urllib`.\n",
    "\n",
    "Requests can be download via Terminal/Command Line with `conda install requests` (if you are running Anaconda/Miniconda) or `pip install requests` (if you are running base Python).\n",
    "\n",
    "Making a GET request to retrieve JSON data with Requests is usually done in the following way:\n",
    "\n",
    "    import requests\n",
    "    url = 'https://baseurl.com/endpoint'\n",
    "    params = {\n",
    "        'field1': 'value1',\n",
    "        'field2': 'value2',\n",
    "    }\n",
    "    r = requests.get(url, params).json()\n",
    "\n",
    "This creates a dictionary, `r`, from which data can be accessed via key-value pairs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Parameters supplied to a GET request are automatically encoded to HTML specifications. \n",
    "\n",
    "> e.g. the parameter `'q': 'César E. Chávez'` is automatically encoded to `q=C%C3%A9sar%20E.%20Ch%C3%A1vez` by Requests\n",
    "\n",
    "Requests has significantly more functionality and many more options beyond simple GET requests. Quickstart documentation is available here: https://2.python-requests.org/en/master/user/quickstart/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Setup\n",
    "--------\n",
    "\n",
    "Import the `requests` library to retrieve data from the NPS Data API. Import the `pandas` library to work with retrieved data and export as tabular files. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import pandas as pd\n",
    "import pprint   # Prints dictionaries/JSON in a more human-readable format"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Assign the API Key to a constant. If the code will be available on GitHub or other public sites, avoid assigning it directly and import it from a local file instead. (And don't add this file to GitHub!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "API Key: API_KEY\n"
     ]
    }
   ],
   "source": [
    "# API_KEY = '1mdaBewB37R0kUA2ZtfA6URe7PeUsig6jLQmSXyx'\n",
    "with open('api_key_file.txt', 'r') as f:\n",
    "    API_KEY = f.read().strip()\n",
    "print(\"API Key: {}\".format(\"API_KEY\")) # Remove quotes to display actual API_KEY"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Make a GET request to the API to retrieve data\n",
    "--------------------------------------------------------\n",
    "\n",
    "Use `requests.get()` to make an **HTTP GET Request** to the API. Any parameters can be provided to requests.get() as a dictionary."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "The following request will return data on up to 100 parks in the state of California. `api_key` is a required parameter for all NPS Data API requests. `stateCode` filters parks based on two-letter US state abbreviations. `fields` specifies additional fields to return in addition to the default fields. `limit` specifies the maximum number of results to return."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "url = 'https://developer.nps.gov/api/v1/parks'\n",
    "params = {\n",
    "    'api_key': API_KEY,\n",
    "    'stateCode': 'CA', \n",
    "    'fields': 'entranceFees',\n",
    "    'limit': 100\n",
    "}\n",
    "r = requests.get(url, params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "`requests.get()` returns a variety of information about the web page retrieved. This info can be useful for troubleshooting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The response code is: 200\n",
      "\n",
      "The retrieved URL is: r.url\n",
      "\n",
      "The first 300 characters of the retrieved text are:\n",
      "{\"total\":\"33\",\"data\":[{\"states\":\"CA\",\"entranceFees\":[{\"cost\":\"39.9000\",\"description\":\"A daily scheduled ferry to Alcatraz has a round-trip fee. Reservations are strongly recommended as we often sell out. See www.alcatrazcruises.com for reservations, and ticket types and pricing.\",\"title\":\"Adult Day \n"
     ]
    }
   ],
   "source": [
    "print(\"The response code is: {}\".format(r.status_code))\n",
    "print(\"\\nThe retrieved URL is: {}\".format(\"r.url\")) # Remove quotes to display URL\n",
    "print(\"\\nThe first 300 characters of the retrieved text are:\\n{}\".format(r.text[:300]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Work with retrieved data\n",
    "\n",
    "### Convert GET request object to dictionary\n",
    "\n",
    "When the API data are supplied in the **JSON** format, they can easily be turned into a Python dictionary using the Requests `.json()` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First item in 'data':\n",
      "\n",
      "{'description': 'Alcatraz reveals stories of American incarceration, justice, '\n",
      "                'and our common humanity. This small island was once a fort, a '\n",
      "                'military prison, and a maximum security federal penitentiary. '\n",
      "                'In 1969, the Indians of All Tribes occupied Alcatraz for 19 '\n",
      "                'months in the name of freedom and Native American civil '\n",
      "                \"rights. We invite you to explore Alcatraz's complex history \"\n",
      "                'and natural beauty.',\n",
      " 'designation': '',\n",
      " 'directionsInfo': 'The Alcatraz Ferry Terminal is located on The Embarcadero '\n",
      "                   'near the intersection of Bay Street at Pier 33.',\n",
      " 'directionsUrl': 'http://home.nps.gov/alca/planyourvisit/directions.htm',\n",
      " 'entranceFees': [{'cost': '39.9000',\n",
      "                   'description': 'A daily scheduled ferry to Alcatraz has a '\n",
      "                                  'round-trip fee. Reservations are strongly '\n",
      "                                  'recommended as we often sell out. See '\n",
      "                                  'www.alcatrazcruises.com for reservations, '\n",
      "                                  'and ticket types and pricing.',\n",
      "                   'title': 'Adult Day Ticket (Ferry plus audio tour)'},\n",
      "                  {'cost': '47.3000',\n",
      "                   'description': 'A daily scheduled ferry to Alcatraz has a '\n",
      "                                  'round-trip fee. Reservations are strongly '\n",
      "                                  'recommended as we often sell out. See '\n",
      "                                  'www.alcatrazcruises.com for reservations, '\n",
      "                                  'and ticket types and pricing.',\n",
      "                   'title': 'Adult Night Ticket (Ferry plus audio tour)'},\n",
      "                  {'cost': '92.3000',\n",
      "                   'description': 'A daily scheduled ferry to Alcatraz has a '\n",
      "                                  'round-trip fee. Reservations are strongly '\n",
      "                                  'recommended as we often sell out. See '\n",
      "                                  'www.alcatrazcruises.com for reservations, '\n",
      "                                  'and ticket types and pricing.',\n",
      "                   'title': 'Adult Behind the Scenes Tour Ticket (Ferry plus '\n",
      "                            'audio tour)'}],\n",
      " 'fullName': 'Alcatraz Island',\n",
      " 'id': 'C08AD828-98FF-478E-A63C-614E7534274B',\n",
      " 'latLong': 'lat:37.82676234, long:-122.4230206',\n",
      " 'name': 'Alcatraz Island',\n",
      " 'parkCode': 'alca',\n",
      " 'states': 'CA',\n",
      " 'url': 'https://www.nps.gov/alca/index.htm',\n",
      " 'weatherInfo': 'The climate on Alcatraz is unpredictable and can change '\n",
      "                'suddenly. Cold, foggy mornings may give way to sunny '\n",
      "                'afternoons, which in turn can shift quickly back to more fog '\n",
      "                'and blustery winds. The most pleasant weather usually occurs '\n",
      "                'in spring and fall. Summers tend to be cool and foggy, winter '\n",
      "                'is our rainy season. Temperatures on Alcatraz seldom rise '\n",
      "                'above 75°F (24°C) or fall below 38°'}\n"
     ]
    }
   ],
   "source": [
    "parks_data = r.json()\n",
    "\n",
    "print(\"First item in 'data':\\n\")\n",
    "pprint.pprint(parks_data['data'][0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Create a Pandas DataFrame\n",
    "\n",
    "Pandas DataFrames make it easy to work with the retrieved data in a tabular format. \n",
    "\n",
    "This code filters the retrieved data to states and associated lat/long coordinate for each park."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>parkCode</th>\n",
       "      <th>fullName</th>\n",
       "      <th>designation</th>\n",
       "      <th>states</th>\n",
       "      <th>latLong</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>alca</td>\n",
       "      <td>Alcatraz Island</td>\n",
       "      <td></td>\n",
       "      <td>CA</td>\n",
       "      <td>lat:37.82676234, long:-122.4230206</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>cabr</td>\n",
       "      <td>Cabrillo National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>CA</td>\n",
       "      <td>lat:32.6722503, long:-117.2415985</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>cali</td>\n",
       "      <td>California National Historic Trail</td>\n",
       "      <td>National Historic Trail</td>\n",
       "      <td>CA,CO,ID,KS,MO,NE,NV,OR,UT,WY</td>\n",
       "      <td></td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>camo</td>\n",
       "      <td>Castle Mountains National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>CA</td>\n",
       "      <td>lat:35.29156348, long:-115.0935606</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>cech</td>\n",
       "      <td>César E. Chávez National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>CA</td>\n",
       "      <td>lat:35.22729389, long:-118.5615781</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>chis</td>\n",
       "      <td>Channel Islands National Park</td>\n",
       "      <td>National Park</td>\n",
       "      <td>CA</td>\n",
       "      <td>lat:33.98680093, long:-119.9112735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>deva</td>\n",
       "      <td>Death Valley National Park</td>\n",
       "      <td>National Park</td>\n",
       "      <td>CA,NV</td>\n",
       "      <td>lat:36.48753731, long:-117.134395</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>depo</td>\n",
       "      <td>Devils Postpile National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>CA</td>\n",
       "      <td>lat:37.6152564, long:-119.0873903</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>euon</td>\n",
       "      <td>Eugene O'Neill National Historic Site</td>\n",
       "      <td>National Historic Site</td>\n",
       "      <td>CA</td>\n",
       "      <td>lat:37.82604456, long:-122.0271566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>fopo</td>\n",
       "      <td>Fort Point National Historic Site</td>\n",
       "      <td>National Historic Site</td>\n",
       "      <td>CA</td>\n",
       "      <td>lat:37.80837439, long:-122.473747</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  parkCode                               fullName              designation  \\\n",
       "0     alca                        Alcatraz Island                            \n",
       "1     cabr             Cabrillo National Monument        National Monument   \n",
       "2     cali     California National Historic Trail  National Historic Trail   \n",
       "3     camo     Castle Mountains National Monument        National Monument   \n",
       "4     cech      César E. Chávez National Monument        National Monument   \n",
       "5     chis          Channel Islands National Park            National Park   \n",
       "6     deva             Death Valley National Park            National Park   \n",
       "7     depo      Devils Postpile National Monument        National Monument   \n",
       "8     euon  Eugene O'Neill National Historic Site   National Historic Site   \n",
       "9     fopo      Fort Point National Historic Site   National Historic Site   \n",
       "\n",
       "                          states                             latLong  \n",
       "0                             CA  lat:37.82676234, long:-122.4230206  \n",
       "1                             CA   lat:32.6722503, long:-117.2415985  \n",
       "2  CA,CO,ID,KS,MO,NE,NV,OR,UT,WY                                      \n",
       "3                             CA  lat:35.29156348, long:-115.0935606  \n",
       "4                             CA  lat:35.22729389, long:-118.5615781  \n",
       "5                             CA  lat:33.98680093, long:-119.9112735  \n",
       "6                          CA,NV   lat:36.48753731, long:-117.134395  \n",
       "7                             CA   lat:37.6152564, long:-119.0873903  \n",
       "8                             CA  lat:37.82604456, long:-122.0271566  \n",
       "9                             CA   lat:37.80837439, long:-122.473747  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "parks_df = pd.DataFrame(parks_data['data'])\n",
    "locations_df = parks_df[['parkCode', 'fullName', 'designation', 'states', 'latLong']]\n",
    "locations_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Restructure/flatten data\n",
    "\n",
    "If the results contain nested data that need to be flattened (e.g. multiple `entranceFees` for each park), or the results could be otherwise restructured in a more \"tidy\" format, a new list of dictionaries can be created by iterating through the data. This list can then be converted to a DataFrame."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "JSON data for Yosemite National Park's multiple entrance fees:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'description': 'Not just a great valley, but a shrine to human foresight, the '\n",
      "                'strength of granite, the power of glaciers, the persistence '\n",
      "                'of life, and the tranquility of the High Sierra.\\n'\n",
      "                '\\n'\n",
      "                'First protected in 1864, Yosemite National Park is best known '\n",
      "                'for its waterfalls, but within its nearly 1,200 square miles, '\n",
      "                'you can find deep valleys, grand meadows, ancient giant '\n",
      "                'sequoias, a vast wilderness area, and much more.',\n",
      " 'designation': 'National Park',\n",
      " 'directionsInfo': 'You can drive to Yosemite all year and enter via Highways '\n",
      "                   '41, 140, and 120 from the west. Tioga Pass Entrance (via '\n",
      "                   'Highway 120 from the east) is closed from around November '\n",
      "                   'through late May or June. Hetch Hetchy is open all year '\n",
      "                   'but may close intermittently due to snow.\\n'\n",
      "                   '\\n'\n",
      "                   'Please note that GPS units do not always provide accurate '\n",
      "                   'directions to or within Yosemite.',\n",
      " 'directionsUrl': 'http://www.nps.gov/yose/planyourvisit/driving.htm',\n",
      " 'entranceFees': [{'cost': '35.0000',\n",
      "                   'description': 'This fee is valid for seven days.',\n",
      "                   'title': 'Non-commercial car, pickup truck, RV, or van with '\n",
      "                            '15 or fewer passenger seats'},\n",
      "                  {'cost': '30.0000',\n",
      "                   'description': 'The fee is valid for seven days. Cost is '\n",
      "                                  'per motorcycle (not per person).',\n",
      "                   'title': 'Motorcycle'},\n",
      "                  {'cost': '20.0000',\n",
      "                   'description': 'This fee is valid for seven days. People 15 '\n",
      "                                  'years and younger are free. Cost is per '\n",
      "                                  'person.',\n",
      "                   'title': 'Foot, bicycle, horse, or non-commercial bus or '\n",
      "                            'van with more than 15 passenger seats'},\n",
      "                  {'cost': '25.0000',\n",
      "                   'description': 'The fee is $25 plus $15 per person.',\n",
      "                   'title': 'Commercial Tour (sedan up to six seats)'},\n",
      "                  {'cost': '125.0000',\n",
      "                   'description': '',\n",
      "                   'title': 'Commercial Tour (van, 7-15 seats, regardless of '\n",
      "                            'occupancy)'},\n",
      "                  {'cost': '200.0000',\n",
      "                   'description': '',\n",
      "                   'title': 'Commercial Tour (mini bus, 16-25 seats, '\n",
      "                            'regardless of occupancy)'},\n",
      "                  {'cost': '300.0000',\n",
      "                   'description': '',\n",
      "                   'title': 'Commercial Tour (motor coach, 26 or more seats, '\n",
      "                            'regardless of occupancy)'}],\n",
      " 'fullName': 'Yosemite National Park',\n",
      " 'id': '4324B2B4-D1A3-497F-8E6B-27171FAE4DB2',\n",
      " 'latLong': 'lat:37.84883288, long:-119.5571873',\n",
      " 'name': 'Yosemite',\n",
      " 'parkCode': 'yose',\n",
      " 'states': 'CA',\n",
      " 'url': 'https://www.nps.gov/yose/index.htm',\n",
      " 'weatherInfo': 'Yosemite National Park covers nearly 1,200 square miles '\n",
      "                '(3,100 square km) in the Sierra Nevada, with elevations '\n",
      "                'ranging from about 2,000 feet (600 m) to 13,000 ft (4,000 m). '\n",
      "                'Yosemite receives 95% of its precipitation between October '\n",
      "                'and May (and over 75% between November and March). Most of '\n",
      "                'Yosemite is blanketed in snow from about November through '\n",
      "                'May. (Yosemite Valley can be rainy or snowy in any given '\n",
      "                'winter storm.)'}\n"
     ]
    }
   ],
   "source": [
    "pprint.pprint(parks_data['data'][-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For each park in the dataset, and for each entrance fee in that park, add some park and fee values as a dictionary to a new `entry_fee_data` list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'designation': '',\n",
      "  'fee_description': 'A daily scheduled ferry to Alcatraz has a round-trip '\n",
      "                     'fee. Reservations are strongly recommended as we often '\n",
      "                     'sell out. See www.alcatrazcruises.com for reservations, '\n",
      "                     'and ticket types and pricing.',\n",
      "  'fee_type': 'Adult Behind the Scenes Tour Ticket (Ferry plus audio tour)',\n",
      "  'fee_usd': '92.3000',\n",
      "  'fullName': 'Alcatraz Island',\n",
      "  'parkCode': 'alca'},\n",
      " {'designation': 'National Monument',\n",
      "  'fee_description': 'The pass is valid for seven full days.',\n",
      "  'fee_type': 'Cabrillo Entrance Fee - Per non commercial vehicle',\n",
      "  'fee_usd': '20.0000',\n",
      "  'fullName': 'Cabrillo National Monument',\n",
      "  'parkCode': 'cabr'}]\n"
     ]
    }
   ],
   "source": [
    "entry_fees_data = []\n",
    "\n",
    "for park in parks_data['data']:\n",
    "    for fee in park['entranceFees']:\n",
    "        entry_fees_data.append({\n",
    "            'parkCode': park['parkCode'],\n",
    "            'fullName': park['fullName'],\n",
    "            'designation': park['designation'],\n",
    "            'fee_usd': fee['cost'],\n",
    "            'fee_type': fee['title'],\n",
    "            'fee_description': fee['description']\n",
    "        })\n",
    "        \n",
    "pprint.pprint(entry_fees_data[2:4])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Convert `entry_fee_data` to a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>parkCode</th>\n",
       "      <th>fullName</th>\n",
       "      <th>designation</th>\n",
       "      <th>fee_usd</th>\n",
       "      <th>fee_type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>alca</td>\n",
       "      <td>Alcatraz Island</td>\n",
       "      <td></td>\n",
       "      <td>39.9</td>\n",
       "      <td>Adult Day Ticket (Ferry plus audio tour)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>alca</td>\n",
       "      <td>Alcatraz Island</td>\n",
       "      <td></td>\n",
       "      <td>47.3</td>\n",
       "      <td>Adult Night Ticket (Ferry plus audio tour)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>alca</td>\n",
       "      <td>Alcatraz Island</td>\n",
       "      <td></td>\n",
       "      <td>92.3</td>\n",
       "      <td>Adult Behind the Scenes Tour Ticket (Ferry plu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>cabr</td>\n",
       "      <td>Cabrillo National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>20.0</td>\n",
       "      <td>Cabrillo Entrance Fee - Per non commercial veh...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>cabr</td>\n",
       "      <td>Cabrillo National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>15.0</td>\n",
       "      <td>Cabrillo Entrance Fee - per motorcycle</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>cabr</td>\n",
       "      <td>Cabrillo National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Cabrillo Entrance Fee - per walker or bicyclist</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>cabr</td>\n",
       "      <td>Cabrillo National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>30.0</td>\n",
       "      <td>Cabrillo Entrance Fee - per commercical vehicl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>cabr</td>\n",
       "      <td>Cabrillo National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>45.0</td>\n",
       "      <td>Cabrillo Entrance Fee - per commercial vehicle...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>cabr</td>\n",
       "      <td>Cabrillo National Monument</td>\n",
       "      <td>National Monument</td>\n",
       "      <td>100.0</td>\n",
       "      <td>Cabrillo Entrance Fee - per commercial vehicle...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>cali</td>\n",
       "      <td>California National Historic Trail</td>\n",
       "      <td>National Historic Trail</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Entrance fees vary site by site</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  parkCode                            fullName              designation  \\\n",
       "0     alca                     Alcatraz Island                            \n",
       "1     alca                     Alcatraz Island                            \n",
       "2     alca                     Alcatraz Island                            \n",
       "3     cabr          Cabrillo National Monument        National Monument   \n",
       "4     cabr          Cabrillo National Monument        National Monument   \n",
       "5     cabr          Cabrillo National Monument        National Monument   \n",
       "6     cabr          Cabrillo National Monument        National Monument   \n",
       "7     cabr          Cabrillo National Monument        National Monument   \n",
       "8     cabr          Cabrillo National Monument        National Monument   \n",
       "9     cali  California National Historic Trail  National Historic Trail   \n",
       "\n",
       "   fee_usd                                           fee_type  \n",
       "0     39.9           Adult Day Ticket (Ferry plus audio tour)  \n",
       "1     47.3         Adult Night Ticket (Ferry plus audio tour)  \n",
       "2     92.3  Adult Behind the Scenes Tour Ticket (Ferry plu...  \n",
       "3     20.0  Cabrillo Entrance Fee - Per non commercial veh...  \n",
       "4     15.0             Cabrillo Entrance Fee - per motorcycle  \n",
       "5     10.0    Cabrillo Entrance Fee - per walker or bicyclist  \n",
       "6     30.0  Cabrillo Entrance Fee - per commercical vehicl...  \n",
       "7     45.0  Cabrillo Entrance Fee - per commercial vehicle...  \n",
       "8    100.0  Cabrillo Entrance Fee - per commercial vehicle...  \n",
       "9      0.0                    Entrance fees vary site by site  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "entry_fees_df = pd.DataFrame(entry_fees_data)\n",
    "entry_fees_df = entry_fees_df[['parkCode', 'fullName', 'designation', 'fee_usd', 'fee_type']]\n",
    "entry_fees_df['fee_usd'] = entry_fees_df['fee_usd'].astype(float)\n",
    "entry_fees_df.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Export data as a tabular file\n",
    "\n",
    "Pandas DataFrames have a method, `.to_csv()`, that allows them to be exported as a CSV or TSV file. This file can be imported into another program for further analysis.\n",
    "\n",
    "CSV file: `df_name.to_csv('output_file_name.csv', index=False)`\n",
    "\n",
    "TSV file: `df_name.to_csv('output_file_name.tsv', sep='\\t', index=False)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "locations_df.to_csv('parks_data.tsv', sep='\\t', index=False)\n",
    "entry_fees_df.to_csv('parks_entry_fees.tsv', sep='\\t', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional API Resources\n",
    "-------------------------------\n",
    "\n",
    "Full **Requests documentation**: https://2.python-requests.org/en/master\n",
    "\n",
    "**data.gov** is a repository of data produced by US Federal and State Government departments. In addition to data files, data.gov also provides a list of API services for Federal data (https://catalog.data.gov/dataset?res_format=API).\n",
    "\n",
    "**Programmable Web** (https://www.programmableweb.com/) is a website that aggregates information on APIs provided by governmental and private organizations. This can be a good resource for locating APIs of interest. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}