pipeline/3_Setup_SageMaker.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import boto3\n",
    "from datetime import datetime\n",
    "import ipywidgets as widgets\n",
    "from os import path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step X: Configure Amazon Sagemaker and Train Model\n",
    "\n",
    "Amazon Sagemaker is a managed machine learning service which allows us to easily experiment with different configurations of input data without having to worry about the computational infrastructure involved in training a complex model on large volumes of data. We use this tool to produce our trained models. \n",
    "\n",
    "Sagemaker works by ingesting a pre-packaged set of **training and inference code**, along with the necessary environment, and running this package on hardware you specify. This package comes in the form of a Docker container. We have created this container such that it contains all of our custom ML code (you can see the specification in `../pipeline/sagemaker/Dockerfile`), and it lives in the Amazon Elastic Container Registry with the following tag: `675906086666.dkr.ecr.us-west-2.amazonaws.com/planet-snowcover:latest`. This container is not publicly accessible, but is accessible to anyone with AWS credentials. \n",
    "\n",
    "We have to do a bit of configuration, then we can begin training our models!\n",
    "\n",
    "## Configuration \n",
    "### Environment\n",
    "\n",
    "In order to tell the AWS machinery knows who you are, you must specify your AWS profile name, which describes a set of credentials configured via the AWS command line interface. If you've run `aws configure`, you'll have an AWS credentials file containing your credentials, most likely under the `default` profile. Running the following cell will tell you which profile names have associated credentials currently configured on your computer. ⚠️ **Note**: if the following cell has no output, you don't have AWS credentials configured. Go back to the [Deployment Guide](../deployment/README.md) to learn how to do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "default\n",
      "esip\n"
     ]
    }
   ],
   "source": [
    "profiles = boto3.session.Session().available_profiles\n",
    "print('\\n'.join(profiles))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given that information, choose a profile from the dropdown below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "65ff679dd387498caef294f07cc4eb9a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='AWS Profile', options=('default', 'esip'), value='default')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "aws_profile = widgets.Dropdown(options = profiles, value = 'default', description=\"AWS Profile\")\n",
    "aws_profile"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AWS Credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# configure sagemaker session with AWS profile\n",
    "botosess = boto3.Session(profile_name=aws_profile.value, region_name = 'us-west-2') # need us-west-2 to access sagemaker image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ⚠️ Model Specification \n",
    "\n",
    "This is the most important part of the machine learning process. Our algorithm and infrastructure tools rely on configuration files to specify which data to use when training, among other parameters relevant to the machine learning process.\n",
    "\n",
    "**Important**: in order to give our machine learning algorithm data to train with, all data must be stored Amazon S3 buckets that you have access to, in the Spherical Web Mercator tile format. If you've completed the previous two steps in this tutorial, that's going to be the case for you. If you'd like more information on the tools necessary to do this, see the [`preprocess`](../preprocess) toolkit. \n",
    "\n",
    "We've created a configuration file template, which resides at [`/experiments/CONFIG-TEMPLATE.toml`](./experiments/CONFIG-TEMPLATE.toml). **Duplicate this file**, and name it something meaningful. Then, open it in a text editor. \n",
    "\n",
    "The most important part of the file to pay attention to is this: \n",
    "\n",
    "```\n",
    "#@@@@@@@ LOOK HERE! @@@@@@@@\n",
    "[dataset]\n",
    "  ### This defines the IMAGERY and SNOW MASK locations that we need to access to \n",
    "  ### complete training, as well as our AWS credentials and other parameters.\n",
    "  \n",
    "  ## CREDENTIALS\n",
    "  aws_profile = \"esip\" # your aws profile as stored in ~/.aws/credentials. Look there to see your stored profiles. \n",
    "  ## DATA - IMAGERY\n",
    "  image_bucket = \"planet-snowcover-imagery\" # The S3 bucket where your imagery is stored.\n",
    "  # regex defines each slippy-map directory, for buckets with many\n",
    "  imagery_directory_regex = '2018042\\d_.*_tiled' # A Regular Expression to select individual image directories.\n",
    "  \n",
    "  ## DATA – MASK\n",
    "  mask_bucket = \"planet-snowcover-snow/ASO_3M_SD_USCAJW_20180423\"\n",
    "  mask_directory_regex = \"ASO_3M_SD_USCAJW_20180423-MASK_02-COG$\" # $ = end of string = only dirs (no .tif)\n",
    "  \n",
    "  ## ML – CONFIG\n",
    "  train_percent = 0.7 # percentage of imagery used for training (1 - train_percent used for validation). \n",
    "\n",
    "\n",
    "```\n",
    "\n",
    "You want to edit the following lines: \n",
    "\n",
    "| Variable                | Description                                                                                     | Notes                                                                                                                                                           |\n",
    "|-------------------------|-------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------|\n",
    "| `aws_profile`           | The named AWS credentials profile you used in the above step.                                   | Verify that the named profile is in your `~/.aws/credentials` file.                                                                                             |\n",
    "| `image_bucket`          | The named S3 bucket (that you have access to!) where all of the imagery is stored.              | The code requires that this bucket contain sub-directories which each contain a slippy-map directory structure (eg. `<aws_bucket>/<image-id>`/<z>/<x>/<y>.tif`) |\n",
    "| `image_directory_regex` | A regular expression to select which directories within `image_bucket` to select for training.  | For help building this regular expression, check out [RegExr.com](https://regexr.com).                                                                          |\n",
    "| `mask_bucket`           | The named S3 bucket where the masks are stored, similar to above.                               | Same idea to the above imagery bucket structure. This structure allows for multiple images and multiple masks to be considered in training.                     |\n",
    "| `mask_directory_regex`  | See above.                                                                                      | See above.                                                                                                                                                      |\n",
    "\n",
    "    \n",
    "**Note** that this format of regular expressions means that **you can specify multiple masks and images** to a single training run. The ML code automagically checks for overlapping image and mask tiles, and only selects image tiles which have a matching mask tile for ground truth. This means you don't need to be *too* careful how you specify your image paths. However, if you provide too many image tiles for the code to sift through, this matching process will take a long time. \n",
    "    \n",
    "Once you've saved this file somewhere, we'll upload the file to another bucket so the algorithm has access to it. \n",
    "\n",
    "    \n",
    "### Upload Model Specification\n",
    "    \n",
    "Next, you'll need to create an S3 bucket to contain your experimentation. This is where we'll put our model specifications, and optionally our training results. For help, check out [\"*How Do I Create an S3 Bucket?*\"](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-bucket.html). **Be sure to create this bucket in the `us-west-2` region!** Once you've done that, run the following cell and choose the bucket that you'd like to use. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "05579ee6032a441086110efca3335ba8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Dropdown(description='Bucket', options=('planet-snowcover-analysis', 'planet-snowcover-experiments', 'planet-s…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "buckets = [b['Name'] for b in botosess.client('s3').list_buckets()['Buckets']]\n",
    "model_bucket = widgets.Dropdown(description='Bucket', options=buckets)\n",
    "model_bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, specify the **absolute location** of the configuration file you've just created above. For example, if the file is called \"`config1.toml`\" and it's in the `experiments` folder, you might give `/Users/you/planet-snowcover/experiments/config1.toml`.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6609e804683847e8acea57a947a38a25",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Text(value='', description='Config Path')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "config_location = widgets.Text(description=\"Config Path\")\n",
    "config_location"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we upload this file to the specified S3 bucket. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "upload: ../experiments/CONFIG-TEMPLATE.toml to s3://planet-snowcover-experiments/CONFIG-TEMPLATE.toml\n"
     ]
    }
   ],
   "source": [
    "! aws s3 --profile {aws_profile.value} cp {config_location.value} s3://{model_bucket.value}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sagemaker Configuration\n",
    "\n",
    "Now that we've setup the infrastructure necessary to actually train our models, we'll tell Amazon Sagemaker all about our training specifications and preferences and use those to actually train a model. We will use the Python Sagemaker API for this. \n",
    "\n",
    "First, we'll need a piece of information from the Terraform configuration that you've completed in the [Deployment Guide](../deployment/README.md). It's a specific kind of AWS credential known as an \"IAM Role,\" which gives permission to the sagemaker service to access your S3 data and control sagemaker from the API. \n",
    "\n",
    "We've already configured an IAM Role for Sagemaker during the deployment process. To access the role, `cd` into the `/deployment` directory from the command line and run `terraform output`. You should see the following: \n",
    "    \n",
    "    sagemaker_role_arn = arn:aws:iam::.....\n",
    "   \n",
    "Copy this value starting with `arn:aws:iam:` and paste it in the box below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "19bf542e435b4524b9d11dd62dbe8d04",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Text(value='', description='Role')"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sageRole = widgets.Text(description=\"Role\")\n",
    "sageRole"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll configure a SageMaker \"estimator.\" An Estimator is a Python object which links into the Sagemaker training infrastructure. We have to specify the following parameters: \n",
    "\n",
    "| Parameter         | Description                                                                                                 |\n",
    "|-------------------|-------------------------------------------------------------------------------------------------------------|\n",
    "| Image             | The Docker image, stored in Amazon ECS, with all training code and environment within it. (We provide this) |\n",
    "| Role              | The Sagemaker Role ARN that you provided above.                                                             |\n",
    "| Instance Count    | The number of instances you'd like Sagemaker to use (1 is all we can handle at this time).                  |\n",
    "| Instance Type     | The type of AWS EC2 instance we'll use for training (`ml.p2.xlarge` is ideal, since it's GPU-enabled)       |\n",
    "| output_path       | The S3 bucket you'd like to store the output of training (including the trained model) in.                  |\n",
    "| sagemaker_session | The credentialed sagemaker session.                                                                         |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we'll configure the session with the AWS profile specified above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "sess = boto3.Session(profile_name=aws_profile.value, region_name='us-west-2')\n",
    "sage_sess = sagemaker.Session(boto_session=sess)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we specify the image name (we've created this for you---it contains all the code necessary to train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'675906086666.dkr.ecr.us-west-2.amazonaws.com/planet-snowcover:latest'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "image = '675906086666.dkr.ecr.us-west-2.amazonaws.com/planet-snowcover:latest'\n",
    "image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we create the estimator:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "e = sagemaker.estimator.Estimator(image, \n",
    "                             sageRole.value, 1, \"ml.p2.xlarge\", \n",
    "                             output_path = model_bucket.value, \n",
    "                             sagemaker_session = sage_sess)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Model Training\n",
    "\n",
    "The final step in this notebook is to actually train (or \"fit\") the ML model. All of the preparation we've done makes this a one-line operation, but it's worth explaining. \n",
    "\n",
    "The `Estimator` we just created has a `fit` method, which accepts a dictionary of \"input channels\", in Sagemaker parlance. Our Docker container is configured to receive one channel named `config`, which is the S3 location of the configuration file you created earlier. Here we'll create a URL to that location: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://planet-snowcover-experiments/config1.toml'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "config_url = \"s3://{}/{}\".format(model_bucket.value, path.split(config_location.value)[-1])\n",
    "config_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Finally**: we'll train the model. \n",
    "\n",
    "🚨**WARNING** 🚨: Running the next cell **will cost you money!!** To reverse this operation, go to the [AWS Web console](https://us-west-2.console.aws.amazon.com/sagemaker/home?region=us-west-2#/jobs) and stop the training job.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "e.fit({\n",
    "    'config': config_url\n",
    "}, wait=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This operation will take a while (at least **6 hours**, if using the default configuration). To check out the initial progress, run the next cell to get the logs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sage_sess.logs_for_job(e.latest_training_job.job_name, wait=False) # change wait=True to see live updates. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'planet-snowcover-2019-10-24-22-54-59-196'"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "e.latest_training_job.name"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Since this is running on separate SageMaker infrastructure**, you can shut down this notebook/lab instance. Just sit back and have some coffee ☕️. \n",
    "\n",
    "### Appendix: Stop the Job\n",
    "\n",
    "If you'd like to stop the currently-running training job, uncomment and run the cell below. You can also do this in the [AWS Web console](https://us-west-2.console.aws.amazon.com/sagemaker/home?region=us-west-2#/jobs)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#sess.client('sagemaker').stop_training_job(TrainingJobName = e.latest_training_job.name)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Environment (conda_pytorch_p36_2)",
   "language": "python",
   "name": "conda_pytorch_p36_2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}