t81_558_class_06_4_keras_images.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class_06_4_keras_images.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YDTXd8-Lmp8Q"
   },
   "source": [
    "# T81-558: Applications of Deep Neural Networks\n",
    "**Module 6: Convolutional Neural Networks (CNN) for Computer Vision**\n",
    "* Instructor: [Jeff Heaton](https://sites.wustl.edu/jeffheaton/), McKelvey School of Engineering, [Washington University in St. Louis](https://engineering.wustl.edu/Programs/Pages/default.aspx)\n",
    "* For more information visit the [class website](https://sites.wustl.edu/jeffheaton/t81-558/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ncNrAEpzmp8S"
   },
   "source": [
    "# Module 6 Material\n",
    "\n",
    "* Part 6.1: Image Processing in Python [[Video]](https://www.youtube.com/watch?v=4Bh3gqHkIgc&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_1_python_images.ipynb)\n",
    "* Part 6.2: Keras Neural Networks for Digits and Fashion MNIST [[Video]](https://www.youtube.com/watch?v=-SA8BmGvWYE&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_2_cnn.ipynb)\n",
    "* Part 6.3: Implementing a ResNet in Keras** [[Video]](https://www.youtube.com/watch?v=qMFKsMeE6fM&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_3_resnet.ipynb)\n",
    "* **Part 6.4: Using Your Own Images with Keras** [[Video]](https://www.youtube.com/watch?v=VcFja1fUNSk&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_4_keras_images.ipynb)\n",
    "* Part 6.5: Recognizing Multiple Images with YOLO Darknet [[Video]](https://www.youtube.com/watch?v=oQcAKvBFli8&list=PLjy4p-07OYzulelvJ5KVaT2pDlxivl_BN) [[Notebook]](t81_558_class_06_5_yolo.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Google CoLab Instructions\n",
    "\n",
    "The following code ensures that Google CoLab is running the correct version of TensorFlow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "fU9UhAxTmp8S"
   },
   "outputs": [],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    COLAB = True\n",
    "    print(\"Note: using Google CoLab\")\n",
    "except:\n",
    "    print(\"Note: not using Google CoLab\")\n",
    "    COLAB = False\n",
    "\n",
    "# Nicely formatted time string\n",
    "def hms_string(sec_elapsed):\n",
    "    h = int(sec_elapsed / (60 * 60))\n",
    "    m = int((sec_elapsed % (60 * 60)) / 60)\n",
    "    s = sec_elapsed % 60\n",
    "    return f\"{h}:{m:>02}:{s:>05.2f}\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nPa_ojR1mp9j"
   },
   "source": [
    "# Part 6.4: Using Your Own Images with Keras\n",
    "\n",
    "So far we've used image data sets that Keras provides convenience functions for accessing.  There are a number of [built-in data sets](https://www.tensorflow.org/api_docs/python/tf/keras/datasets) for Keras.  While these convenience functions do make it easier to create Keras models for these data sets, these functions also hide the internal workings.  You might be wondering how you would train a neural network from your own sets of images.  \n",
    "\n",
    "Consider the convenience functions provided for CIFAR-10:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bjsQ2JmOmp9j"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import cifar10\n",
    "import numpy as np\n",
    "\n",
    "# Load the CIFAR10 data.\n",
    "(x_train, y_train), (x_test, y_test) = cifar10.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TnTtsJyBmp9p"
   },
   "source": [
    "The above code extracts the training and test sets from CIFAR-10.  Often these datasets are already pre-split between test and training data.  This allows comparison between many researchers who are working on models for this data.  Without these splits, it would be difficult to compare accuracy results between two different researchers that were using two different train/test splits.  Consider the shape of the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 144,
     "status": "ok",
     "timestamp": 1558743283897,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 300
    },
    "id": "cLh4SNjymp9r",
    "outputId": "955d4369-e39b-431f-b032-a34fcbf4878f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50000, 32, 32, 3)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7u5pTMromp9t"
   },
   "source": [
    "We are provided with 50,000 training elements.  Each training element is an image that is 32x32 pixels with 3 color channels.  Typically you will either see 1 color channel (grayscale) or 3 color channels (RGB color). \n",
    "\n",
    "If we look inside of one of the 50,000 elements we can see the structure of each image.  It is a matrix of RGB values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 850
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 134,
     "status": "ok",
     "timestamp": 1558743283898,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 300
    },
    "id": "b8OEUZxKmp9u",
    "outputId": "6584cc79-a917-4e2b-9245-38a06991e970"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[ 59,  62,  63],\n",
       "        [ 43,  46,  45],\n",
       "        [ 50,  48,  43],\n",
       "        ...,\n",
       "        [158, 132, 108],\n",
       "        [152, 125, 102],\n",
       "        [148, 124, 103]],\n",
       "\n",
       "       [[ 16,  20,  20],\n",
       "        [  0,   0,   0],\n",
       "        [ 18,   8,   0],\n",
       "        ...,\n",
       "        [123,  88,  55],\n",
       "        [119,  83,  50],\n",
       "        [122,  87,  57]],\n",
       "\n",
       "       [[ 25,  24,  21],\n",
       "        [ 16,   7,   0],\n",
       "        [ 49,  27,   8],\n",
       "        ...,\n",
       "        [118,  84,  50],\n",
       "        [120,  84,  50],\n",
       "        [109,  73,  42]],\n",
       "\n",
       "       ...,\n",
       "\n",
       "       [[208, 170,  96],\n",
       "        [201, 153,  34],\n",
       "        [198, 161,  26],\n",
       "        ...,\n",
       "        [160, 133,  70],\n",
       "        [ 56,  31,   7],\n",
       "        [ 53,  34,  20]],\n",
       "\n",
       "       [[180, 139,  96],\n",
       "        [173, 123,  42],\n",
       "        [186, 144,  30],\n",
       "        ...,\n",
       "        [184, 148,  94],\n",
       "        [ 97,  62,  34],\n",
       "        [ 83,  53,  34]],\n",
       "\n",
       "       [[177, 144, 116],\n",
       "        [168, 129,  94],\n",
       "        [179, 142,  87],\n",
       "        ...,\n",
       "        [216, 184, 140],\n",
       "        [151, 118,  84],\n",
       "        [123,  92,  72]]], dtype=uint8)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XGn_Jd-Amp9w"
   },
   "source": [
    "It is also important to note that the data type is uint8, which is unsigned integer 8-bits (1 byte).  This corresponds well with image binary data (that is typically 24-bit, 8 bits per 3 color channel = 24 bit).  However, while the images may be 8-bit based, neural networks typically expect floating point input.  Because of this, some transformation/normalization of the data is needed.\n",
    "\n",
    "When training, it is usually necessary to handle multiple images at a time.  The code presented here will load in multiple images and convert them so that they are all the same size.  Processing training data images so that each image is of a uniform height and width is a very common step for computer vision programs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vJSybIFMmp9x"
   },
   "outputs": [],
   "source": [
    "training_data = []\n",
    "\n",
    "%matplotlib inline\n",
    "from PIL import Image, ImageFile\n",
    "from matplotlib.pyplot import imshow\n",
    "import requests\n",
    "import numpy as np\n",
    "from io import BytesIO\n",
    "from IPython.display import display, HTML\n",
    "\n",
    "IMAGE_WIDTH = 200\n",
    "IMAGE_HEIGHT = 200\n",
    "IMAGE_CHANNELS = 3\n",
    "\n",
    "images = [\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/9/92/Brookings.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/f/ff/WashU_Graham_Chapel.JPG\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/9/9e/SeigleHall.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/a/aa/WUSTLKnight.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/3/32/WashUABhall.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/c/c0/Brown_Hall.jpg\",\n",
    "    \"https://upload.wikimedia.org/wikipedia/commons/f/f4/South40.jpg\"    \n",
    "]\n",
    "\n",
    "\n",
    "def make_square(img):\n",
    "    cols,rows = img.size\n",
    "    \n",
    "    if rows>cols:\n",
    "        pad = (rows-cols)/2\n",
    "        img = img.crop((pad,0,cols,cols))\n",
    "    else:\n",
    "        pad = (cols-rows)/2\n",
    "        img = img.crop((0,pad,rows,rows))\n",
    "    \n",
    "    return img\n",
    "        \n",
    "for url in images:\n",
    "    ImageFile.LOAD_TRUNCATED_IMAGES = False\n",
    "    response = requests.get(url)\n",
    "    img = Image.open(BytesIO(response.content))\n",
    "    img.load()\n",
    "    img = make_square(img)\n",
    "    img = img.resize((IMAGE_WIDTH,IMAGE_HEIGHT),Image.ANTIALIAS)\n",
    "    training_data.append(np.asarray(img))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Mn9iXlz5mp91"
   },
   "source": [
    "The above code contains a function called **make_square** that ensures that each of the images have the same height and width.  Square images are particularly easy to deal with because each image will have the same aspect ratio.  There are several techniques that can be used to make an image square.  Fundamentally the image will either be cropped or padded to make it square. Padding adds extra space to the image to cause a square shape.  Cropping removes pixels (and therefore information) from the images to make them square. \n",
    "\n",
    "The technique above crops the images to make them square.  The row and column sizes are analyzed and the image is adjusted based on if the row or column size is smaller.  If there are fewer rows and than columns then the extra columns are dropped to cause the row and column count to be equal. Similarly, if there are fewer columns and than rows then the extra columns are rows to cause the row and column count to be equal. \n",
    "\n",
    "For each image in the set, the image is first adjusted to be square and then resized to the common size that we are forcing all images to be.  The resulting images are each added to a list.  The training data, at this point, is shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 5746
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 115,
     "status": "ok",
     "timestamp": 1558743283904,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 300
    },
    "id": "Zn6KEEAump92",
    "outputId": "d42eff79-0662-46b8-eaa5-5e8b0e5a5e90"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[array([[[112, 161, 203],\n",
       "         [112, 161, 203],\n",
       "         [114, 162, 203],\n",
       "         ...,\n",
       "         [ 86, 121, 162],\n",
       "         [ 49,  85, 130],\n",
       "         [ 76, 113, 154]],\n",
       " \n",
       "        [[106, 158, 203],\n",
       "         [109, 161, 206],\n",
       "         [113, 162, 203],\n",
       "         ...,\n",
       "         [ 75,  95, 125],\n",
       "         [ 67,  87, 124],\n",
       "         [ 24,  38,  73]],\n",
       " \n",
       "        [[108, 159, 205],\n",
       "         [110, 161, 207],\n",
       "         [114, 162, 204],\n",
       "         ...,\n",
       "         [ 82, 108, 136],\n",
       "         [ 50,  80, 115],\n",
       "         [120, 151, 182]],\n",
       " \n",
       "        ...,\n",
       " \n",
       "        [[127,  77,  72],\n",
       "         [125,  80,  69],\n",
       "         [120,  80,  65],\n",
       "         ...,\n",
       "         [ 67,  28,  35],\n",
       "         [ 56,  29,  29],\n",
       "         [ 56,  30,  34]],\n",
       " \n",
       "        [[134,  85,  84],\n",
       "         [130,  88,  78],\n",
       "         [121,  87,  73],\n",
       "         ...,\n",
       "         [ 45,  23,  32],\n",
       "         [ 37,  18,  28],\n",
       "         [ 26,  23,  31]],\n",
       " \n",
       "        [[128,  79,  74],\n",
       "         [127,  84,  77],\n",
       "         [121,  83,  75],\n",
       "         ...,\n",
       "         [ 23,  14,  31],\n",
       "         [ 32,  20,  36],\n",
       "         [ 19,  17,  32]]], dtype=uint8), array([[[140, 132,  11],\n",
       "         [113, 109,   0],\n",
       "         [137, 138,  71],\n",
       "         ...,\n",
       "         [144, 208, 255],\n",
       "         [144, 208, 254],\n",
       "         [144, 209, 254]],\n",
       " \n",
       "        [[168, 160,  12],\n",
       "         [143, 135,  10],\n",
       "         [180, 179, 102],\n",
       "         ...,\n",
       "         [145, 207, 255],\n",
       "         [145, 207, 254],\n",
       "         [145, 209, 254]],\n",
       " \n",
       "        [[141, 136,   3],\n",
       "         [ 94,  98,   3],\n",
       "         [121, 110,  58],\n",
       "         ...,\n",
       "         [146, 208, 254],\n",
       "         [145, 208, 254],\n",
       "         [146, 209, 254]],\n",
       " \n",
       "        ...,\n",
       " \n",
       "        [[ 19,  20,  16],\n",
       "         [ 12,  17,  11],\n",
       "         [ 15,  10,  10],\n",
       "         ...,\n",
       "         [ 56,  39,  22],\n",
       "         [ 42,  28,  15],\n",
       "         [ 52,  28,  16]],\n",
       " \n",
       "        [[ 18,  21,  10],\n",
       "         [ 11,  15,   6],\n",
       "         [ 10,   6,   9],\n",
       "         ...,\n",
       "         [ 58,  35,  18],\n",
       "         [ 50,  25,  16],\n",
       "         [ 54,  30,  18]],\n",
       " \n",
       "        [[ 14,  16,   9],\n",
       "         [ 11,  13,   8],\n",
       "         [  7,   7,   8],\n",
       "         ...,\n",
       "         [ 46,  32,  16],\n",
       "         [ 51,  31,  16],\n",
       "         [ 52,  31,  16]]], dtype=uint8), array([[[107, 137, 231],\n",
       "         [106, 138, 231],\n",
       "         [110, 135, 233],\n",
       "         ...,\n",
       "         [172, 181, 233],\n",
       "         [175, 183, 233],\n",
       "         [175, 182, 233]],\n",
       " \n",
       "        [[110, 134, 233],\n",
       "         [106, 137, 231],\n",
       "         [108, 137, 231],\n",
       "         ...,\n",
       "         [173, 183, 233],\n",
       "         [175, 185, 233],\n",
       "         [176, 186, 233]],\n",
       " \n",
       "        [[107, 136, 233],\n",
       "         [106, 138, 231],\n",
       "         [106, 136, 232],\n",
       "         ...,\n",
       "         [178, 183, 233],\n",
       "         [178, 185, 233],\n",
       "         [177, 185, 233]],\n",
       " \n",
       "        ...,\n",
       " \n",
       "        [[ 93,  84,  14],\n",
       "         [102,  92,  20],\n",
       "         [ 82,  81,  13],\n",
       "         ...,\n",
       "         [133, 110,  32],\n",
       "         [136, 111,  33],\n",
       "         [136, 115,  34]],\n",
       " \n",
       "        [[ 99,  89,  16],\n",
       "         [102,  88,  15],\n",
       "         [ 90,  84,  15],\n",
       "         ...,\n",
       "         [136, 109,  30],\n",
       "         [144, 114,  42],\n",
       "         [156, 118,  55]],\n",
       " \n",
       "        [[ 87,  81,  17],\n",
       "         [ 97,  87,  19],\n",
       "         [ 93,  88,  18],\n",
       "         ...,\n",
       "         [133, 108,  28],\n",
       "         [141, 105,  38],\n",
       "         [146, 106,  42]]], dtype=uint8), array([[[ 76, 139, 212],\n",
       "         [ 77, 138, 213],\n",
       "         [ 77, 138, 213],\n",
       "         ...,\n",
       "         [ 61, 125, 202],\n",
       "         [ 62, 126, 202],\n",
       "         [ 60, 125, 202]],\n",
       " \n",
       "        [[ 77, 139, 213],\n",
       "         [ 75, 139, 212],\n",
       "         [ 79, 139, 215],\n",
       "         ...,\n",
       "         [ 59, 125, 204],\n",
       "         [ 61, 125, 202],\n",
       "         [ 60, 127, 203]],\n",
       " \n",
       "        [[ 78, 140, 213],\n",
       "         [ 75, 141, 212],\n",
       "         [ 77, 140, 214],\n",
       "         ...,\n",
       "         [ 61, 125, 206],\n",
       "         [ 61, 124, 207],\n",
       "         [ 61, 126, 205]],\n",
       " \n",
       "        ...,\n",
       " \n",
       "        [[ 59,  53,  45],\n",
       "         [ 46,  41,  34],\n",
       "         [ 64,  53,  45],\n",
       "         ...,\n",
       "         [ 57,  68,  54],\n",
       "         [ 73,  89,  71],\n",
       "         [103, 115,  94]],\n",
       " \n",
       "        [[ 72,  61,  51],\n",
       "         [ 81,  86,  75],\n",
       "         [ 81,  81,  75],\n",
       "         ...,\n",
       "         [104,  94,  91],\n",
       "         [136, 118, 108],\n",
       "         [137, 112,  99]],\n",
       " \n",
       "        [[ 75,  61,  54],\n",
       "         [ 99,  86,  87],\n",
       "         [ 90,  84,  88],\n",
       "         ...,\n",
       "         [ 58,  56,  45],\n",
       "         [ 65,  62,  48],\n",
       "         [102,  95,  79]]], dtype=uint8), array([[[164, 177, 229],\n",
       "         [165, 177, 229],\n",
       "         [167, 180, 226],\n",
       "         ...,\n",
       "         [147, 165, 233],\n",
       "         [141, 167, 232],\n",
       "         [141, 167, 233]],\n",
       " \n",
       "        [[167, 179, 226],\n",
       "         [165, 176, 230],\n",
       "         [167, 176, 231],\n",
       "         ...,\n",
       "         [147, 168, 233],\n",
       "         [146, 168, 233],\n",
       "         [144, 166, 233]],\n",
       " \n",
       "        [[169, 176, 229],\n",
       "         [169, 176, 230],\n",
       "         [167, 176, 228],\n",
       "         ...,\n",
       "         [154, 173, 231],\n",
       "         [155, 171, 231],\n",
       "         [149, 167, 233]],\n",
       " \n",
       "        ...,\n",
       " \n",
       "        [[105,  94,  23],\n",
       "         [115,  97,  31],\n",
       "         [106,  95,  32],\n",
       "         ...,\n",
       "         [108,  97,  26],\n",
       "         [101,  97,  23],\n",
       "         [103,  93,  21]],\n",
       " \n",
       "        [[105,  93,  24],\n",
       "         [118,  96,  30],\n",
       "         [112,  95,  30],\n",
       "         ...,\n",
       "         [106,  97,  22],\n",
       "         [109,  99,  26],\n",
       "         [101,  96,  21]],\n",
       " \n",
       "        [[102,  93,  29],\n",
       "         [107,  95,  29],\n",
       "         [103,  95,  25],\n",
       "         ...,\n",
       "         [ 90,  86,  20],\n",
       "         [101,  93,  24],\n",
       "         [ 98,  90,  20]]], dtype=uint8), array([[[254, 254, 254],\n",
       "         [254, 254, 254],\n",
       "         [254, 254, 254],\n",
       "         ...,\n",
       "         [ 26,  23,  21],\n",
       "         [ 32,  33,  32],\n",
       "         [ 45,  45,  45]],\n",
       " \n",
       "        [[254, 254, 254],\n",
       "         [254, 254, 254],\n",
       "         [254, 254, 254],\n",
       "         ...,\n",
       "         [ 33,  26,  23],\n",
       "         [ 25,  25,  23],\n",
       "         [ 37,  35,  30]],\n",
       " \n",
       "        [[254, 254, 254],\n",
       "         [254, 254, 254],\n",
       "         [254, 254, 254],\n",
       "         ...,\n",
       "         [ 30,  27,  21],\n",
       "         [ 29,  25,  25],\n",
       "         [ 34,  28,  23]],\n",
       " \n",
       "        ...,\n",
       " \n",
       "        [[ 26,  26,  25],\n",
       "         [ 24,  18,  17],\n",
       "         [ 40,  31,  31],\n",
       "         ...,\n",
       "         [108, 105, 102],\n",
       "         [105, 104, 103],\n",
       "         [109, 105, 105]],\n",
       " \n",
       "        [[ 32,  29,  28],\n",
       "         [ 24,  22,  16],\n",
       "         [ 32,  27,  25],\n",
       "         ...,\n",
       "         [112, 106, 109],\n",
       "         [106, 105, 107],\n",
       "         [107, 105, 103]],\n",
       " \n",
       "        [[ 28,  25,  20],\n",
       "         [ 29,  21,  18],\n",
       "         [ 50,  36,  34],\n",
       "         ...,\n",
       "         [103,  99,  99],\n",
       "         [104, 100,  99],\n",
       "         [106, 100, 101]]], dtype=uint8), array([[[ 27,  79, 195],\n",
       "         [ 26,  78, 197],\n",
       "         [ 27,  77, 197],\n",
       "         ...,\n",
       "         [ 30,  89, 205],\n",
       "         [ 29,  86, 204],\n",
       "         [ 29,  88, 205]],\n",
       " \n",
       "        [[ 26,  78, 197],\n",
       "         [ 29,  78, 195],\n",
       "         [ 28,  78, 197],\n",
       "         ...,\n",
       "         [ 29,  90, 205],\n",
       "         [ 30,  85, 206],\n",
       "         [ 30,  87, 205]],\n",
       " \n",
       "        [[ 27,  82, 197],\n",
       "         [ 28,  82, 196],\n",
       "         [ 27,  82, 199],\n",
       "         ...,\n",
       "         [ 29,  89, 206],\n",
       "         [ 30,  89, 206],\n",
       "         [ 30,  89, 206]],\n",
       " \n",
       "        ...,\n",
       " \n",
       "        [[186, 125,  60],\n",
       "         [184, 122,  60],\n",
       "         [185, 123,  61],\n",
       "         ...,\n",
       "         [211, 160,  89],\n",
       "         [213, 162,  94],\n",
       "         [215, 158,  86]],\n",
       " \n",
       "        [[183, 124,  63],\n",
       "         [183, 123,  59],\n",
       "         [188, 122,  56],\n",
       "         ...,\n",
       "         [214, 154,  84],\n",
       "         [213, 155,  86],\n",
       "         [213, 158,  87]],\n",
       " \n",
       "        [[185, 122,  59],\n",
       "         [182, 120,  57],\n",
       "         [178, 119,  59],\n",
       "         ...,\n",
       "         [207, 151,  82],\n",
       "         [209, 152,  78],\n",
       "         [211, 153,  84]]], dtype=uint8)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "A3xREkQCmp95"
   },
   "source": [
    "We now have a 1D list of 3D images (height by width by color depth).  We would like this to be a 4D Numpy array.  Passing the list to **np.array** performs this conversion/reshaping.\n",
    "\n",
    "The training data is divided by 127.5 and subtracted by one to normalize to between -1 and 1.  This causes the RGB values to be centered around zero and gives greater predictive power to the neural network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "QlMIM2c3mp95"
   },
   "outputs": [],
   "source": [
    "training_data = np.array(training_data) / 127.5 - 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "HvAOVoS9mp97"
   },
   "source": [
    "We can display the normalized training data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 5236
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 91,
     "status": "ok",
     "timestamp": 1558743283909,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 300
    },
    "id": "PixWCqjhmp98",
    "outputId": "c56b3ab2-e14f-41c8-df78-7006e65caeca"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[[[-0.12156863,  0.2627451 ,  0.59215686],\n",
       "         [-0.12156863,  0.2627451 ,  0.59215686],\n",
       "         [-0.10588235,  0.27058824,  0.59215686],\n",
       "         ...,\n",
       "         [-0.3254902 , -0.05098039,  0.27058824],\n",
       "         [-0.61568627, -0.33333333,  0.01960784],\n",
       "         [-0.40392157, -0.11372549,  0.20784314]],\n",
       "\n",
       "        [[-0.16862745,  0.23921569,  0.59215686],\n",
       "         [-0.14509804,  0.2627451 ,  0.61568627],\n",
       "         [-0.11372549,  0.27058824,  0.59215686],\n",
       "         ...,\n",
       "         [-0.41176471, -0.25490196, -0.01960784],\n",
       "         [-0.4745098 , -0.31764706, -0.02745098],\n",
       "         [-0.81176471, -0.70196078, -0.42745098]],\n",
       "\n",
       "        [[-0.15294118,  0.24705882,  0.60784314],\n",
       "         [-0.1372549 ,  0.2627451 ,  0.62352941],\n",
       "         [-0.10588235,  0.27058824,  0.6       ],\n",
       "         ...,\n",
       "         [-0.35686275, -0.15294118,  0.06666667],\n",
       "         [-0.60784314, -0.37254902, -0.09803922],\n",
       "         [-0.05882353,  0.18431373,  0.42745098]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[-0.00392157, -0.39607843, -0.43529412],\n",
       "         [-0.01960784, -0.37254902, -0.45882353],\n",
       "         [-0.05882353, -0.37254902, -0.49019608],\n",
       "         ...,\n",
       "         [-0.4745098 , -0.78039216, -0.7254902 ],\n",
       "         [-0.56078431, -0.77254902, -0.77254902],\n",
       "         [-0.56078431, -0.76470588, -0.73333333]],\n",
       "\n",
       "        [[ 0.05098039, -0.33333333, -0.34117647],\n",
       "         [ 0.01960784, -0.30980392, -0.38823529],\n",
       "         [-0.05098039, -0.31764706, -0.42745098],\n",
       "         ...,\n",
       "         [-0.64705882, -0.81960784, -0.74901961],\n",
       "         [-0.70980392, -0.85882353, -0.78039216],\n",
       "         [-0.79607843, -0.81960784, -0.75686275]],\n",
       "\n",
       "        [[ 0.00392157, -0.38039216, -0.41960784],\n",
       "         [-0.00392157, -0.34117647, -0.39607843],\n",
       "         [-0.05098039, -0.34901961, -0.41176471],\n",
       "         ...,\n",
       "         [-0.81960784, -0.89019608, -0.75686275],\n",
       "         [-0.74901961, -0.84313725, -0.71764706],\n",
       "         [-0.85098039, -0.86666667, -0.74901961]]],\n",
       "\n",
       "\n",
       "       [[[ 0.09803922,  0.03529412, -0.91372549],\n",
       "         [-0.11372549, -0.14509804, -1.        ],\n",
       "         [ 0.0745098 ,  0.08235294, -0.44313725],\n",
       "         ...,\n",
       "         [ 0.12941176,  0.63137255,  1.        ],\n",
       "         [ 0.12941176,  0.63137255,  0.99215686],\n",
       "         [ 0.12941176,  0.63921569,  0.99215686]],\n",
       "\n",
       "        [[ 0.31764706,  0.25490196, -0.90588235],\n",
       "         [ 0.12156863,  0.05882353, -0.92156863],\n",
       "         [ 0.41176471,  0.40392157, -0.2       ],\n",
       "         ...,\n",
       "         [ 0.1372549 ,  0.62352941,  1.        ],\n",
       "         [ 0.1372549 ,  0.62352941,  0.99215686],\n",
       "         [ 0.1372549 ,  0.63921569,  0.99215686]],\n",
       "\n",
       "        [[ 0.10588235,  0.06666667, -0.97647059],\n",
       "         [-0.2627451 , -0.23137255, -0.97647059],\n",
       "         [-0.05098039, -0.1372549 , -0.54509804],\n",
       "         ...,\n",
       "         [ 0.14509804,  0.63137255,  0.99215686],\n",
       "         [ 0.1372549 ,  0.63137255,  0.99215686],\n",
       "         [ 0.14509804,  0.63921569,  0.99215686]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[-0.85098039, -0.84313725, -0.8745098 ],\n",
       "         [-0.90588235, -0.86666667, -0.91372549],\n",
       "         [-0.88235294, -0.92156863, -0.92156863],\n",
       "         ...,\n",
       "         [-0.56078431, -0.69411765, -0.82745098],\n",
       "         [-0.67058824, -0.78039216, -0.88235294],\n",
       "         [-0.59215686, -0.78039216, -0.8745098 ]],\n",
       "\n",
       "        [[-0.85882353, -0.83529412, -0.92156863],\n",
       "         [-0.91372549, -0.88235294, -0.95294118],\n",
       "         [-0.92156863, -0.95294118, -0.92941176],\n",
       "         ...,\n",
       "         [-0.54509804, -0.7254902 , -0.85882353],\n",
       "         [-0.60784314, -0.80392157, -0.8745098 ],\n",
       "         [-0.57647059, -0.76470588, -0.85882353]],\n",
       "\n",
       "        [[-0.89019608, -0.8745098 , -0.92941176],\n",
       "         [-0.91372549, -0.89803922, -0.9372549 ],\n",
       "         [-0.94509804, -0.94509804, -0.9372549 ],\n",
       "         ...,\n",
       "         [-0.63921569, -0.74901961, -0.8745098 ],\n",
       "         [-0.6       , -0.75686275, -0.8745098 ],\n",
       "         [-0.59215686, -0.75686275, -0.8745098 ]]],\n",
       "\n",
       "\n",
       "       [[[-0.16078431,  0.0745098 ,  0.81176471],\n",
       "         [-0.16862745,  0.08235294,  0.81176471],\n",
       "         [-0.1372549 ,  0.05882353,  0.82745098],\n",
       "         ...,\n",
       "         [ 0.34901961,  0.41960784,  0.82745098],\n",
       "         [ 0.37254902,  0.43529412,  0.82745098],\n",
       "         [ 0.37254902,  0.42745098,  0.82745098]],\n",
       "\n",
       "        [[-0.1372549 ,  0.05098039,  0.82745098],\n",
       "         [-0.16862745,  0.0745098 ,  0.81176471],\n",
       "         [-0.15294118,  0.0745098 ,  0.81176471],\n",
       "         ...,\n",
       "         [ 0.35686275,  0.43529412,  0.82745098],\n",
       "         [ 0.37254902,  0.45098039,  0.82745098],\n",
       "         [ 0.38039216,  0.45882353,  0.82745098]],\n",
       "\n",
       "        [[-0.16078431,  0.06666667,  0.82745098],\n",
       "         [-0.16862745,  0.08235294,  0.81176471],\n",
       "         [-0.16862745,  0.06666667,  0.81960784],\n",
       "         ...,\n",
       "         [ 0.39607843,  0.43529412,  0.82745098],\n",
       "         [ 0.39607843,  0.45098039,  0.82745098],\n",
       "         [ 0.38823529,  0.45098039,  0.82745098]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[-0.27058824, -0.34117647, -0.89019608],\n",
       "         [-0.2       , -0.27843137, -0.84313725],\n",
       "         [-0.35686275, -0.36470588, -0.89803922],\n",
       "         ...,\n",
       "         [ 0.04313725, -0.1372549 , -0.74901961],\n",
       "         [ 0.06666667, -0.12941176, -0.74117647],\n",
       "         [ 0.06666667, -0.09803922, -0.73333333]],\n",
       "\n",
       "        [[-0.22352941, -0.30196078, -0.8745098 ],\n",
       "         [-0.2       , -0.30980392, -0.88235294],\n",
       "         [-0.29411765, -0.34117647, -0.88235294],\n",
       "         ...,\n",
       "         [ 0.06666667, -0.14509804, -0.76470588],\n",
       "         [ 0.12941176, -0.10588235, -0.67058824],\n",
       "         [ 0.22352941, -0.0745098 , -0.56862745]],\n",
       "\n",
       "        [[-0.31764706, -0.36470588, -0.86666667],\n",
       "         [-0.23921569, -0.31764706, -0.85098039],\n",
       "         [-0.27058824, -0.30980392, -0.85882353],\n",
       "         ...,\n",
       "         [ 0.04313725, -0.15294118, -0.78039216],\n",
       "         [ 0.10588235, -0.17647059, -0.70196078],\n",
       "         [ 0.14509804, -0.16862745, -0.67058824]]],\n",
       "\n",
       "\n",
       "       ...,\n",
       "\n",
       "\n",
       "       [[[ 0.28627451,  0.38823529,  0.79607843],\n",
       "         [ 0.29411765,  0.38823529,  0.79607843],\n",
       "         [ 0.30980392,  0.41176471,  0.77254902],\n",
       "         ...,\n",
       "         [ 0.15294118,  0.29411765,  0.82745098],\n",
       "         [ 0.10588235,  0.30980392,  0.81960784],\n",
       "         [ 0.10588235,  0.30980392,  0.82745098]],\n",
       "\n",
       "        [[ 0.30980392,  0.40392157,  0.77254902],\n",
       "         [ 0.29411765,  0.38039216,  0.80392157],\n",
       "         [ 0.30980392,  0.38039216,  0.81176471],\n",
       "         ...,\n",
       "         [ 0.15294118,  0.31764706,  0.82745098],\n",
       "         [ 0.14509804,  0.31764706,  0.82745098],\n",
       "         [ 0.12941176,  0.30196078,  0.82745098]],\n",
       "\n",
       "        [[ 0.3254902 ,  0.38039216,  0.79607843],\n",
       "         [ 0.3254902 ,  0.38039216,  0.80392157],\n",
       "         [ 0.30980392,  0.38039216,  0.78823529],\n",
       "         ...,\n",
       "         [ 0.20784314,  0.35686275,  0.81176471],\n",
       "         [ 0.21568627,  0.34117647,  0.81176471],\n",
       "         [ 0.16862745,  0.30980392,  0.82745098]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[-0.17647059, -0.2627451 , -0.81960784],\n",
       "         [-0.09803922, -0.23921569, -0.75686275],\n",
       "         [-0.16862745, -0.25490196, -0.74901961],\n",
       "         ...,\n",
       "         [-0.15294118, -0.23921569, -0.79607843],\n",
       "         [-0.20784314, -0.23921569, -0.81960784],\n",
       "         [-0.19215686, -0.27058824, -0.83529412]],\n",
       "\n",
       "        [[-0.17647059, -0.27058824, -0.81176471],\n",
       "         [-0.0745098 , -0.24705882, -0.76470588],\n",
       "         [-0.12156863, -0.25490196, -0.76470588],\n",
       "         ...,\n",
       "         [-0.16862745, -0.23921569, -0.82745098],\n",
       "         [-0.14509804, -0.22352941, -0.79607843],\n",
       "         [-0.20784314, -0.24705882, -0.83529412]],\n",
       "\n",
       "        [[-0.2       , -0.27058824, -0.77254902],\n",
       "         [-0.16078431, -0.25490196, -0.77254902],\n",
       "         [-0.19215686, -0.25490196, -0.80392157],\n",
       "         ...,\n",
       "         [-0.29411765, -0.3254902 , -0.84313725],\n",
       "         [-0.20784314, -0.27058824, -0.81176471],\n",
       "         [-0.23137255, -0.29411765, -0.84313725]]],\n",
       "\n",
       "\n",
       "       [[[ 0.99215686,  0.99215686,  0.99215686],\n",
       "         [ 0.99215686,  0.99215686,  0.99215686],\n",
       "         [ 0.99215686,  0.99215686,  0.99215686],\n",
       "         ...,\n",
       "         [-0.79607843, -0.81960784, -0.83529412],\n",
       "         [-0.74901961, -0.74117647, -0.74901961],\n",
       "         [-0.64705882, -0.64705882, -0.64705882]],\n",
       "\n",
       "        [[ 0.99215686,  0.99215686,  0.99215686],\n",
       "         [ 0.99215686,  0.99215686,  0.99215686],\n",
       "         [ 0.99215686,  0.99215686,  0.99215686],\n",
       "         ...,\n",
       "         [-0.74117647, -0.79607843, -0.81960784],\n",
       "         [-0.80392157, -0.80392157, -0.81960784],\n",
       "         [-0.70980392, -0.7254902 , -0.76470588]],\n",
       "\n",
       "        [[ 0.99215686,  0.99215686,  0.99215686],\n",
       "         [ 0.99215686,  0.99215686,  0.99215686],\n",
       "         [ 0.99215686,  0.99215686,  0.99215686],\n",
       "         ...,\n",
       "         [-0.76470588, -0.78823529, -0.83529412],\n",
       "         [-0.77254902, -0.80392157, -0.80392157],\n",
       "         [-0.73333333, -0.78039216, -0.81960784]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[-0.79607843, -0.79607843, -0.80392157],\n",
       "         [-0.81176471, -0.85882353, -0.86666667],\n",
       "         [-0.68627451, -0.75686275, -0.75686275],\n",
       "         ...,\n",
       "         [-0.15294118, -0.17647059, -0.2       ],\n",
       "         [-0.17647059, -0.18431373, -0.19215686],\n",
       "         [-0.14509804, -0.17647059, -0.17647059]],\n",
       "\n",
       "        [[-0.74901961, -0.77254902, -0.78039216],\n",
       "         [-0.81176471, -0.82745098, -0.8745098 ],\n",
       "         [-0.74901961, -0.78823529, -0.80392157],\n",
       "         ...,\n",
       "         [-0.12156863, -0.16862745, -0.14509804],\n",
       "         [-0.16862745, -0.17647059, -0.16078431],\n",
       "         [-0.16078431, -0.17647059, -0.19215686]],\n",
       "\n",
       "        [[-0.78039216, -0.80392157, -0.84313725],\n",
       "         [-0.77254902, -0.83529412, -0.85882353],\n",
       "         [-0.60784314, -0.71764706, -0.73333333],\n",
       "         ...,\n",
       "         [-0.19215686, -0.22352941, -0.22352941],\n",
       "         [-0.18431373, -0.21568627, -0.22352941],\n",
       "         [-0.16862745, -0.21568627, -0.20784314]]],\n",
       "\n",
       "\n",
       "       [[[-0.78823529, -0.38039216,  0.52941176],\n",
       "         [-0.79607843, -0.38823529,  0.54509804],\n",
       "         [-0.78823529, -0.39607843,  0.54509804],\n",
       "         ...,\n",
       "         [-0.76470588, -0.30196078,  0.60784314],\n",
       "         [-0.77254902, -0.3254902 ,  0.6       ],\n",
       "         [-0.77254902, -0.30980392,  0.60784314]],\n",
       "\n",
       "        [[-0.79607843, -0.38823529,  0.54509804],\n",
       "         [-0.77254902, -0.38823529,  0.52941176],\n",
       "         [-0.78039216, -0.38823529,  0.54509804],\n",
       "         ...,\n",
       "         [-0.77254902, -0.29411765,  0.60784314],\n",
       "         [-0.76470588, -0.33333333,  0.61568627],\n",
       "         [-0.76470588, -0.31764706,  0.60784314]],\n",
       "\n",
       "        [[-0.78823529, -0.35686275,  0.54509804],\n",
       "         [-0.78039216, -0.35686275,  0.5372549 ],\n",
       "         [-0.78823529, -0.35686275,  0.56078431],\n",
       "         ...,\n",
       "         [-0.77254902, -0.30196078,  0.61568627],\n",
       "         [-0.76470588, -0.30196078,  0.61568627],\n",
       "         [-0.76470588, -0.30196078,  0.61568627]],\n",
       "\n",
       "        ...,\n",
       "\n",
       "        [[ 0.45882353, -0.01960784, -0.52941176],\n",
       "         [ 0.44313725, -0.04313725, -0.52941176],\n",
       "         [ 0.45098039, -0.03529412, -0.52156863],\n",
       "         ...,\n",
       "         [ 0.65490196,  0.25490196, -0.30196078],\n",
       "         [ 0.67058824,  0.27058824, -0.2627451 ],\n",
       "         [ 0.68627451,  0.23921569, -0.3254902 ]],\n",
       "\n",
       "        [[ 0.43529412, -0.02745098, -0.50588235],\n",
       "         [ 0.43529412, -0.03529412, -0.5372549 ],\n",
       "         [ 0.4745098 , -0.04313725, -0.56078431],\n",
       "         ...,\n",
       "         [ 0.67843137,  0.20784314, -0.34117647],\n",
       "         [ 0.67058824,  0.21568627, -0.3254902 ],\n",
       "         [ 0.67058824,  0.23921569, -0.31764706]],\n",
       "\n",
       "        [[ 0.45098039, -0.04313725, -0.5372549 ],\n",
       "         [ 0.42745098, -0.05882353, -0.55294118],\n",
       "         [ 0.39607843, -0.06666667, -0.5372549 ],\n",
       "         ...,\n",
       "         [ 0.62352941,  0.18431373, -0.35686275],\n",
       "         [ 0.63921569,  0.19215686, -0.38823529],\n",
       "         [ 0.65490196,  0.2       , -0.34117647]]]])"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZHMjRVAbmp9_"
   },
   "source": [
    "It is sometimes useful to to save a training set.  For image and higher dimensional data, as CSV file is not sufficient. Also, Pickle can experience problems with very large datasets.  Because of this I prefer to use Numpy's own format for binary data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 80,
     "status": "ok",
     "timestamp": 1558743283912,
     "user": {
      "displayName": "",
      "photoUrl": "",
      "userId": ""
     },
     "user_tz": 300
    },
    "id": "eemePDLwmp-A",
    "outputId": "99cdf73b-423a-44a9-f509-c4a1fdf4ba18"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving training image binary...\n",
      "Done.\n"
     ]
    }
   ],
   "source": [
    "print(\"Saving training image binary...\")\n",
    "np.save(\"training\",training_data) # Saves as \"training.npy\"\n",
    "print(\"Done.\")"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "4aQApI1jmp8o",
    "pAf59S0Wmp82",
    "6JN_aTuvmp86",
    "BqWngCONmp8-",
    "TRXv8k0Ymp9B",
    "ptDi0ZiQmp9J",
    "FDk1ekrxmp9X",
    "SzNJPXncmp9a"
   ],
   "name": "Copy of Copy of t81_558_class06_backpropagation.ipynb",
   "provenance": [
    {
     "file_id": "https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class06_cnn.ipynb",
     "timestamp": 1558743458994
    },
    {
     "file_id": "https://github.com/jeffheaton/t81_558_deep_learning/blob/master/t81_558_class06_backpropagation.ipynb",
     "timestamp": 1558698961697
    }
   ],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3.6 (yolo)",
   "language": "python",
   "name": "yolo"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}