skopt_workshop_part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gc9IfCbORvux"
   },
   "source": [
    "# Bayesian Optimization Workshop: <BR>from hyperparameters optimization to Neural architecture search (NAS) \n",
    "\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lmassaron/kaggledays-2019-gbdt/blob/master/skopt_workshop_part2.ipynb)\n",
    "\n",
    "\n",
    "## Instructors\n",
    "* Luca Massaron [@lmassaron](https://www.linkedin.com/in/lmassaron/) - Data Scientist / Author / Google Developer Expert in Machine Learning\n",
    "\n",
    "## About the workshop\n",
    "In this workshop we demonstrate how to use different optimization approaches based on [Scikit-Optimize](https://github.com/scikit-optimize/scikit-optimize), a library built on top of NumPy, SciPy and Scikit-Learn, and we present an easy and fast approach to set them ready and usable.\n",
    "\n",
    "In the first part, we start from trying to optimize Gradient Boosting Decision Trees (GBDT), which presently represent the state of the art for building predictors for flat table data. However, GBDT seldom perform the best out-of-the-box (using default values) because of the many hyper-parameters to tune. Especially in the most recent GBDT implementations, such as LightGBM, the over-sophistication of hyper-parameters renders finding the optimal settings by hand or simple grid search difficult because of high combinatorial complexity and long running times for experiments.\n",
    "\n",
    "In the second part, we try to leverage what we have learned so far and challenge the Neural Architecture Search (NAS) problem.\n",
    "\n",
    "## Prerequisites\n",
    "You should be aware of the role and importance of hyper-parameter optimization in machine learning.  \n",
    "\n",
    "\n",
    "## Dealing with the Tutorial Material\n",
    "In order to make the workshop easily accessible, we are prepared cloud access:\n",
    "* Using [Google Colab](https://colab.research.google.com/github/lmassaron/kaggledays-2019-gbdt/blob/master/skopt_workshop.ipynb)\n",
    "\n",
    "## References\n",
    "* [Random Optimization](https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf) (BERGSTRA, James; BENGIO, Yoshua. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 2012, 13.Feb: 281-305.) \n",
    "* [Bayesian Optimization](https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf) (SNOEK, Jasper; LAROCHELLE, Hugo; ADAMS, Ryan P. Practical bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems. 2012. p. 2951-2959)\n",
    "\n",
    "For a brief introduction about the key models we will be using during this works we suggest consulting: BOSCHETTI, Alberto; MASSARON, Luca. Python data science essentials. Packt Publishing Ltd, 3rd ed., 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9XoG1kDoRvxB"
   },
   "source": [
    "# PART II : towards an effective usage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 241
    },
    "colab_type": "code",
    "id": "YNp6uFiBpTWP",
    "outputId": "c694a908-4edb-4158-9018-7ead1c7e3a6b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/scikit-optimize/scikit-optimize.git\n",
      "  Cloning https://github.com/scikit-optimize/scikit-optimize.git to /tmp/pip-req-build-f8qdfbu0\n",
      "Requirement already satisfied (use --upgrade to upgrade): scikit-optimize==0.5.2+49.g074ce8e from git+https://github.com/scikit-optimize/scikit-optimize.git in /usr/local/lib/python3.6/dist-packages\n",
      "Requirement already satisfied: pyaml in /usr/local/lib/python3.6/dist-packages (from scikit-optimize==0.5.2+49.g074ce8e) (18.11.0)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from scikit-optimize==0.5.2+49.g074ce8e) (1.16.2)\n",
      "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from scikit-optimize==0.5.2+49.g074ce8e) (1.2.1)\n",
      "Requirement already satisfied: scikit-learn>=0.19.1 in /usr/local/lib/python3.6/dist-packages (from scikit-optimize==0.5.2+49.g074ce8e) (0.20.3)\n",
      "Requirement already satisfied: PyYAML in /usr/local/lib/python3.6/dist-packages (from pyaml->scikit-optimize==0.5.2+49.g074ce8e) (3.13)\n",
      "Building wheels for collected packages: scikit-optimize\n",
      "  Building wheel for scikit-optimize (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /tmp/pip-ephem-wheel-cache-brqmbtbb/wheels/11/6f/86/2b772172db85ad0b4487d67e325e535ee8e7782b2a1dfcadf5\n",
      "Successfully built scikit-optimize\n"
     ]
    }
   ],
   "source": [
    "# Installing the most recent version of skopt directly from Github\n",
    "!pip install git+https://github.com/scikit-optimize/scikit-optimize.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 156
    },
    "colab_type": "code",
    "id": "0KzRjOc5pTjl",
    "outputId": "66357605-45b3-4ef4-f626-c0a6b2324f0e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: catboost in /usr/local/lib/python3.6/dist-packages (0.14.1)\n",
      "Requirement already satisfied, skipping upgrade: pandas>=0.19.1 in /usr/local/lib/python3.6/dist-packages (from catboost) (0.23.4)\n",
      "Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from catboost) (1.11.0)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.11.1 in /usr/local/lib/python3.6/dist-packages (from catboost) (1.16.2)\n",
      "Requirement already satisfied, skipping upgrade: enum34 in /usr/local/lib/python3.6/dist-packages (from catboost) (1.1.6)\n",
      "Requirement already satisfied, skipping upgrade: pytz>=2011k in /usr/local/lib/python3.6/dist-packages (from pandas>=0.19.1->catboost) (2018.9)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2.5.0 in /usr/local/lib/python3.6/dist-packages (from pandas>=0.19.1->catboost) (2.5.3)\n"
     ]
    }
   ],
   "source": [
    "# Assuring you have the most recent CatBoost release\n",
    "!pip install catboost -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SZn4vmNYpTua"
   },
   "outputs": [],
   "source": [
    "# Importing core libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from time import time\n",
    "import pprint\n",
    "import joblib\n",
    "\n",
    "# Suppressing warnings because of skopt verbosity\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "# Our example dataset\n",
    "from sklearn.datasets import load_boston\n",
    "\n",
    "# Classifiers\n",
    "from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "from catboost import CatBoostClassifier\n",
    "\n",
    "# Hyperparameters distributions\n",
    "from scipy.stats import randint\n",
    "from scipy.stats import uniform\n",
    "\n",
    "# Model selection\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Metrics\n",
    "from sklearn.metrics import average_precision_score\n",
    "from sklearn.metrics import make_scorer\n",
    "\n",
    "# Plotting\n",
    "from matplotlib import pyplot as plt\n",
    "import seaborn\n",
    "%matplotlib inline\n",
    "\n",
    "# Skopt functions\n",
    "from skopt import BayesSearchCV\n",
    "from skopt import gp_minimize # Bayesian optimization using Gaussian Processes\n",
    "from skopt.space import Real, Categorical, Integer\n",
    "from skopt.utils import use_named_args # decorator to convert a list of parameters to named arguments\n",
    "from skopt.callbacks import DeadlineStopper # Stop the optimization before running out of a fixed budget of time.\n",
    "from skopt.callbacks import VerboseCallback # Callback to control the verbosity\n",
    "from skopt.callbacks import DeltaXStopper # Stop the optimization If the last two positions at which the objective has been evaluated are less than delta\n",
    "\n",
    "n_jobs=-1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "B7R_4ujcqilp"
   },
   "outputs": [],
   "source": [
    "# Reporting util for different optimizers\n",
    "def report_perf(optimizer, X, y, title, callbacks=None):\n",
    "    \"\"\"\n",
    "    A wrapper for measuring time and performances of different optmizers\n",
    "    \n",
    "    optimizer = a sklearn or a skopt optimizer\n",
    "    X = the training set \n",
    "    y = our target\n",
    "    title = a string label for the experiment\n",
    "    \"\"\"\n",
    "    start = time()\n",
    "    if callbacks:\n",
    "        optimizer.fit(X, y, callback=callbacks)\n",
    "    else:\n",
    "        optimizer.fit(X, y)\n",
    "    best_score = optimizer.best_score_\n",
    "    best_score_std = optimizer.cv_results_['std_test_score'][optimizer.best_index_]\n",
    "    best_params = optimizer.best_params_\n",
    "    print((title + \" took %.2f seconds,  candidates checked: %d, best CV score: %.3f \"\n",
    "           +u\"\\u00B1\"+\" %.3f\") % (time() - start, \n",
    "                                  len(optimizer.cv_results_['params']),\n",
    "                                  best_score,\n",
    "                                  best_score_std))    \n",
    "    print('Best parameters:')\n",
    "    pprint.pprint(best_params)\n",
    "    print()\n",
    "    return best_params\n",
    "\n",
    "# Converting average precision score into a scorer suitable for model selection\n",
    "avg_prec = make_scorer(average_precision_score, greater_is_better=True, needs_proba=True)\n",
    "\n",
    "# Setting a 5-fold stratified cross-validation (note: shuffle=True)\n",
    "skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "CQLT-3vLrFA5"
   },
   "outputs": [],
   "source": [
    "# Uploading the Boston dataset\n",
    "X, y = load_boston(return_X_y=True)\n",
    "\n",
    "# Transforming the problem into a classification (unbalanced)\n",
    "y_bin = (y > np.percentile(y, 90)).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "6FwRTftcrFsu"
   },
   "outputs": [],
   "source": [
    "#CRIM - per capita crime rate by town\n",
    "#ZN - proportion of residential land zoned for lots over 25,000 sq.ft.\n",
    "#INDUS - proportion of non-retail business acres per town.\n",
    "#CHAS - Charles River dummy variable (1 if tract bounds river; 0 otherwise)\n",
    "#NOX - nitric oxides concentration (parts per 10 million)\n",
    "#RM - average number of rooms per dwelling\n",
    "#AGE - proportion of owner-occupied units built prior to 1940\n",
    "#DIS - weighted distances to five Boston employment centres\n",
    "#RAD - index of accessibility to radial highways\n",
    "#TAX - full-value property-tax rate per $10,000\n",
    "#PTRATIO - pupil-teacher ratio by town\n",
    "#B - 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
    "#LSTAT - % lower status of the population\n",
    "#MEDV - Median value of owner-occupied homes in $1000's this is our target variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 280
    },
    "colab_type": "code",
    "id": "EUyt9-RBrHyA",
    "outputId": "cfb37257-aab2-45f3-d8fc-7ed20ad6042f"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW8AAAEHCAYAAABhm8cjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAFpBJREFUeJzt3X+UZHV55/F3O03JzNhAw5Q4Aifo\nHn3AmKxHQhYxE8Y4BA2/kh2FbJgRBFYERV015pyNEYTdkMUfGIVVWF1A5piAOWuciXEwQ4CQVRNg\n8RcrD8IK8kunxcaZgck0A7N/1G1T3fR0V1dXdfV36v06Z85U3Xvr1tNPV3/62997b9XArl27kCSV\n5Xm9LkCSNHuGtyQVyPCWpAIZ3pJUIMNbkgpkeEtSgQZ7XYDKExG7gEMy8+GmZWcAazJzVUS8Ezgw\nM/9kmn38O2B7Zn6n6wV3WEQsAv4OeAlwUmZ+t2ndGcDVwImZ+TdNyxcDPwH+V2aeUW13BfDQpN0/\nkpmvn7R+r2rdeuDDmflERPwX4OWZecqk2l4O3AUsz8wtHfqStQAZ3uq4zLy8hc3eCvwjUFx4Ay8G\njgH2zsynp1j/EPAHwN80LTsBeGLSdt/IzFXTPM8v1kfEvsCfAbdExFHANcB3ImKfSSG9FviSwb3n\nM7zVcRFxIXBwZp4dEW8GLgAWAU8D7wIOA94CnBQRLwQ+AVwMrK528U3gHZn5ZES8Gri+Wr6u2uZd\nwAPA16t1r87MYyLiJOC/AjVgG3BWZn4rIlYClwD/BJwE/Ax4B40wPBy4MjMvmOLr+FXg08ABwL8A\nfwRsAm6hMeX43Yg4NTO/Pemh/xt4XUQsycynqmW/D3yNNn/mMvPnwLkRcRvwlsy8KiL+D/Am4H82\nbXoa8LZ2nkNlcc5b3fbfgeMz83DgPBrTDJ8B/hn4QGZ+HDgFeCNwBPDLwH7Af6oefxXw8cx8GfBz\n4OVN+14GfKsK7kHgWuA/ZmYAXwY+2rTtq4G/Bv4N8CxwOXA8sAr4zxGxd3PREfE84C+ByzPzMOBs\n4C+AJcDrgWcy87ApghtgB42QP7na1z7Aq2j8spmrDcDrqtvXAGuaan4tjV8Of9+B59ECZ3irXbdE\nxD3j/2iMbKeyGXh7RPxSZv5jZr53im2OB67NzCcz8xkac8a/Xc0TH0EjNKExBzzQ9Li9gC8BZOZO\n4IWZ+c1q3W3AS5u2fSIzb8nMXcDdwK3VqPhuGn8V1CfV9BLgRTQCnMy8A3gQOHKanjT7SxpTJwC/\nSyN0n520zWuae1j9m6o/zbYA+1a3bwCOjIiDq/trgc9n5uTn0R7IaRO1a+VUByyn2O4k4IPAnRHx\nEPCezLx10jZ1YLTp/ijwQmAY2JWZTwBk5tMRsblpu2cmze2+KyJOB54P7A00v3HP1ubH0ZhWITN3\nRcSzNAJ8ck1PVGE/ua7/N8XXOdnXgM9GxP40pkwuBmLSNjPNeU/lUBq/EMnMLRHxZeC0iLiMxhTK\nUbPcnwrlyFtdlZn3Z+ZbaYTenwNfmGKzn9CYVx53QLVsCzAQEUsAqqmRySNkqnVH05iTPqmaNjl7\njqX/BNg/IppH+uN1zag6kLkBOB14WWZ+Y471jJ/l8rs0fjGMuwb4D8BxwD2Zed9cn0dlMLzVNRFR\nj4i/q86IeJbGgcjxkezTNOa2oXFWxpqIWFIF9FnAVzJzG/B9GnPiAOcwcTTd7IU0RqQ/qsL+dGDp\npPCdjQeAh4FTq6/laBrTKP88i338BY1fKF9qs4ZfiIilNOb/R2lMl4z7exp/obyHRpCrTzhtoq7J\nzJGI2AjcHhHPAGM0ghkagfaRiHgp8D7gV4E7acxp3wx8struPOB/RMQf0jgg+QhTB/jGatv7q23e\nQ2MK4a+AT7VR+66I+H3gMxFxAfAk8ObqDJgpR/9TuJXGPPf1u1n/mup4wWSvn7R+EbCYxkHY46r5\n/fE6n42I62gc4P29FuvSHmDA9/PWQhcRA+NzzxExAqzazVkeUt9w2kQLWkR8EfhAdfu3aIzM7+1p\nUdIC4MhbC1pEHE7j1MH9aUy7/GFmfrW3VUm9Z3hLUoGcNpGkAs3L2SYjI1uLH94PDy9hdPSpmTfs\nE/ZjIvvxr+zFRHPpR70+tNtTXR15t2hwcPIFeP3NfkxkP/6VvZioW/0wvCWpQIa3JBXI8JakAhne\nklQgw1uSCmR4S1KBDG9JKpDhLUkF8v28JfWdNWsWd3R/69Ztn3GbT37yY9x99/cYGBjg3e9+H4cf\n/stzek7DW1Oa6cVdq8HY2HO3aeVFLPWbu+66k4cffogrr7yaBx74IZdcchFXXnn1nPbptIkkddmd\nd97OihUrATj00JewdesWnnxy25z2aXhLUpc9/vjj7Lfffr+4v99+wzz++ONz2qfhLUnzrBOfo2B4\nS1KXLVu2bMJI+6c//SnLli2b0z4Nb0nqsl//9aO45ZabAMi8h2XLlrFkydI57dOzTST1nfk+K+pX\nfuXfEnE4b3/7mQwMDPDe9/7RnPdpeEvSPDj33PM7uj+nTSSpQIa3JBVoxmmTiDgLWNu06NeA1wKf\nBnYB38nMc7tTniRpKjOOvDPzc5m5MjNXAhcA1wKfAN6dma8F9o2IN3a3TElSs9lOm3wI+G/ASzLz\n9mrZBmBVR6uSJE2r5fCOiCOBh4CdwGjTqs3A8g7XJUmaxmxOFTwbuGaK5QMzPXB4eAmDg4tm8VQL\nU70+1OsS5k2t1so2z3359FOPJuvnr32yBd+LE0/s7P42bJh2db0+xL333st5553HGWecwZo1a+b8\nlLMJ75XA+TQOUh7QtPwg4NHpHjg6+tSsC1to6vUhRka29rqMeTPV2702q9UGGRvb+ZzlIyP9+Zaw\n/fb6mE4JvdhnitfuXGyZ5uut14f40Y8286EPXcirXvVrbNv2Ly33Z7pfgi1Nm0TEi4FtmTmWmU8D\n90TEb1Sr/z2wsaVKJKkP7bXXXnz0o38+5/czadbqyHs5jbntce8BroyI5wH/lJmbOlaRJO1hBgcH\nGRzs7AXtLe0tM+8E3th0//8CKzpaiSSpZV5hKUkFMrwlqUC+q6CkvrNl3Q3z+nz33PN9Lr/8Mn78\n48cYHBzk5ptv4k//9CPss8++be/T8JakLjvssMO5/PKrOrpPp00kqUCGtyQVyPCWpAIZ3pJUIMNb\nkgpkeEtSgQxvSSqQ53mro9asmf6tZKeybl1/vo2sNBeOvCWpQIa3JBXI8JakAhneklQgw1uSCuTZ\nJuq5ds5QAc9SUX9z5C1JBWpp5B0RpwEfAHYCHwK+A1wHLAIeA9Zm5o5uFSlJmmjGkXdEHABcAPwG\ncAJwMnARcEVmrgDuA87sZpGSpIlamTZZBWzKzK2Z+Vhmvg1YCayv1m+otpEkzZNWpk0OBZZExHpg\nGLgQWNo0TbIZWD7dDoaHlzA4uGgOZS4M9fpQr0uYN7VaK9v09nj3Qvt+LLR6esleTNSNfrTy0zcA\nHAD8HvBLwM3Vsub10xodfaqt4haSen2IkZGtvS5j3oyNTX8GSK02yNjYznmqZmojIwvnbJN+e31M\nx15MNJd+TBf6rUyb/AT4embuzMz7ga3A1ogY/+k+CHi0rcokSW1pJby/BvxWRDyvOnj5AmATsLpa\nvxrY2KX6JElTmDG8M/MR4K+AbwJfBc6ncfbJ6RFxG7A/cG03i5QkTdTSEafMvBK4ctLiYztfjiSp\nFV5hKUkFMrwlqUCGtyQVyPCWpAIZ3pJUIMNbkgpkeEtSgQxvSSqQ4S1JBTK8JalAhrckFcjwlqQC\nGd6SVCDDW5IKZHhLUoEMb0kqkOEtSQUyvCWpQIa3JBVoxs+wjIiVwBeBu6tF3wUuBa4DFgGPAWsz\nc0eXapQkTdLqyPvWzFxZ/TsfuAi4IjNXAPcBZ3atQknSc7Q7bbISWF/d3gCs6kg1kqSWzDhtUnlF\nRKwH9gc+DCxtmibZDCyf7sHDw0sYHFzUfpULRL0+1OsS5k2t1so2rb58umOhfT8WWj29ZC8m6kY/\nWvnp+wGNwL4BeClw86THDcy0g9HRp9oqbiGp14cYGdna6zLmzdjY4mnX12qDjI3tnKdqpjYysr2n\nz9+s314f07EXE82lH9OF/ozhnZmPANdXd++PiB8DR0bE4szcDhwEPNpWZZKktsw45x0Rp0XE+6vb\nLwIOBK4GVlebrAY2dq1CSdJztDJtsh74QkScDNSAc4G7gM9HxDnAg8C13StRkjRZK9MmW4ETp1h1\nbOfLkSS1wissJalAhrckFcjwlqQCGd6SVCDDW5IKZHhLUoEMb0kqkOEtSQUyvCWpQIa3JBXI8Jak\nAhneklQgw1uSCtTbz7HSvFizZvpPxZFUHkfeklQgw1uSCmR4S1KBDG9JKlBLBywjYjHwPeBi4Cbg\nOmAR8BiwNjN3dK1CSdJztDry/iDws+r2RcAVmbkCuA84sxuFSZJ2b8bwjojDgFcAX6kWraTxifIA\nG4BVXalMkrRbrUybfAx4J3B6dX9p0zTJZmD5TDsYHl7C4OCi9ipcQOr1oV6X0JZarVv77e1lAgvt\n+7HQ6uklezFRN/ox7U9fRLwF+EZm/jAiptpkoJUnGR19qo3SFpZ6fYiRka29LqMtY2Odv0inVhtk\nbGxnx/c7GyMj23v6/M1Kfn10mr2YaC79mC70Zxo6HQ+8NCJOAA4GdgDbImJxZm4HDgIebasqSVLb\npg3vzDx1/HZEXAg8ABwNrAbWVf9v7F55e652L1lft27hjDYl9U4753lfAJweEbcB+wPXdrYkSdJM\nWj7ilJkXNt09tvOlSJJa5RWWklQgw1uSCmR4S1KBDG9JKpDhLUkFMrwlqUCGtyQVyPCWpAL56fGF\n8ZPgJYEjb0kqkuEtSQUyvCWpQIa3JBXI8JakAhneklQgw1uSCmR4S1KBDG9JKpDhLUkFmvHy+IhY\nAlwDHAjsDVwMfBu4DlgEPAaszcwd3StTktSslZH3icAdmXkMcArwceAi4IrMXAHcB5zZvRIlSZPN\nOPLOzOub7h4CPAysBN5eLdsAvB/4dKeLkyRNreV3FYyIrwMHAycAm5qmSTYDy6d77PDwEgYHF7Vd\n5EJRrw91bF+1Wsd21TO1Wm/flLKT349OWGj19JK9mKgb/Wj5py8zj46IVwHrgIGmVQO7ecgvjI4+\n1UZpC0u9PsTIyNaO7W9srOy3dq3VBhkb29nTGkZGtvf0+Zt1+vVRMnsx0Vz6MV3ozzjnHRFHRMQh\nAJn5LRqBvzUixtPnIODRtiqTJLWllQOWvwm8DyAiDgReAGwCVlfrVwMbu1KdJGlKrUybfAb4XETc\nBiwG3gHcAXw+Is4BHgSu7V6JkqTJWjnbZDvwB1OsOrbz5UiSWuEVlpJUIMNbkgpkeEtSgQxvSSqQ\n4S1JBTK8JalAhrckFcjwlqQCGd6SVCDDW5IKZHhLUoEMb0kqkOEtSQXq7edYSXOwZs3sP41o3bqF\n8+k70lw48pakAhneklQgw1uSCmR4S1KBWjpgGRGXAiuq7S8BbgeuAxYBjwFrM3NHt4qUJE0048g7\nIl4HvDIzXwO8AfgEcBFwRWauAO4DzuxqlZKkCVqZNvkH4M3V7SeApcBKYH21bAOwquOVSZJ2q5VP\nj38GeLK6exbwt8BxTdMkm4Hl3SlPkjSVli/SiYiTaYT3bwM/aFo1MNNjh4eXMDi4aPbVLTD1+lDH\n9lWrdWxXPVOrlXeNVye/h/O579LYi4m60Y9WD1geB/wx8IbM/HlEbIuIxZm5HTgIeHS6x4+OPjX3\nSnusXh9iZGRrx/Y3Njb7qwMXklptkLGxnb0uY9ZGRrpzhWWnXx8lsxcTzaUf04X+jOEdEfsCHwFW\nZebPqsWbgNXAuur/jW1VJs0zL6nXXOyz5pTZP+jGr3a+EFobeZ8KLANuiIjxZacDn42Ic4AHgWu7\nUp0kaUqtHLC8CrhqilXHdr4cSVIryjvitAC186e4JM2Fl8dLUoEMb0kq0B47beJZBZL2ZI68JalA\nhrckFWiPnTaROqWVKbha7blXzToNp25y5C1JBXLkLXWJB83VTY68JalAhrckFcjwlqQCGd6SVCDD\nW5IK5NkmTaY7O2Cq83ilhaSdDwrYsu6GLlSi+eDIW5IKZHhLUoGcNpEWkHY/2MOLe/qPI29JKlBL\nI++IeCXwZeCyzLw8Ig4BrgMWAY8BazNzR/fKlCQ1m3HkHRFLgU8BNzUtvgi4IjNXAPcBZ3anPEnS\nVFqZNtkB/A7waNOylcD66vYGYFVny5IkTWfGaZPM3AnsjIjmxUubpkk2A8un28fw8BIGBxe1XWQ7\narVu7NPju83sx0S97Ee9PgRtPH+9PtSFarq3355r83vcjX504tU2MNMGo6NPdeBpZqfTF9TUaoOM\nje3s6D5LZj8m6nU/Rka2s08bz79lZGvHa6nXhxjpwn4XgnZ6/Hxoux/ThX67Z5tsi4jxdDyIiVMq\nkqQuaze8NwGrq9urgY2dKUeS1IoZp00i4gjgY8ChwNMR8SbgNOCaiDgHeBC4tptFSpImauWA5Z00\nzi6Z7NiOVyNJaolXWEpSgQxvSSqQ4S1JBTK8JalAhrckFcjwlqQCGd6SVCDDW5IK5NvCSX1stp84\nP1+fNj/bumD+alsoHHlLUoEMb0kqUBHTJu1+orYk7akceUtSgQxvSSpQEdMmksrVzpkj8/E8pZ+d\n4shbkgrkyFtSy1oa3dYG2/qgXs2OI29JKpDhLUkFanvaJCIuA44CdgHvzszbO1aVJHXZfB1I7Za2\nRt4RcQzwssx8DXAW8MmOViVJmla70yavB/4aIDO/DwxHxD4dq0qSNK12p01eBNzZdH+kWrZlqo3r\n9aGBNp8HgBtvnMujO8mTcyayHxP1sh9DcONXe/j8Ez2/1wUsMPX6UMf32akDlnMKZ0nS7LQb3o/S\nGGmPezHw2NzLkSS1ot3w/hrwJoCIeDXwaGZu7VhVkqRpDezatautB0bEnwG/CTwLvCMzv93JwiRJ\nu9d2eEuSescrLCWpQIa3JBXIE3WnERGvBL4MXJaZl0fEIcB1wCIaZ9eszcwdvaxxvkTEpcAKGq+Z\nS4Db6cNeRMQS4BrgQGBv4GLg2/RhL5pFxGLgezT6cRN92o+IWAl8Ebi7WvRd4FK60A9H3rsREUuB\nT9F4IY67CLgiM1cA9wFn9qK2+RYRrwNeWb0dwhuAT9CnvQBOBO7IzGOAU4CP07+9aPZB4GfV7X7v\nx62ZubL6dz5d6ofhvXs7gN+hcU77uJXA+ur2BmDVPNfUK/8AvLm6/QSwlD7tRWZen5mXVncPAR6m\nT3sxLiIOA14BfKVatJI+7scUVtKFfjhtshuZuRPYGRHNi5c2/bmzGVg+74X1QGY+AzxZ3T0L+Fvg\nuH7sxbiI+DpwMHACsKmfewF8DHgncHp1vy9/Tpq8IiLWA/sDH6ZL/XDk3b6+e0uAiDiZRni/c9Kq\nvutFZh4NnASsY+LX31e9iIi3AN/IzB/uZpO+6gfwAxqBfTKNX2afY+IguWP9MLxnZ1t1YAbgICZO\nqezRIuI44I+BN2bmz+nTXkTEEdWBazLzWzR+MLf2Yy8qxwMnR8Q3gbOBP6FPXxsAmflINbW2KzPv\nB35M411XO94Pw3t2NgGrq9urgY09rGXeRMS+wEeAEzJz/KBUX/aCxlXF7wOIiAOBF9C/vSAzT83M\nIzPzKOCzNM426dt+RMRpEfH+6vaLaJyVdDVd6IdXWO5GRBxBYy7vUOBp4BHgNBqnie0NPAi8NTOf\n7lGJ8yYi3gZcCNzbtPh0Gj+s/daLxTT+FD4EWEzjT+Q7gM/TZ72YLCIuBB4AbqRP+xERQ8AXgP2A\nGo3Xx110oR+GtyQVyGkTSSqQ4S1JBTK8JalAhrckFcjwlqQCGd6SVCDDW5IK9P8BAaMMZR/VUCUA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Histogram highlighting the top 10% we use as a target\n",
    "plt.hist(y[y <= np.percentile(y, 90)], bins='auto', alpha=0.7, label='0', color='b')\n",
    "plt.hist(y[y > np.percentile(y, 90)], bins=8, alpha=0.7, label='1', color='r')\n",
    "plt.title(\"Histogram of MEDV\")\n",
    "plt.legend(loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "XEP-4N8BRvxC"
   },
   "source": [
    "# A Practical Example: Optimizing LightGBM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Re1MTGLzRvxD"
   },
   "source": [
    "### LightGbm\n",
    "The high-performance [LightGBM](https://github.com/Microsoft/LightGBM) algorithm is capable of being distributed and of fast-handling large amounts of data. It has been developed by a team at Microsoft as an open source project on GitHub (there is also an [academic paper](https://papers.nips.cc/paper/6907-lightgbm-a-highly-efficientgradient-boosting-decision-tree)). \n",
    "\n",
    "LightGBM is based on decision trees, as well as XGBoost, yet it follows a different strategy.\n",
    "Whereas XGBoost uses decision trees to split on a variable and exploring different cuts at that variable (the level-wise tree growth strategy), LightGBM concentrates on a split and goes on splitting from there in order to achieve a better fitting (this is the leaf-wise tree\n",
    "growth strategy). This allows LightGBM to reach first and fast a good fit of the data, and to generate alternative solutions compared to XGBoost (which is good, if you expect to blend, i.e. average, the two solutions together in order to reduce the variance of the estimated). Algorithmically talking, figuring out as a graph the structures of cuts operated by a decision tree, XGBoost peruses a breadth-first search (BFS), whereas LightGBM a depthfirst search (DFS).\n",
    "\n",
    "Tuning LightGBM may appear daunting with more than a [hundred parameters](https://github.com/Microsoft/LightGBM/blob/master/docs/Parameters.rst) (also to be found [here](https://lightgbm.readthedocs.io/en/latest/Parameters.html)) to tune."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 306
    },
    "colab_type": "code",
    "id": "anDzO9nBRvxF",
    "outputId": "b410deae-fc0d-4ef2-edba-c4b8e2f4f214"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LightGBM took 270.63 seconds,  candidates checked: 31, best CV score: 0.927 ± 0.058\n",
      "Best parameters:\n",
      "{'colsample_bytree': 0.3200094536415732,\n",
      " 'learning_rate': 0.01,\n",
      " 'max_bin': 100,\n",
      " 'max_depth': 179,\n",
      " 'min_child_samples': 0,\n",
      " 'min_child_weight': 2,\n",
      " 'n_estimators': 7676,\n",
      " 'num_leaves': 500,\n",
      " 'reg_alpha': 0.9444340710142283,\n",
      " 'reg_lambda': 0.6597373818413536,\n",
      " 'scale_pos_weight': 0.6594079576173991,\n",
      " 'subsample': 0.9923669100402286,\n",
      " 'subsample_for_bin': 355905,\n",
      " 'subsample_freq': 6}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "clf = lgb.LGBMClassifier(boosting_type='gbdt',\n",
    "                         class_weight='balanced',\n",
    "                         objective='binary',\n",
    "                         n_jobs=1, \n",
    "                         verbose=0)\n",
    "\n",
    "search_spaces = {\n",
    "        'learning_rate': Real(0.01, 1.0, 'log-uniform'),\n",
    "        'num_leaves': Integer(2, 500),\n",
    "        'max_depth': Integer(0, 500),\n",
    "        'min_child_samples': Integer(0, 200), # minimal number of data in one leaf\n",
    "        'max_bin': Integer(100, 100000), # max number of bins that feature values will be bucketed\n",
    "        'subsample': Real(0.01, 1.0, 'uniform'),\n",
    "        'subsample_freq': Integer(0, 10), # bagging fraction\n",
    "        'colsample_bytree': Real(0.01, 1.0, 'uniform'), # enabler of bagging fraction\n",
    "        'min_child_weight': Integer(0, 10), # minimal number of data in one leaf.\n",
    "        'subsample_for_bin': Integer(100000, 500000), # number of data that sampled for histogram bins\n",
    "        'reg_lambda': Real(1e-9, 1000, 'log-uniform'), # L2 regularization\n",
    "        'reg_alpha': Real(1e-9, 1.0, 'log-uniform'), # L1 regularization\n",
    "        'scale_pos_weight': Real(1e-6, 500, 'log-uniform'), \n",
    "        'n_estimators': Integer(10, 10000)        \n",
    "        }\n",
    "\n",
    "opt = BayesSearchCV(clf,\n",
    "                    search_spaces,\n",
    "                    scoring=avg_prec,\n",
    "                    cv=skf,\n",
    "                    n_iter=40,\n",
    "                    n_jobs=-1,\n",
    "                    return_train_score=False,\n",
    "                    refit=True,\n",
    "                    optimizer_kwargs={'base_estimator': 'GP'},\n",
    "                    random_state=22)\n",
    "    \n",
    "best_params = report_perf(opt, X, y_bin,'LightGBM', \n",
    "                          callbacks=[DeltaXStopper(0.001), \n",
    "                                     DeadlineStopper(60*5)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "JUTFLEnoRvxM"
   },
   "outputs": [],
   "source": [
    "counter = 0\n",
    "\n",
    "clf = lgb.LGBMClassifier(boosting_type='gbdt',\n",
    "                         class_weight='balanced',\n",
    "                         objective='binary',\n",
    "                         n_jobs=1, \n",
    "                         verbose=0)\n",
    "\n",
    "dimensions = [Real(0.01, 1.0, 'log-uniform', name='learning_rate'),\n",
    "              Integer(2, 500, name='num_leaves'),\n",
    "              Integer(0, 500, name='max_depth'),\n",
    "              Integer(0, 200, name='min_child_samples'),\n",
    "              Integer(100, 100000, name='max_bin'),\n",
    "              Real(0.01, 1.0, 'uniform', name='subsample'),\n",
    "              Integer(0, 10, name='subsample_freq'),\n",
    "              Real(0.01, 1.0, 'uniform', name='colsample_bytree'),\n",
    "              Integer(0, 10, name='min_child_weight'),\n",
    "              Integer(100000, 500000, name='subsample_for_bin'),\n",
    "              Real(1e-9, 1000, 'log-uniform', name='reg_lambda'),\n",
    "              Real(1e-9, 1.0, 'log-uniform', name='reg_alpha'),\n",
    "              Real(1e-6, 500, 'log-uniform', name='scale_pos_weight'),\n",
    "              Integer(10, 10000, name='n_estimators')]\n",
    "\n",
    "# The objective function to be minimized\n",
    "def make_objective(model, X, y, space, cv, scoring):\n",
    "    # This decorator converts your objective function with named arguments into one that\n",
    "    # accepts a list as argument, while doing the conversion automatically.\n",
    "    @use_named_args(space) \n",
    "    def objective(**params):\n",
    "        model.set_params(**params)\n",
    "        return -np.mean(cross_val_score(model, \n",
    "                                        X, y, \n",
    "                                        cv=cv, \n",
    "                                        n_jobs=-1,\n",
    "                                        scoring=scoring))\n",
    "    return objective\n",
    "\n",
    "objective = make_objective(clf,\n",
    "                           X, y_bin,\n",
    "                           space=dimensions,\n",
    "                           cv=skf,\n",
    "                           scoring=avg_prec)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 377
    },
    "colab_type": "code",
    "id": "w6kUwIXJRvxP",
    "outputId": "84b654a0-59a9-438b-a3de-6831bb8fbb78"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last eval:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]  - Score  -0.1007765482430596\n",
      "Current iter:  0  - Score  -0.1007765482430596  - Args:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]\n",
      "Last eval:  [0.2211233200687724, 348, 173, 186, 26332, 0.7532551005821186, 3, 0.8527816404253235, 2, 416305, 178.4645772067005, 1.095045483795558e-05, 0.0022184775182806722, 3581]  - Score  -0.1007765482430596\n",
      "Current iter:  1  - Score  -0.1007765482430596  - Args:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]\n",
      "Last eval:  [0.02436190508232831, 52, 241, 123, 95130, 0.3552688118981681, 3, 0.45204083607316786, 10, 171831, 5.159278367758748e-08, 5.643816697371209e-09, 0.05759462438024217, 3109]  - Score  -0.1007765482430596\n",
      "Current iter:  2  - Score  -0.1007765482430596  - Args:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]\n",
      "Last eval:  [0.036765413368033246, 461, 217, 84, 98624, 0.7956342983145053, 3, 0.307335685328769, 5, 349302, 0.07650111834271556, 2.1489283637488493e-07, 0.0033482236099689746, 6676]  - Score  -0.1007765482430596\n",
      "Current iter:  3  - Score  -0.1007765482430596  - Args:  [0.02848906260926589, 161, 489, 91, 30870, 0.2712321306475197, 1, 0.4251784883123234, 0, 311106, 26.645108834238002, 9.496274156079435e-07, 0.002619348997981635, 6747]\n",
      "Last eval:  [0.07568374719544937, 391, 342, 147, 48993, 0.9544514920010971, 2, 0.6526276640345808, 6, 447413, 5.290980835305423e-08, 1.0743335173861622e-05, 0.5428137986259088, 6620]  - Score  -0.5184708747168858\n",
      "Current iter:  4  - Score  -0.5184708747168858  - Args:  [0.07568374719544937, 391, 342, 147, 48993, 0.9544514920010971, 2, 0.6526276640345808, 6, 447413, 5.290980835305423e-08, 1.0743335173861622e-05, 0.5428137986259088, 6620]\n",
      "Last eval:  [0.47548110586714587, 267, 20, 19, 79910, 0.4083645994626534, 7, 0.5069042541174733, 4, 414327, 260.63557428396865, 2.0969660888077091e-07, 0.03329930701856852, 8682]  - Score  -0.1007765482430596\n",
      "Current iter:  5  - Score  -0.5184708747168858  - Args:  [0.07568374719544937, 391, 342, 147, 48993, 0.9544514920010971, 2, 0.6526276640345808, 6, 447413, 5.290980835305423e-08, 1.0743335173861622e-05, 0.5428137986259088, 6620]\n",
      "Last eval:  [0.08293216060373823, 57, 400, 100, 19798, 0.8773143587445786, 3, 0.24456885606236203, 4, 254361, 6.267403019545165e-07, 3.8195699908270306e-08, 0.0029281270257610842, 8418]  - Score  -0.4932644929680439\n",
      "Current iter:  6  - Score  -0.5184708747168858  - Args:  [0.07568374719544937, 391, 342, 147, 48993, 0.9544514920010971, 2, 0.6526276640345808, 6, 447413, 5.290980835305423e-08, 1.0743335173861622e-05, 0.5428137986259088, 6620]\n",
      "Last eval:  [0.3745080737521305, 321, 248, 95, 47476, 0.9441414325971597, 7, 0.051157271586472694, 8, 305360, 1.3839138544254465e-07, 3.372981975245276e-06, 149.1039700696765, 2137]  - Score  -0.6843172927234125\n",
      "Current iter:  7  - Score  -0.6843172927234125  - Args:  [0.3745080737521305, 321, 248, 95, 47476, 0.9441414325971597, 7, 0.051157271586472694, 8, 305360, 1.3839138544254465e-07, 3.372981975245276e-06, 149.1039700696765, 2137]\n",
      "Last eval:  [0.42324568665282264, 203, 293, 37, 81251, 0.5850389357102426, 4, 0.8001005930590471, 5, 267367, 1.6977743026226538e-09, 0.0014254878711280096, 0.01761769861494498, 2214]  - Score  -0.3623421719518635\n",
      "Current iter:  8  - Score  -0.6843172927234125  - Args:  [0.3745080737521305, 321, 248, 95, 47476, 0.9441414325971597, 7, 0.051157271586472694, 8, 305360, 1.3839138544254465e-07, 3.372981975245276e-06, 149.1039700696765, 2137]\n",
      "Last eval:  [0.12620776965554556, 262, 244, 53, 52776, 0.9846210751689275, 6, 0.6332613654964749, 9, 485008, 0.010500201470771384, 3.0484544036415625e-08, 0.0025495207496935173, 9446]  - Score  -0.1007765482430596\n",
      "Current iter:  9  - Score  -0.6843172927234125  - Args:  [0.3745080737521305, 321, 248, 95, 47476, 0.9441414325971597, 7, 0.051157271586472694, 8, 305360, 1.3839138544254465e-07, 3.372981975245276e-06, 149.1039700696765, 2137]\n"
     ]
    }
   ],
   "source": [
    "gp_round = gp_minimize(func=objective,\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge',\n",
    "                       n_calls=10, # Minimum is 10 calls\n",
    "                       callback=[onstep],\n",
    "                       random_state=7)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "colab_type": "code",
    "id": "zY46ARHqRvxT",
    "outputId": "ee9c344f-ea64-430b-e055-60ebc8976e7e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.12324384123200562, 414, 117, 41, 95838, 0.7684912331585715, 2, 0.47468749760756623, 5, 168945, 3.8096250370376167e-06, 2.625825905482537e-07, 51.89197450218198, 5038] -0.8572449837163276\n"
     ]
    }
   ],
   "source": [
    "x0, y0 = joblib.load('checkpoint.pkl')\n",
    "\n",
    "gp_round = gp_minimize(func=objective,\n",
    "                       x0=x0,              # already examined values for x\n",
    "                       y0=y0,              # observed values for x0\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge', # Expected Improvement.\n",
    "                       n_calls=10,\n",
    "                       #callback=[onstep],\n",
    "                       random_state=3)\n",
    "\n",
    "best_parameters = gp_round.x\n",
    "best_result = gp_round.fun\n",
    "print(best_parameters, best_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O3SsbSwPRvxX"
   },
   "source": [
    "# Practical example: Optimizing XGBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VDvBGSmORvxY"
   },
   "source": [
    "### XGBoost\n",
    "\n",
    "[XGBoost](https://github.com/dmlc/XGBoost) stands for eXtreme Gradient Boosting, an open source project that is not part of Scikit-learn, though recently it has been expanded by a Scikit-Learn wrapper interface that renders using models based on XGBoost more integrated into your data pipeline.\n",
    "\n",
    "The XGBoost algorithm has gained recently gained momentum and popularity in datascience competitions such as Kaggle and the KDD-cup 2015. As the authors (Tianqui Chen, Tong He, and Carlos Guestrin) report on papers they wrote on the algorithm, among 29 challenges held on Kaggle during 2015, 17 winning solutions used XGBoost as a standalone solution or as part of an ensemble of multiple different models.\n",
    "\n",
    "Apart from the successful performances in both accuracy and computational efficiency, XGBoost is also a scalable solution under different points of view. XGBoost represents a new generation of GBM algorithms thanks to important tweaks to the initial tree boost GBM algorithm:\n",
    "\n",
    "* A sparse-aware algorithm; it can leverage sparse matrices, saving both memory (no need for dense matrices) and computation time (zero values are handled in a special way).\n",
    "* Approximate tree learning (weighted quantile sketch), which bears similar results but in much less time than the classical complete explorations of possible branch cuts.\n",
    "* Parallel computing on a single machine (using multi-threading in the phase of the search for the best split) and similarly distributed computations on multiple ones. \n",
    "* Out-of-core computations on a single machine leveraging a data storage solution called Column Block. This arranges data on a disk by columns, thus saving time by pulling data from the disk as the optimization algorithm (which works on column vectors) expects it.\n",
    "* XGBoost can also deal with missing data in an effective way. Other tree ensembles based on standard decision trees require missing data first to be imputed using an off-scale value, such as a negative number, in order to develop an appropriate branching of the tree to deal with missing values.\n",
    "\n",
    "As for as XGBoost's [parameters](https://xgboost.readthedocs.io/en/latest/parameter.html), we have decided to work on a few key ones you will find across competitions and projects.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2pu82Fy1RvxZ"
   },
   "outputs": [],
   "source": [
    "clf = xgb.XGBClassifier(\n",
    "        n_jobs = 1,\n",
    "        objective = 'binary:logistic',\n",
    "        silent=1,\n",
    "        tree_method='approx')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S4fGg7KqRvxc"
   },
   "outputs": [],
   "source": [
    "search_spaces = {'learning_rate': Real(0.01, 1.0, 'log-uniform'),\n",
    "                 'min_child_weight': Integer(0, 10),\n",
    "                 'max_depth': Integer(1, 50),\n",
    "                 'max_delta_step': Integer(0, 20), # Maximum delta step we allow each leaf output\n",
    "                 'subsample': Real(0.01, 1.0, 'uniform'),\n",
    "                 'colsample_bytree': Real(0.01, 1.0, 'uniform'), # subsample ratio of columns by tree\n",
    "                 'colsample_bylevel': Real(0.01, 1.0, 'uniform'), # subsample ratio by level in trees\n",
    "                 'reg_lambda': Real(1e-9, 1000, 'log-uniform'), # L2 regularization\n",
    "                 'reg_alpha': Real(1e-9, 1.0, 'log-uniform'), # L1 regularization\n",
    "                 'gamma': Real(1e-9, 0.5, 'log-uniform'), # Minimum loss reduction for partition\n",
    "                 'n_estimators': Integer(50, 100),\n",
    "                 'scale_pos_weight': Real(1e-6, 500, 'log-uniform')}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 272
    },
    "colab_type": "code",
    "id": "3SH81Up9Rvxh",
    "outputId": "23de6f26-d6c4-4c06-f065-e214b8967449"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "XGBoost took 107.17 seconds,  candidates checked: 40, best CV score: 0.922 ± 0.081\n",
      "Best parameters:\n",
      "{'colsample_bylevel': 0.04334407493951863,\n",
      " 'colsample_bytree': 1.0,\n",
      " 'gamma': 4.94716569768279e-05,\n",
      " 'learning_rate': 0.07046086205661303,\n",
      " 'max_delta_step': 0,\n",
      " 'max_depth': 49,\n",
      " 'min_child_weight': 1,\n",
      " 'n_estimators': 61,\n",
      " 'reg_alpha': 0.01532015629161949,\n",
      " 'reg_lambda': 0.07669037760197192,\n",
      " 'scale_pos_weight': 7.06962814248252,\n",
      " 'subsample': 1.0}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "opt = BayesSearchCV(clf,\n",
    "                    search_spaces,\n",
    "                    scoring=avg_prec,\n",
    "                    cv=skf,\n",
    "                    n_iter=40,\n",
    "                    n_jobs=-1,\n",
    "                    return_train_score=False,\n",
    "                    refit=True,\n",
    "                    optimizer_kwargs={'base_estimator': 'GP'},\n",
    "                    random_state=22)\n",
    "    \n",
    "best_params = report_perf(opt, X, y_bin,'XGBoost',                           \n",
    "                          callbacks=[DeltaXStopper(0.001), \n",
    "                                     DeadlineStopper(60*5)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "z2uGBuGURvxl"
   },
   "source": [
    "# Practical Example: Optimizing CatBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PxHLiZZrRvxm"
   },
   "source": [
    "### CatBoost\n",
    "In July 2017, another interesting GBM algorithm was made public by Yandex, the Russian search engine: it is [CatBoost](https://catboost.yandex/), whose name comes from putting together the two words Category and Boosting. In fact, its strongest point is the capability of handling categorical variables, which actually make the most of information in most relational databases, by adopting a mixed strategy of one-hot-encoding and mean encoding (a way to express categorical levels by assigning them an appropriate numeric value for the problem at hand; more on that later).\n",
    "\n",
    "The idea used by CatBoost to encode the categorical variables is not new, but it has been a kind of feature engineering used various times, mostly in data science competitions like at Kaggle’s. Mean encoding, also known as likelihood encoding, impact coding, or target coding, is simply a way to transform your labels into a number based on their association with the target variable. If you have a regression, you could transform labels based on the mean target value typical of that level; if it is a classification, it is simply the probability of classification of your target given that label (probability of your target, conditional on each category value). It may appear as a simple and smart feature engineering trick, but actually, it has side effects, mostly in terms of overfitting because you are taking information from the target into your predictors.\n",
    "\n",
    "CatBoost has quite a few [parameters](https://tech.yandex.com/catboost/doc/dg/concepts/python-reference_parameters-list-docpage/#python-reference_parameters-list), we have delimited our search to the 9 most important ones. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iCsu8V-hRvxn"
   },
   "outputs": [],
   "source": [
    "clf = CatBoostClassifier(loss_function='Logloss',\n",
    "                         verbose = False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "srqK_CRuRvx6"
   },
   "outputs": [],
   "source": [
    "search_spaces = {'iterations': Integer(10, 100),\n",
    "                 'depth': Integer(1, 8),\n",
    "                 'learning_rate': Real(0.01, 1.0, 'log-uniform'),\n",
    "                 'random_strength': Real(1e-9, 10, 'log-uniform'), # randomness for scoring splits\n",
    "                 'bagging_temperature': Real(0.0, 1.0), # settings of the Bayesian bootstrap\n",
    "                 'border_count': Integer(1, 255), # splits for numerical features\n",
    "                 'l2_leaf_reg': Integer(2, 30), # L2 regularization\n",
    "                 'scale_pos_weight':Real(0.01, 10.0, 'uniform')}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "O9_YKa0HRvyA",
    "outputId": "f246dab6-dc27-4048-d78c-c910209d7354"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CatBoost took 276.67 seconds,  candidates checked: 37, best CV score: 0.935 ± 0.063\n",
      "Best parameters:\n",
      "{'bagging_temperature': 0.9410700873598651,\n",
      " 'border_count': 202,\n",
      " 'depth': 3,\n",
      " 'iterations': 100,\n",
      " 'l2_leaf_reg': 19,\n",
      " 'learning_rate': 1.0,\n",
      " 'random_strength': 0.005016205748160196,\n",
      " 'scale_pos_weight': 0.6343004871558543}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "opt = BayesSearchCV(clf,\n",
    "                    search_spaces,\n",
    "                    scoring=avg_prec,\n",
    "                    cv=skf,\n",
    "                    n_iter=40,\n",
    "                    n_jobs=1,  # use just 1 job with CatBoost in order to avoid segmentation fault\n",
    "                    return_train_score=False,\n",
    "                    refit=True,\n",
    "                    optimizer_kwargs={'base_estimator': 'GP'},\n",
    "                    random_state=22)\n",
    "\n",
    "best_params = report_perf(opt, X, y_bin,'CatBoost', \n",
    "                          callbacks=[DeltaXStopper(0.001), \n",
    "                                     DeadlineStopper(60*5)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ZIZJvHCGRvyF"
   },
   "source": [
    "# Practical Example: Neural Architecture Search (NAS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "id": "ybA-BNfRRvyF",
    "outputId": "386116ba-131a-4e52-f8ad-abc14fec26ff"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow version: 1.13.1\n",
      "_DeviceAttributes(/job:localhost/replica:0/task:0/device:CPU:0, CPU, 268435456, 13672192623543012388)\n",
      "_DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_CPU:0, XLA_CPU, 17179869184, 6389059333328053870)\n",
      "_DeviceAttributes(/job:localhost/replica:0/task:0/device:XLA_GPU:0, XLA_GPU, 17179869184, 1154598800763684967)\n",
      "_DeviceAttributes(/job:localhost/replica:0/task:0/device:GPU:0, GPU, 14800692839, 16655201108311178928)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "print(\"TensorFlow version: %s\" % tf.__version__)\n",
    "with tf.Session() as sess:\n",
    "    devices = sess.list_devices()\n",
    "    for item in devices:\n",
    "        print(item)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 241
    },
    "colab_type": "code",
    "id": "L_C16zqNRvyL",
    "outputId": "271939ae-3bf2-4851-aec7-6c4d3a52dad9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/scikit-optimize/scikit-optimize.git\n",
      "  Cloning https://github.com/scikit-optimize/scikit-optimize.git to /tmp/pip-req-build-uk1sai5s\n",
      "Requirement already satisfied (use --upgrade to upgrade): scikit-optimize==0.5.2+49.g074ce8e from git+https://github.com/scikit-optimize/scikit-optimize.git in /usr/local/lib/python3.6/dist-packages\n",
      "Requirement already satisfied: pyaml in /usr/local/lib/python3.6/dist-packages (from scikit-optimize==0.5.2+49.g074ce8e) (18.11.0)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.6/dist-packages (from scikit-optimize==0.5.2+49.g074ce8e) (1.16.2)\n",
      "Requirement already satisfied: scipy>=0.14.0 in /usr/local/lib/python3.6/dist-packages (from scikit-optimize==0.5.2+49.g074ce8e) (1.2.1)\n",
      "Requirement already satisfied: scikit-learn>=0.19.1 in /usr/local/lib/python3.6/dist-packages (from scikit-optimize==0.5.2+49.g074ce8e) (0.20.3)\n",
      "Requirement already satisfied: PyYAML in /usr/local/lib/python3.6/dist-packages (from pyaml->scikit-optimize==0.5.2+49.g074ce8e) (3.13)\n",
      "Building wheels for collected packages: scikit-optimize\n",
      "  Building wheel for scikit-optimize (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Stored in directory: /tmp/pip-ephem-wheel-cache-6kj43e0b/wheels/11/6f/86/2b772172db85ad0b4487d67e325e535ee8e7782b2a1dfcadf5\n",
      "Successfully built scikit-optimize\n"
     ]
    }
   ],
   "source": [
    "# Installing the most recent version of skopt directly from Github\n",
    "!pip install git+https://github.com/scikit-optimize/scikit-optimize.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "kF2rpLRkRvyO",
    "outputId": "838e4000-832a-4bb6-d9e5-93c532643447"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Keras vs: 2.2.4\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import keras\n",
    "print(\"Keras vs: %s\" % keras.__version__)\n",
    "\n",
    "K = keras.backend\n",
    "layers = keras.layers\n",
    "models = keras.models\n",
    "optimizers = keras.optimizers\n",
    "losses = keras.losses\n",
    "pad_sequences = keras.preprocessing.sequence.pad_sequences\n",
    "\n",
    "from keras.models import Sequential\n",
    "from keras.layers import LeakyReLU\n",
    "from keras.layers import Activation\n",
    "from keras.optimizers import SGD, Adam\n",
    "from keras.layers import Dense, Dropout\n",
    "from keras.layers import Flatten, RepeatVector, dot, multiply, Permute, Lambda\n",
    "from keras.utils import multi_gpu_model\n",
    "from keras.wrappers.scikit_learn import KerasClassifier\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "7nOkqPY5RvyR",
    "outputId": "f6965ade-2d7e-455d-ff96-90df20c161d9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 2s 0us/step\n"
     ]
    }
   ],
   "source": [
    "imdb = keras.datasets.imdb\n",
    "(train_data, train_labels), (test_data, test_labels) = imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "yW4QXgQ5RvyT",
    "outputId": "ccb35059-db4a-43c4-e6c9-0ca1f91544a2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/imdb_word_index.json\n",
      "1646592/1641221 [==============================] - 3s 2us/step\n"
     ]
    }
   ],
   "source": [
    "# A dictionary mapping words to an integer index\n",
    "word_index = imdb.get_word_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9B2aL5SQRvya"
   },
   "outputs": [],
   "source": [
    "# The first indices are reserved\n",
    "word_index = {k:(v+3) for k,v in word_index.items()} \n",
    "word_index[\"<PAD>\"] = 0\n",
    "word_index[\"<START>\"] = 1\n",
    "word_index[\"<UNK>\"] = 2  # unknown\n",
    "word_index[\"<UNUSED>\"] = 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "vDzVpPDRRvye"
   },
   "outputs": [],
   "source": [
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "BfNhSVNRRvyg"
   },
   "outputs": [],
   "source": [
    "def decode_review(text):\n",
    "    return ' '.join([reverse_word_index.get(i, '?') for i in text])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ei-qD5NTRvyj"
   },
   "outputs": [],
   "source": [
    "pad_length = 256\n",
    "\n",
    "train_data = keras.preprocessing.sequence.pad_sequences(train_data,\n",
    "                                                           value=word_index[\"<PAD>\"],\n",
    "                                                           padding='post',\n",
    "                                                           maxlen=pad_length)\n",
    "\n",
    "test_data = keras.preprocessing.sequence.pad_sequences(test_data,\n",
    "                                                          value=word_index[\"<PAD>\"],\n",
    "                                                          padding='post',\n",
    "                                                          maxlen=pad_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9Gb6x96rRvyn"
   },
   "outputs": [],
   "source": [
    "def attention(layer):\n",
    "    # --- Attention is all you need --- #\n",
    "    _,_,units = layer.shape.as_list()\n",
    "    attention = Dense(1, activation='tanh')(layer)\n",
    "    attention = Flatten()(attention)\n",
    "    attention = Activation('softmax')(attention)\n",
    "    attention = RepeatVector(units)(attention)\n",
    "    attention = Permute([2, 1])(attention)\n",
    "    representation = multiply([layer, attention])\n",
    "    representation = Lambda(lambda x: K.sum(x, axis=-2), \n",
    "                            output_shape=(units,))(representation)\n",
    "    # ---------------------------------- #\n",
    "    return representation\n",
    "\n",
    "def create_model(vocab_size = 10000, embedding_size = 256, multigpu = False):\n",
    "\n",
    "    inputs = layers.Input(name='inputs',shape=[pad_length])\n",
    "    layer  = layers.Embedding(vocab_size, embedding_size, input_length=pad_length)(inputs)\n",
    "    layer  = layers.SpatialDropout1D(0.3)(layer)\n",
    "    layer  = layers.Bidirectional(layers.CuDNNGRU(128, return_sequences=True))(layer)\n",
    "\n",
    "    sent_representation_1 = attention(layer)\n",
    "\n",
    "    conv = layers.Conv1D(filters=128, kernel_size=4, \n",
    "                         padding='valid', kernel_initializer='he_uniform')(layer)\n",
    "    avg_pool_conv = layers.GlobalAveragePooling1D()(conv)\n",
    "    max_pool_conv = layers.GlobalMaxPooling1D()(conv)\n",
    "\n",
    "    layer  = layers.Bidirectional(layers.CuDNNGRU(64, return_sequences=True))(layer)\n",
    "\n",
    "    sent_representation_2 = attention(layer)\n",
    "\n",
    "    layer = layers.concatenate([sent_representation_1, sent_representation_2,\n",
    "                                avg_pool_conv, max_pool_conv])\n",
    "\n",
    "    layer = layers.Dropout(0.3)(layer)\n",
    "    layer = layers.Dense(32, name='FC1')(layer)\n",
    "    layer  = layers.LeakyReLU()(layer)\n",
    "    layer = layers.Dropout(0.3)(layer)\n",
    "    layer  = layers.Dense(1, name='out_layer')(layer)\n",
    "    outputs  = layers.Activation('sigmoid')(layer)\n",
    "\n",
    "    model  = models.Model(inputs=inputs, outputs=outputs)\n",
    "    model.summary()\n",
    "    \n",
    "    if multigpu:\n",
    "        model = multi_gpu_model(model, gpus=2)\n",
    "\n",
    "    model.compile(optimizer=Adam(),\n",
    "                  loss='binary_crossentropy',\n",
    "                  metrics=['acc'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "v5EH_nC-Rvys"
   },
   "outputs": [],
   "source": [
    "early_stopping = EarlyStopping(monitor='val_loss', \n",
    "                               mode = 'min', \n",
    "                               patience=3, \n",
    "                               verbose=1)\n",
    "\n",
    "model_checkpoint = ModelCheckpoint('imdb.model', \n",
    "                                   monitor='val_acc', \n",
    "                                   mode = 'max', \n",
    "                                   save_best_only=True, \n",
    "                                   verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1329
    },
    "colab_type": "code",
    "id": "jGrKfKwxRvyv",
    "outputId": "852a2cae-e310-4c23-f69b-c897ee535933"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Colocations handled automatically by placer.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:3445: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.\n",
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "inputs (InputLayer)             (None, 256)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "embedding_1 (Embedding)         (None, 256, 256)     2560000     inputs[0][0]                     \n",
      "__________________________________________________________________________________________________\n",
      "spatial_dropout1d_1 (SpatialDro (None, 256, 256)     0           embedding_1[0][0]                \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_1 (Bidirectional) (None, 256, 256)     296448      spatial_dropout1d_1[0][0]        \n",
      "__________________________________________________________________________________________________\n",
      "bidirectional_2 (Bidirectional) (None, 256, 128)     123648      bidirectional_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dense_1 (Dense)                 (None, 256, 1)       257         bidirectional_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "dense_2 (Dense)                 (None, 256, 1)       129         bidirectional_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "flatten_1 (Flatten)             (None, 256)          0           dense_1[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "flatten_2 (Flatten)             (None, 256)          0           dense_2[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "activation_1 (Activation)       (None, 256)          0           flatten_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_2 (Activation)       (None, 256)          0           flatten_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "repeat_vector_1 (RepeatVector)  (None, 256, 256)     0           activation_1[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "repeat_vector_2 (RepeatVector)  (None, 128, 256)     0           activation_2[0][0]               \n",
      "__________________________________________________________________________________________________\n",
      "permute_1 (Permute)             (None, 256, 256)     0           repeat_vector_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "permute_2 (Permute)             (None, 256, 128)     0           repeat_vector_2[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "multiply_1 (Multiply)           (None, 256, 256)     0           bidirectional_1[0][0]            \n",
      "                                                                 permute_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "multiply_2 (Multiply)           (None, 256, 128)     0           bidirectional_2[0][0]            \n",
      "                                                                 permute_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "conv1d_1 (Conv1D)               (None, 253, 128)     131200      bidirectional_1[0][0]            \n",
      "__________________________________________________________________________________________________\n",
      "lambda_1 (Lambda)               (None, 256)          0           multiply_1[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "lambda_2 (Lambda)               (None, 128)          0           multiply_2[0][0]                 \n",
      "__________________________________________________________________________________________________\n",
      "global_average_pooling1d_1 (Glo (None, 128)          0           conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "global_max_pooling1d_1 (GlobalM (None, 128)          0           conv1d_1[0][0]                   \n",
      "__________________________________________________________________________________________________\n",
      "concatenate_1 (Concatenate)     (None, 640)          0           lambda_1[0][0]                   \n",
      "                                                                 lambda_2[0][0]                   \n",
      "                                                                 global_average_pooling1d_1[0][0] \n",
      "                                                                 global_max_pooling1d_1[0][0]     \n",
      "__________________________________________________________________________________________________\n",
      "dropout_1 (Dropout)             (None, 640)          0           concatenate_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "FC1 (Dense)                     (None, 32)           20512       dropout_1[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "leaky_re_lu_1 (LeakyReLU)       (None, 32)           0           FC1[0][0]                        \n",
      "__________________________________________________________________________________________________\n",
      "dropout_2 (Dropout)             (None, 32)           0           leaky_re_lu_1[0][0]              \n",
      "__________________________________________________________________________________________________\n",
      "out_layer (Dense)               (None, 1)            33          dropout_2[0][0]                  \n",
      "__________________________________________________________________________________________________\n",
      "activation_3 (Activation)       (None, 1)            0           out_layer[0][0]                  \n",
      "==================================================================================================\n",
      "Total params: 3,132,227\n",
      "Trainable params: 3,132,227\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = create_model()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 513
    },
    "colab_type": "code",
    "id": "1GFMoGO6Rvyy",
    "outputId": "fe1bdfa1-d1cb-4e5a-e099-ae950f5568bc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_ops.py:3066: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.cast instead.\n",
      "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:102: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n",
      "Train on 15000 samples, validate on 10000 samples\n",
      "Epoch 1/30\n",
      "15000/15000 [==============================] - 15s 980us/step - loss: 0.6648 - acc: 0.5785 - val_loss: 0.4914 - val_acc: 0.7609\n",
      "\n",
      "Epoch 00001: val_acc improved from -inf to 0.76090, saving model to imdb.model\n",
      "Epoch 2/30\n",
      "15000/15000 [==============================] - 10s 644us/step - loss: 0.3711 - acc: 0.8391 - val_loss: 0.2959 - val_acc: 0.8729\n",
      "\n",
      "Epoch 00002: val_acc improved from 0.76090 to 0.87290, saving model to imdb.model\n",
      "Epoch 3/30\n",
      "15000/15000 [==============================] - 10s 654us/step - loss: 0.2072 - acc: 0.9209 - val_loss: 0.3002 - val_acc: 0.8789\n",
      "\n",
      "Epoch 00003: val_acc improved from 0.87290 to 0.87890, saving model to imdb.model\n",
      "Epoch 4/30\n",
      "15000/15000 [==============================] - 10s 663us/step - loss: 0.1351 - acc: 0.9525 - val_loss: 0.4293 - val_acc: 0.8501\n",
      "\n",
      "Epoch 00004: val_acc did not improve from 0.87890\n",
      "Epoch 5/30\n",
      "15000/15000 [==============================] - 10s 676us/step - loss: 0.1052 - acc: 0.9621 - val_loss: 0.4235 - val_acc: 0.8722\n",
      "\n",
      "Epoch 00005: val_acc did not improve from 0.87890\n",
      "Epoch 00005: early stopping\n"
     ]
    }
   ],
   "source": [
    "x_val = train_data[:10000]\n",
    "partial_x_train = train_data[10000:]\n",
    "\n",
    "y_val = train_labels[:10000]\n",
    "partial_y_train = train_labels[10000:]\n",
    "\n",
    "history = model.fit(partial_x_train,\n",
    "                    partial_y_train,\n",
    "                    epochs=30,\n",
    "                    batch_size=512,\n",
    "                    validation_data=(x_val, y_val),\n",
    "                    callbacks=[early_stopping, model_checkpoint],\n",
    "                    shuffle=True,\n",
    "                    verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "id": "Vt7PhfNSRvy0",
    "outputId": "d7d93306-29d4-450a-d70f-60e0561cc48d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 18s 731us/step\n",
      "Loss 0.313 | Accuracy 0.871\n"
     ]
    }
   ],
   "source": [
    "model =keras.models.load_model('imdb.model', custom_objects={})\n",
    "results = model.evaluate(test_data, test_labels)\n",
    "print(\"Loss %0.3f | Accuracy %0.3f\" % tuple(results))\n",
    "\n",
    "history_dict = history.history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 573
    },
    "colab_type": "code",
    "id": "iAUk2xRoRvy4",
    "outputId": "1d96c732-8619-48a5-af07-188d323c5fd0"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEWCAYAAACJ0YulAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XucTfX++PHX28ygGHedcgsRxn1M\nKOR6nG6InKIklZx8k0p1kpSObpIQqZMu6kQkUlJyOhGpjuuRkkTFL/dLuSVleP/++KwZ29gzs+ey\n99oz+/18PPbDvqy91nvWmP3e63N5f0RVMcYYYwCK+B2AMcaY6GFJwRhjTDpLCsYYY9JZUjDGGJPO\nkoIxxph0lhSMMcaks6Rg8pWIxInIYRGplp/b+klEaolIvo/dFpFOIrI54PEGEWkTyra5ONZLIjIs\nt+/PYr+Pisir+b1f4594vwMw/hKRwwEPzwR+B457j/+mqtNysj9VPQ6UzO9tY4Gq1smP/YhIf6CP\nqrYL2Hf//Ni3KfwsKcQ4VU3/UPa+ifZX1f9ktr2IxKtqaiRiM8ZEnjUfmSx5zQNvish0ETkE9BGR\nC0XkvyKyX0R2iMgEEUnwto8XERWR6t7jqd7r80XkkIh8ISI1crqt9/qlIvKdiBwQkYki8pmI9Msk\n7lBi/JuIbBKRX0RkQsB740RknIjsE5EfgEuyOD8PiMiMDM9NEpGx3v3+IrLe+3m+977FZ7avrSLS\nzrt/poi87sW2DmiWYdvhIvKDt991ItLVe74h8CzQxmua2xtwbh8OeP+t3s++T0TeEZFzQjk32RGR\n7l48+0VkoYjUCXhtmIhsF5GDIvJtwM/aUkRWe8/vEpGnQj2eCQNVtZvdUFWAzUCnDM89CvwBdMF9\niTgDuABogbvSrAl8Bwzyto8HFKjuPZ4K7AVSgATgTWBqLrY9CzgEdPNeGwIcA/pl8rOEEuO7QGmg\nOvBz2s8ODALWAVWA8sAS96cS9Dg1gcNAiYB97wZSvMddvG0E6AD8BjTyXusEbA7Y11agnXd/DPAJ\nUBY4F/gmw7ZXA+d4v5NrvRj+5L3WH/gkQ5xTgYe9+529GJsAxYHngIWhnJsgP/+jwKve/XpeHB28\n39EwYIN3vz6wBTjb27YGUNO7vwLo7d1PBFr4/bcQyze7UjChWKqq76nqCVX9TVVXqOoyVU1V1R+A\nyUDbLN4/S1VXquoxYBruwyin214BrFHVd73XxuESSFAhxviEqh5Q1c24D+C0Y10NjFPVraq6DxiV\nxXF+AL7GJSuAPwO/qOpK7/X3VPUHdRYCHwNBO5MzuBp4VFV/UdUtuG//gcedqao7vN/JG7iEnhLC\nfgGuA15S1TWqehQYCrQVkSoB22R2brLSC5irqgu939EoXGJpAaTiElB9rwnyR+/cgUvutUWkvKoe\nUtVlIf4cJgwsKZhQ/BT4QETqisj7IrJTRA4CI4EKWbx/Z8D9I2TduZzZtpUC41BVxX2zDirEGEM6\nFu4bblbeAHp796/1HqfFcYWILBORn0VkP+5belbnKs05WcUgIv1E5EuvmWY/UDfE/YL7+dL3p6oH\ngV+AygHb5OR3ltl+T+B+R5VVdQNwN+73sNtrjjzb2/RGIAnYICLLReSyEH8OEwaWFEwoMg7HfAH3\n7biWqpYCHsI1j4TTDlxzDgAiIpz6IZZRXmLcAVQNeJzdkNmZQCcRqYy7YnjDi/EMYBbwBK5ppwzw\n7xDj2JlZDCJSE3geGAiU9/b7bcB+sxs+ux3XJJW2v0RcM9W2EOLKyX6L4H5n2wBUdaqqtsI1HcXh\nzguqukFVe+GaCJ8GZotI8TzGYnLJkoLJjUTgAPCriNQD/haBY84DkkWki4jEA3cAFcMU40zgThGp\nLCLlgfuy2lhVdwJLgVeBDaq60XupGFAU2AMcF5ErgI45iGGYiJQRN49jUMBrJXEf/Htw+fEW3JVC\nml1AlbSO9SCmAzeLSCMRKYb7cP5UVTO98spBzF1FpJ137Htx/UDLRKSeiLT3jvebdzuB+wGuF5EK\n3pXFAe9nO5HHWEwuWVIwuXE3cAPuD/4FXIdwWKnqLuAaYCywDzgP+B9uXkV+x/g8ru3/K1wn6KwQ\n3vMGruM4velIVfcDdwFzcJ21PXHJLRQjcFcsm4H5wL8C9rsWmAgs97apAwS2w38EbAR2iUhgM1Da\n+z/ENePM8d5fDdfPkCequg53zp/HJaxLgK5e/0IxYDSuH2gn7srkAe+tlwHrxY1uGwNco6p/5DUe\nkzvimmaNKVhEJA7XXNFTVT/1Ox5jCgu7UjAFhohc4jWnFAMexI1aWe5zWMYUKpYUTEHSGvgB1zTx\nF6C7qmbWfGSMyQVrPjLGGJPOrhSMMcakK3AF8SpUqKDVq1f3OwxjjClQVq1atVdVsxrGDRTApFC9\nenVWrlzpdxjGGFOgiEh2M/MBaz4yxhgTwJKCMcaYdJYUjDHGpCtwfQrGmMg6duwYW7du5ejRo36H\nYkJQvHhxqlSpQkJCZqWvsmZJwRiTpa1bt5KYmEj16tVxxWlNtFJV9u3bx9atW6lRo0b2bwjCmo+M\nMVk6evQo5cuXt4RQAIgI5cuXz9NVnSUFY0y2LCEUHHn9XcVOUtiwAe6/H6yshzHGZCp2ksIHH8Co\nUfDPf/odiTEmB/bt20eTJk1o0qQJZ599NpUrV05//McfoS27cOONN7Jhw4Yst5k0aRLTpk3Lj5Bp\n3bo1a9asyZd9RVrsdDTfcQd89BHcdRe0agWNGvkdkTEmBOXLl0//gH344YcpWbIk99xzzynbqCqq\nSpEiwb/nTpkyJdvj3HbbbXkPthCInSuFIkXg1VehbFno1Qt+/dXviIwxebBp0yaSkpK47rrrqF+/\nPjt27GDAgAGkpKRQv359Ro4cmb5t2jf31NRUypQpw9ChQ2ncuDEXXnghu3fvBmD48OGMHz8+ffuh\nQ4fSvHlz6tSpw+effw7Ar7/+ylVXXUVSUhI9e/YkJSUl2yuCqVOn0rBhQxo0aMCwYcMASE1N5frr\nr09/fsKECQCMGzeOpKQkGjVqRJ8+ffL9nIUidq4UAM46C6ZOhT//2V05vPSS3xEZU7DceSfkd7NI\nkybgfRjn1Lfffsu//vUvUlJSABg1ahTlypUjNTWV9u3b07NnT5KSkk55z4EDB2jbti2jRo1iyJAh\nvPLKKwwdOvS0fasqy5cvZ+7cuYwcOZIPP/yQiRMncvbZZzN79my+/PJLkpOTs4xv69atDB8+nJUr\nV1K6dGk6derEvHnzqFixInv37uWrr74CYP/+/QCMHj2aLVu2ULRo0fTnIi12rhTSdOwIw4bByy/D\n9Ol+R2OMyYPzzjsvPSEATJ8+neTkZJKTk1m/fj3ffPPNae8544wzuPTSSwFo1qwZmzdvDrrvHj16\nnLbN0qVL6dWrFwCNGzemfv36Wca3bNkyOnToQIUKFUhISODaa69lyZIl1KpViw0bNjB48GAWLFhA\n6dKlAahfvz59+vRh2rRpuZ58llexdaWQ5uGH4ZNP4G9/g+bN4bzz/I7ImIIhl9/ow6VEiRLp9zdu\n3MgzzzzD8uXLKVOmDH369Ak6Xr9o0aLp9+Pi4khNTQ2672LFimW7TW6VL1+etWvXMn/+fCZNmsTs\n2bOZPHkyCxYsYPHixcydO5fHH3+ctWvXEhcXl6/Hzk7sXSkAxMfDG29AXBz07g0hjmAwxkSvgwcP\nkpiYSKlSpdixYwcLFizI92O0atWKmTNnAvDVV18FvRIJ1KJFCxYtWsS+fftITU1lxowZtG3blj17\n9qCq/PWvf2XkyJGsXr2a48ePs3XrVjp06MDo0aPZu3cvR44cyfefITuxeaUAUK0avPIK9OgBDzwA\nTz3ld0TGmDxITk4mKSmJunXrcu6559KqVat8P8btt99O3759SUpKSr+lNf0EU6VKFR555BHatWuH\nqtKlSxcuv/xyVq9ezc0334yqIiI8+eSTpKamcu2113Lo0CFOnDjBPffcQ2JiYr7/DNkpcGs0p6Sk\naL4usnPbbfDcc24eg9fOaIw5af369dSrV8/vMKJCamoqqampFC9enI0bN9K5c2c2btxIfHx0fb8O\n9jsTkVWqmpLJW9JF10/ih6efhqVLoW9f+PJLqFTJ74iMMVHq8OHDdOzYkdTUVFSVF154IeoSQl4V\nrp8mN4oXhxkzICUFrr8e/v1v19dgjDEZlClThlWrVvkdRljFZkdzRvXqwbPPwsKF8OSTfkdjjDG+\nsaSQpl8/NxLpoYfgs8/8jsYYY3xhSSGNiCuWd+65cO218PPPfkdkjDERZ0khUKlS8OabsGMH9O9v\nZbaNMTHHkkJGKSmuxPacOfD8835HY0zMa9++/WkT0caPH8/AgQOzfF/JkiUB2L59Oz179gy6Tbt2\n7chuiPv48eNPmUR22WWX5UtdoocffpgxY8bkeT/5zZJCMHfe6eYsDBkCa9f6HY0xMa13797MmDHj\nlOdmzJhB7969Q3p/pUqVmDVrVq6PnzEpfPDBB5QpUybX+4t2lhSCSSuzXa4cXHONldk2xkc9e/bk\n/fffT19QZ/PmzWzfvp02bdqkzxtITk6mYcOGvPvuu6e9f/PmzTRo0ACA3377jV69elGvXj26d+/O\nb7/9lr7dwIED08tujxgxAoAJEyawfft22rdvT/v27QGoXr06e/fuBWDs2LE0aNCABg0apJfd3rx5\nM/Xq1eOWW26hfv36dO7c+ZTjBLNmzRpatmxJo0aN6N69O7/88kv68dNKaacV4lu8eHH6IkNNmzbl\n0KFDuT63wdg8hcykldnu1AkGD3ZVVY2JcX5Uzi5XrhzNmzdn/vz5dOvWjRkzZnD11VcjIhQvXpw5\nc+ZQqlQp9u7dS8uWLenatWum6xQ///zznHnmmaxfv561a9eeUvr6scceo1y5chw/fpyOHTuydu1a\nBg8ezNixY1m0aBEVKlQ4ZV+rVq1iypQpLFu2DFWlRYsWtG3blrJly7Jx40amT5/Oiy++yNVXX83s\n2bOzXB+hb9++TJw4kbZt2/LQQw/xj3/8g/HjxzNq1Ch+/PFHihUrlt5kNWbMGCZNmkSrVq04fPgw\nxYsXz8HZzp5dKWSlQwdXF+mVV1wBPWOMLwKbkAKbjlSVYcOG0ahRIzp16sS2bdvYtWtXpvtZsmRJ\n+odzo0aNaBSwAuPMmTNJTk6madOmrFu3Lttid0uXLqV79+6UKFGCkiVL0qNHDz799FMAatSoQZMm\nTYCsy3ODW99h//79tG3bFoAbbriBJUuWpMd43XXXMXXq1PSZ061atWLIkCFMmDCB/fv35/uM6rBe\nKYjIJcAzQBzwkqqOCrLN1cDDgAJfquq14Ywpx0aMgEWL4NZboUULK7NtYppflbO7devGXXfdxerV\nqzly5AjNmjUDYNq0aezZs4dVq1aRkJBA9erVg5bLzs6PP/7ImDFjWLFiBWXLlqVfv3652k+atLLb\n4EpvZ9d8lJn333+fJUuW8N577/HYY4/x1VdfMXToUC6//HI++OADWrVqxYIFC6hbt26uY80obFcK\nIhIHTAIuBZKA3iKSlGGb2sD9QCtVrQ/cGa54ci2tzHZ8vFvG08psGxNxJUuWpH379tx0002ndDAf\nOHCAs846i4SEBBYtWsSWLVuy3M/FF1/MG95V/9dff81abyDJwYMHKVGiBKVLl2bXrl3Mnz8//T2J\niYlB2+3btGnDO++8w5EjR/j111+ZM2cObdq0yfHPVrp0acqWLZt+lfH666/Ttm1bTpw4wU8//UT7\n9u158sknOXDgAIcPH+b777+nYcOG3HfffVxwwQV8++23OT5mVsJ5pdAc2KSqPwCIyAygGxB4TXYL\nMElVfwFQ1d1hjCf3qlVzfQo9erhV26JwGJkxhV3v3r3p3r37KSORrrvuOrp06ULDhg1JSUnJ9hvz\nwIEDufHGG6lXrx716tVLv+Jo3LgxTZs2pW7dulStWvWUstsDBgzgkksuoVKlSixatCj9+eTkZPr1\n60fz5s0B6N+/P02bNs2yqSgzr732GrfeeitHjhyhZs2aTJkyhePHj9OnTx8OHDiAqjJ48GDKlCnD\ngw8+yKJFiyhSpAj169dPX0Uuv4StdLaI9AQuUdX+3uPrgRaqOihgm3eA74BWuCamh1X1wyD7GgAM\nAKhWrVqz7L4NhM2gQTBpErz/Plx2mT8xGBNhVjq74MlL6Wy/O5rjgdpAO6A38KKInDYAWFUnq2qK\nqqZUrFgxwiEGGDMGGjWCG26A7dv9i8MYY8IknElhG1A14HEV77lAW4G5qnpMVX/EXTXUDmNMeVO8\nuCuDceQI9OkDx4/7HZExxuSrcCaFFUBtEakhIkWBXsDcDNu8g7tKQEQqAOcDP4QxpryrW9eV2V60\nyJXDMCYGFLQVGmNZXn9XYUsKqpoKDAIWAOuBmaq6TkRGikhXb7MFwD4R+QZYBNyrqvvCFVO+6dfP\nVVIdMcKt2mZMIVa8eHH27dtniaEAUFX27duXpwlttkZzbh08CMnJbojqmjWuJIYxhdCxY8fYunVr\nnsbtm8gpXrw4VapUISEh4ZTnbY3mcCtVyi3jedFFcPPN8Pbbbk0GYwqZhIQEatSo4XcYJkL8Hn1U\nsKWkuOU733nHymwbYwoFSwp5deedbs7CkCHw5Zd+R2OMMXliSSGvRKzMtjGm0LCkkB8qVoRp0+C7\n7+D22/2Oxhhjcs2SQn5p3x6GD4cpU6zMtjGmwLKkkJ8eeghat3Zltjdt8jsaY4zJMUsK+Sk+3jUj\nWZltY0wBZUkhv1Wr5lZqW7UK7r/f72iMMSZHLCmEw5VXujLbY8e6MtvGGFNAWFIIl6eegsaNXZ0k\nK7NtjCkgLCmEi5XZNsYUQJYUwqlOHbdS26JF8MQTfkdjjDHZsqQQbjfcANdd58psewtzG2NMtLKk\nEG4irlhezZpuDYaff/Y7ImOMyZQlhUhITHRltnftgptuggK2hoUxJnZYUoiUZs1cme1334XnnvM7\nGmOMCcqSQiTdeSdcfrkrs71mjd/RGGPMaSwpRJKIK5hXoYIrg3H4sN8RGWPMKSwpRJqV2TbGRDFL\nCn5o186V2X71VZcgjDEmSlhS8IuV2TbGRCFLCn6Jj3eL8SQkuGU8f//d74iMMcaSgq+qVnUdz6tX\nW5ltY0xUsKTgt27dXIfzuHEwb57f0RhjYpwlhWgwevTJMtvbtvkdjTEmhoU1KYjIJSKyQUQ2icjQ\nIK/3E5E9IrLGu/UPZzxRK63M9tGjVmbbGOOrsCUFEYkDJgGXAklAbxFJCrLpm6raxLu9FK54ol5a\nme1PPoHHH/c7GmNMjArnlUJzYJOq/qCqfwAzgG5hPF7B17evu1J4+GErs22M8UU4k0Jl4KeAx1u9\n5zK6SkTWisgsEakabEciMkBEVorIyj179oQj1ugg4orlpZXZ3rfP74iMMTHG747m94DqqtoI+Ah4\nLdhGqjpZVVNUNaVixYoRDTDirMy2McZH4UwK24DAb/5VvOfSqeo+VU2btfUS0CxcwRw6BO+9F669\n57NmzdyIpLlzXT+DMcZESDiTwgqgtojUEJGiQC9gbuAGInJOwMOuwPpwBTNqFHTtCmPHhusI+eyO\nO+CKK+Duu63MtjEmYsKWFFQ1FRgELMB92M9U1XUiMlJEunqbDRaRdSLyJTAY6BeueB56CHr2dJ+x\nf/97AWiVCSyzfc01VmbbGBMRolH/6XiqlJQUXblyZa7ee/w4DB7s+nJvuAFefNGVHopqixdDhw5w\n/fWuqqoxxuSCiKxS1ZTstvO7ozmi4uLg2Wdh5Eh47TW48kr49Ve/o8pG27auzPZrr8HUqX5HY4wp\n5GIqKYBrlXnwQXjhBfjwQ+jUqQCM/HzwQWjTBgYOhI0b/Y7GGFOIxVxSSDNgALz1Fvzvf+7z9qef\nsn+Pb+Lj3WI8RYu6ZTytzLYxJkxiNikA9OgBCxa4GnQXXQTffON3RFkILLM99LQyUsYYky9iOimA\na7JfsgRSU91CaF984XdEWeja1ZXZHj++AE26MMYUJDGfFMBVrf78cyhfHjp2hPff9zuiLIweDU2a\nwI03WpltY0y+s6TgqVEDPvsMkpLcujevBS24EQWKF3dlMI4eheuuszLbxph8ZUkhwFlnwaJF0L69\nW+9m9OgoneRWp46bbLF4MTz2mN/RGGMKEUsKGSQmuuajXr3gvvvgnnvgxAm/owqib183oe0f/3Cd\nIsYYkw8sKQRRtKgbAXr77a5WUt++8McffkcVxKRJVmbbGJOvLClkokgReOYZtwjatGlu4E/UlR9K\nTHTLeO7e7Tqeo7KtyxhTkFhSyIII3H8/vPQSfPSRG5m0d6/fUWWQnAxPPeWGqD77rN/RGGMKOEsK\nIbj5ZpgzB9audXMZtmzxO6IMBg92ZbbvucdN0TbGmFyypBCirl3h3/92C6JddBF8/bXfEQVIK7Nd\nsaIrs33okN8RGWMKKEsKOdCmzcmBPm3awNKl/sZzigoVXOfH99/DoEF+R2OMKaAsKeRQw4Zu9vOf\n/gR//rNbMTNqtG3rKqr+61/w+ut+R2OMKYAsKeTCuee6q4RGjaB7d3j5Zb8jCjB8OFx8sSuz/d13\nfkdjjClgLCnkUoUK8PHH7mqhf3944okoGRGaVma7WDErs22MybF4vwMoyEqWdM1HN90Ew4bBzp0w\nbpyb4+CrKlVcx3O3bm5a9vjxPgdkzKmOH3dfqo4edd9fgt2KFj39Od//tmKAJYU8KlrUNeGfdZZL\nCLt3u2J6RYv6HFjXrm6o6jPPuAkWXbr4HJAxzhdfwG235W70dHx81kkju6QS6us5eW98IfsULWQ/\njj+KFIGnn4azz3ZfzPfuhbffdhOOfTV6NHz6qZvtvGaNu4Iwxie7d7v1oaZMgcqV3ZLjdeu6Fs7A\n2x9/nP5cTl8/dCjr148dy7+fq0iRyCWkhg3delvhZEkhn4jA3//urhj694cOHVxhvbPO8jGoYsVc\nme3kZFdme+FCiIvzMSATi44fh3/+042BOHzY/Z08+KBrfvXLiROZJ5fskk5uE9cvv2T9eij11Z5/\nHm69NbznxpJCPuvXz3VCX321m/28YIFbq8E355/v/if17QuPPgojRvgYjIk1gU1FHTvCxIlQr57f\nUblv98WLu1u0UD2ZLDJLOueeG/44RKNiyEzoUlJSdOXKlX6Hka3PP3eVJ4oVgw8/dKu7+apvXzcq\naeFCN5/BmDDK2FQ0bhz07OmuqI0/RGSVqqZkt5315YfJRRe5uQzx8W7awOLFPgc0aRKcd55rRrIy\n2yZMjh93/9Xq1HHzJ++7D779Fv76V0sIBYUlhTBKSnJXDJUrw1/+4orq+SYx0fUv7NljZbZNWHz+\nOaSkuCorKSnw1VcwapS/fQcm5ywphFnVqm4AUNOm7vJ58mQfgwkssz1xoo+BmMIkbTmPVq3cyLuZ\nM13xyLp1/Y7M5EZISUFEzhORYt79diIyWETKhPC+S0Rkg4hsEpGhWWx3lYioiGTb3lUQlS8P//kP\nXHIJ/O1v8MgjPn5Rv/12N2fh3nth9WqfgjCFQWqqW8Lj/PNdd9V998H69dZUVNCFeqUwGzguIrWA\nyUBV4I2s3iAiccAk4FIgCegtIklBtksE7gCW5SDuAqdECXjnHdff+9BD7rP5+HEfAgkss92rl5XZ\nNrny2Weuiej22+GCC9xaI9ZUVDiEmhROqGoq0B2YqKr3Audk857mwCZV/UFV/wBmAN2CbPcI8CRw\nNMRYCqyEBHj1VfclfdIk6N3bp9JE5cufLLN9220+BGAKql273LDr1q3deIW33rKmosIm1KRwTER6\nAzcA87znErJ5T2Xgp4DHW73n0olIMlBVVd/PakciMkBEVorIyj179oQYcnQScRONx4xxf1CXXQYH\nD/oQSNu27pLl9dddnQ5jspCa6rqh6tSBN95ww03Xr7dhpoVRqEnhRuBC4DFV/VFEagB5KtgvIkWA\nscDd2W2rqpNVNUVVUypWrJiXw0aNu+92n8dLlkC7du4bWMQNH+6Sw//9n5XZNplKayoaPBiaN3ej\nip54wpqKCquQkoKqfqOqg1V1uoiUBRJV9cls3rYN1/eQpor3XJpEoAHwiYhsBloCcwtrZ3Mwffq4\nKqsbNriRG99/H+EA4uJcAZpixdwynlZm2wQI1lS0YIG7WjCFV6ijjz4RkVIiUg5YDbwoImOzedsK\noLaI1BCRokAvIH2dMlU9oKoVVLW6qlYH/gt0VdXon66cjy691E0y3r/fJYbcVI7MkypVXEfHmjWu\nKI2JecGair791pqKYkWozUelVfUg0AP4l6q2ADpl9QavY3oQsABYD8xU1XUiMlJEuuYl6MKmRQs3\n+7loUdeas2hRhAPo0gXuuAMmTIiy9UVNpC1dCs2and5UVKKE35GZSAk1KcSLyDnA1ZzsaM6Wqn6g\nquer6nmq+pj33EOqetonj6q2i7WrhEB167oZodWqufkMs2ZFOIAnn3Qz7G68EbZujfDBjd927YIb\nboA2bVw1z1mzrKkoVoWaFEbivvF/r6orRKQmsDF8YcWmKlXc7OcLLnBVVp9/PoIHL1YM3nzTlWe8\n9lrXhmAKvdRUd4F4/vkwfTrcf78bVXTVVdZUFKtC7Wh+S1UbqepA7/EPqnpVeEOLTWXLunHfV1zh\nBgWNGBHB2c+1a8Nzz7nM9OijETqo8cunn7qmojvugJYtXVPR449bU1GsC7WjuYqIzBGR3d5ttojY\nMl5hcuaZbuW2m26CkSPdohoRm/18/fVu2vUjj8Ann0TooCaSdu50v+KLL3ZNRbNnu/Lu1lRkIPTm\noym4kUOVvNt73nMmTOLj4aWX3OX85MmunszRSM35DiyzvW5dhA5qwi011S3ZXaeOaykcNsw1FfXo\nYU1F5qRQk0JFVZ2iqqne7VWgcMwii2Ii7nJ+/HhXdvsvf3FDV8OuZEk3KD011Q1BeT1P8xRNFPj0\nU1ck98474cILXVPRY49ZU5E5XahJYZ+I9BGROO/WB7CVWiLkjjvcePEvvnBDVnfsiMBBGzd2kyZS\nUlxbw4ABEbxUMfll507XInjxxXDggGuWnD/fdSwbE0yoSeEm3HDUncAOoCfQL0wxmSB694Z589ys\n54sugo2RGPtVqRJ8/LGbvfTtbh+ZAAAWwUlEQVTii+4rZsSnXZvcSE11V5h16rj1DR54wDUVde9u\nTUUma6GOPtqiql1VtaKqnqWqVwI2+ijCOnd2E9sOH3aznyOyVHV8vJu99N57sGWLa4N4++0IHNjk\n1pIl7td0110uj3/9tRtMduaZfkdmCoK8rLw2JN+iMCG74AJXoOzMM6F9e/joowgd+Ior3KI8deq4\nQexDhrg5DSZq7Njhmoratj21qah2bb8jMwVJXpKCXYT65Pzz3eznGjXg8svd0ssRUb2667EcNAjG\njXPlXX/6Kbt3mTCzpiKTn/KSFGzldx9VquSaCVq2dP0NEyZE6MDFirlqaTNmuCEsTZu6Qe7GF0uW\nuF/BXXe5JkVrKjJ5lWVSEJFDInIwyO0Qbr6C8VGZMq4+zZVXuhFKDzwQwdnP11wDq1a57HTZZfDg\ngz6tLxqbduxwpdfbtnWLNM2ZAx98YE1FJu+yTAqqmqiqpYLcElU1PlJBmsydcYabUnDLLW5Owy23\nRLBs0fnnw3//64ruP/qo6wn3ZbWg2HHsmGu5q1PH/d6HD3dNRVdeaU1FJn/kpfnIRIn4eHjhBfdl\n/eWXXT/wb79F6OBnngmvvOJun3/u2jKWLInQwWNL2qiiIUPcwjdff+2qkVhTkclPlhQKCRFXJ2ni\nRDd6tHNnV9cmYm68EZYtc7OhO3RwpbhPnIhgAIVXYFPRoUPwzjvw/vvWVGTCw5JCITNokOsDXrbM\nzWLdti379+SbRo3c5IkePdyEt27d4OefIxhA4ZKxqejBB+Gbb9xptaYiEy6WFAqhq69249M3b3az\nn7/9NoIHL1XKVVubONH1gicnw4oVEQygcFi82LXEpTUVrVvnrgStqciEmyWFQqpjR/fBcvSo+1BZ\nvjyCBxdxlyxLl7rhUK1awbPPRnBoVMG1fbsrTtuunZu5ntZUVKuW35GZWGFJoRBLTnazn0uXdrOf\nIz6doHlzV1Svc2e4/XY3oeLQoQgHUTAcOwZjx7qmotmzranI+MeSQiFXq5ZLDLVrQ5cuMG1ahAMo\nVw7mznX1k956y1Vd/eqrCAcR3T75xDUV3X236wf6+mtrKjL+saQQA84+2zUltW7tRrGMGxfhAIoU\ncR3PCxe6mVYtWsCrr0Y4iOizfbtbDrt9e/j1V3j3XVcJ15qKjJ8sKcSI0qVd53NaLbv77vOhib9t\nW9ec1LKlG8J6880RnFARPY4dg6efdk1Fb78NDz3kmoq6drWmIuM/SwoxpHhxNzDo1lth9Gj3uXzs\nWISDOPtsV9r1gQfchLeWLeG77yIchH8++QSaNIF77nFNRevWwT/+4WamGxMNLCnEmLg4eO45ePhh\neO01V0nzyBEfgnj0UVesZ9s218/w1lsRDiKyApuKjhw52VR03nl+R2bMqSwpxCARGDECnn/efS53\n6uTTHLNLL3XNSfXru8kVgwcXujUarKnIFDSWFGLYrbe6L+irVrlOaF+WRqha1fWC33mnm/DWpo1b\n4a0QWLToZFNR27bWVGQKhrAmBRG5REQ2iMgmERka5PVbReQrEVkjIktFJCmc8ZjTXXWVm3i8bZub\n/bx+vQ9BFC3qhkTNmuWmXzdt6mZsFVDbtrkpGR06uKaiuXOtqcgUHGFLCiISB0wCLgWSgN5BPvTf\nUNWGqtoEGA2MDVc8JnPt2rkv68eOuSuGL77wKZCrrnKXLdWqueU/hw2LYB3wvDt2DMaMgbp13foG\nI0a4pqIuXfyOzJjQhXNNhObAJlX9AUBEZgDdgG/SNlDVgwHbl8BWc/NNkyau8nXnzq5ExltvuaU+\nI65WLZeVBg92E94+/xymT4dzzjltU1X3QRzK7Y8/Qt82t+/99lvYuNElgfHjoWZNH86fMXkUzqRQ\nGQhspd4KtMi4kYjcBgwBigIdgu1IRAYAAwCqVauW74Eap2ZNN/v5sstceYVnnnHJIvIfsGdw7NiL\nHDtnDMeW7ONYleMcK/M7f1DslO0itdBbkSKQkOBauRISMr+dc47rVLYrA1OQ+b56mqpOAiaJyLXA\ncOCGINtMBiYDpKSk2NVEGP3pT66DtEcPV9MuL7L6AM14K1rULf9csmTA80mlSfhVSVi6kISf95HQ\noikJLZqRUFROeV9OjpOb9xSx4RgmhoQzKWwDqgY8ruI9l5kZwPNhjMeEqFQpN1R1yRK3Tk5uPnDj\n4vJryGUZONwZBgyA6QOg3KXw+utQvnx+7NwYk0E4k8IKoLaI1MAlg17AtYEbiEhtVd3oPbwc2IiJ\nCkWLuvkLUaFkSVfJr00bN3S1aVOYOdPNhjbG5KuwXRiraiowCFgArAdmquo6ERkpIl29zQaJyDoR\nWYPrVzit6cgYwF12DBzoOj3i4lyNiGeesTUajMlnogXsjyolJUVXrlzpdxjGT7/8Av36uQkAV10F\nL7/sKv4ZYzIlIqtUNSW77awLzRQ8Zcu6JclGj3b/pqTAl1/6HZUxhYIlBVMwicC997qyo0eOuP6F\nl1+25iRj8siSginYWrd2RfVat4b+/V2z0q+/+h2VMQWWJQVT8J11lluAesQIN1y1RQs3vdgYk2OW\nFEzhEBfnFon48EPYtQsuuABmzPA7KmMKHEsKpnDp3Nk1JzVu7EqV3nYb/P6731EZU2BYUjCFT5Uq\nrlbHPfe4ZeZat4Yff/Q7KmMKBEsKpnBKSICnnnI1rDduhORkeO89v6MyJupZUjCF25VXwurVrgRs\n165w332uxKoxJihLCqbwS6sJfuutbsJbhw5ueTRjzGksKZjYULw4PP+8K6z3v/+5onr/+Y/fURkT\ndSwpmNhy7bWwYgVUrOhGKo0c6eqDG2MASwomFtWrB8uXQ58+bsLbpZfCnj1+R2VMVLCkYGJTiRLw\n2msweTIsXuyakz77zO+ojPGdJQUTu0Tgllvgiy9cn0O7djB2rBXVMzHNkoIxTZvCqlXQpQvcfbdb\noHr/fr+jMsYXlhSMAbdIz+zZ7kph3jxo1szNbzAmxlhSMCaNCNx1l+tj+OMPuOgieOEFa04yMcWS\ngjEZXXSRm8vQrp2b8Hb99XD4sN9RGRMRlhSMCaZCBfjgA3jkEZg+HZo3h2++8TsqY8LOkoIxmSlS\nBIYPh48+gn373BoNU6f6HZUxYWVJwZjsdOjgmpOaNXNNSX/7Gxw96ndUxoSFJQVjQlGpEixc6Kqs\nTp7s+h2+/97vqIzJd5YUjAlVfDyMGuXWZdi82V05zJnjd1TG5CtLCsbk1BVXuDkMtWu7iW53321r\nNJhCw5KCMblRvTosXQqDBrkJb+3awdatfkdlTJ6FNSmIyCUiskFENonI0CCvDxGRb0RkrYh8LCLn\nhjMeY/JVsWIwcSLMmAFr10Ljxu6q4b//tQlvpsAKW1IQkThgEnApkAT0FpGkDJv9D0hR1UbALGB0\nuOIxJmyuuQZWroTWreHZZ+HCC92VxD33wLJlliBMgRLOK4XmwCZV/UFV/wBmAN0CN1DVRap6xHv4\nX6BKGOMxJnzq1IF334Vdu1xJ7kaNYMIEaNkSatRwCWL5cksQJuqFMylUBn4KeLzVey4zNwPzg70g\nIgNEZKWIrNxji6GYaFamDPTt60Yo7d7tEkSDBi5BtGjhEsS997rV3yxBmCgUFR3NItIHSAGeCva6\nqk5W1RRVTalYsWJkgzMmt9ISxLx57gri1Vehfn0YP96VzahZE/7+d9f0ZAnCRIlwJoVtQNWAx1W8\n504hIp2AB4Cuqvp7GOMxxj9ly8INN8D777sriClT3LKg48a58hnnnecmxq1aZQnC+CqcSWEFUFtE\naohIUaAXMDdwAxFpCryASwi7wxiLMdGjbFno188V3Nu1C155xfVJjB0LKSlQqxYMHermQliCMBEW\ntqSgqqnAIGABsB6YqarrRGSkiHT1NnsKKAm8JSJrRGRuJrszpnAqVw5uvBHmz3cJ4uWX3aS4MWPc\njOlateD++13tJUsQJgJEC9h/tJSUFF25cqXfYRgTXvv2wTvvwMyZ8PHHcPy4a2K6+mp3a9zYLQpk\nTIhEZJWqpmS7nSUFY6Lc3r0nE8TChS5B1Kp1MkE0amQJwmTLkoIxhdHeva4I38yZsGiRSxC1a59M\nEA0bWoIwQVlSMKaw27Pn1ARx4gScf75LDn/9qyUIcwpLCsbEkt27TyaITz5xCaJOnZMJokEDSxAx\nzpKCMbFq9254+22XIBYvdgmibt2TTUz16/sdofGBJQVjjBvmmpYglixxCaJevZNXEJYgYoYlBWPM\nqXbudAnirbfcFYQqJCWdTBBJGYsYm8LEkoIxJnM7d8Ls2S5BLFniEkT9+icTRL16fkdo8pklBWNM\naHbsOJkgPv3UJYgGDU4miLp1/Y7Q5ANLCsaYnNu+/WSCWLrUJYiGDU8miDp1/I7Q5JIlBWNM3mzb\ndmqCADd7Oi1BnH++v/GZHAk1KUTFegrGmChUuTIMHuyalH76ya0DkZgIw4e7K4YmTeDxx2HjRr8j\nNfnIkoIxJntVqsAdd7grhrQEUaIEPPCAu2Jo2tQSRCFhScEYkzNpCeKzz+D//T+3UNAZZ5xMEMnJ\n8MQTsGmT35GaXLCkYIzJvapV4c474fPPXYIYOxaKFYNhw1yhvmbNYNQo+P57vyM1IbKkYIzJH1Wr\nwl13wRdfwJYt8PTTkJDgFgmqVcutKvfkk/DDD35HarJgo4+MMeG1ZQvMmuVKbSxf7p5r1gx69nR9\nEbVqwbnnQny8v3EWcjYk1RgTfTZvPpkgVqw4+Xx8vEsMtWq5FeYC/61Rw/VZmDyxpGCMiW47d8J3\n37n+hk2bTv67aRMcOHDqtlWqnJ4s0v4tVcqf+AuYUJOCXa8ZY/xx9tnudvHFpz6vCj//fGqSSLs/\nb56r/BqoQgWXHIIljAoVbB2JHLKkYIyJLiJQvry7NW9++uuHDrnO6sBk8f33rrDftGkuqaRJTAye\nLGrVgkqVoIiNtcnIkoIxpmBJTITGjd0to99/hx9/PL1J6ssv4d134dixk9sWK+aSRLBmqXPPdSOn\nYpAlBWNM4VGsmKvqGqyya2qqm42dMWF8/z385z/w228nt42LO73jO+1+zZqFuuPbkoIxJjbEx7uR\nTDVqQKdOp76m6jq+MyaLTZvcMNr9+0/dvnLl4M1S550HpUtH7mcKA0sKxhgjAuec425t2pz++s8/\nB08YH3zgkkmgChUyHylVsWLUd3xbUjDGmOyUK+c6vYN1fB8+HLzje+lSeOON0zu+M0sYlStHRce3\nJQVjjMmLkiXdOhONGp3+2u+/uwl7GRPG2rXBO75r1jy9DyNtxneEOr7DmhRE5BLgGSAOeElVR2V4\n/WJgPNAI6KWqs8IZjzHGRFSxYm7tiWAr1h0/nnnH98KFcOTIyW3TOr4ffRR69w5ryGFLCiISB0wC\n/gxsBVaIyFxV/SZgs/8H9APuCVccxhgTleLioHp1d+vY8dTXVN0kvYzJ4qyzwh5WOK8UmgObVPUH\nABGZAXQD0pOCqm72XjsRxjiMMaZgETk547t164geOpy9GpWBnwIeb/WeyzERGSAiK0Vk5Z49e/Il\nOGOMMafzv6s7BKo6WVVTVDWlYsWKfodjjDGFVjiTwjagasDjKt5zxhhjolQ4k8IKoLaI1BCRokAv\nYG4Yj2eMMSaPwpYUVDUVGAQsANYDM1V1nYiMFJGuACJygYhsBf4KvCAi68IVjzHGmOyFdZ6Cqn4A\nfJDhuYcC7q/ANSsZY4yJAgWio9kYY0xkWFIwxhiTrsCt0Swie4AtuXx7BWBvPoaTXyyunLG4ci5a\nY7O4ciYvcZ2rqtmO6S9wSSEvRGRlKAtXR5rFlTMWV85Fa2wWV85EIi5rPjLGGJPOkoIxxph0sZYU\nJvsdQCYsrpyxuHIuWmOzuHIm7HHFVJ+CMcaYrMXalYIxxpgsWFIwxhiTrtAlBRF5RUR2i8jXmbwu\nIjJBRDaJyFoRSY6SuNqJyAERWePdHgq2XRjiqioii0TkGxFZJyJ3BNkm4ucsxLgifs5EpLiILBeR\nL724/hFkm2Ii8qZ3vpaJSPUoiaufiOwJOF/9wx1XwLHjROR/IjIvyGsRP18hxuXn+dosIl95x10Z\n5PXw/U2qaqG6ARcDycDXmbx+GTAfEKAlsCxK4moHzPPhfJ0DJHv3E4HvgCS/z1mIcUX8nHnnoKR3\nPwFYBrTMsM3/Af/07vcC3oySuPoBz0b6/5h37CHAG8F+X36crxDj8vN8bQYqZPF62P4mC92Vgqou\nAX7OYpNuwL/U+S9QRkTOiYK4fKGqO1R1tXf/EK6ibcYV8iJ+zkKMK+K8c3DYe5jg3TKO1ugGvObd\nnwV0FBGJgrh8ISJVgMuBlzLZJOLnK8S4olnY/iYLXVIIQb4tExoGF3qX//NFpH6kD+5dtjfFfcsM\n5Os5yyIu8OGceU0Oa4DdwEeqmun5UldC/gBQPgriArjKa26YJSJVg7weDuOBvwOZrcXuy/kKIS7w\n53yBS+j/FpFVIjIgyOth+5uMxaQQrVbjapM0BiYC70Ty4CJSEpgN3KmqByN57KxkE5cv50xVj6tq\nE1zZ9+Yi0iASx81OCHG9B1RX1UbAR5z8dh42InIFsFtVV4X7WDkRYlwRP18BWqtqMnApcJuIXByp\nA8diUojKZUJV9WDa5b+6dSgSRKRCJI4tIgm4D95pqvp2kE18OWfZxeXnOfOOuR9YBFyS4aX08yUi\n8UBpYJ/fcanqPlX93Xv4EtAsAuG0ArqKyGZgBtBBRKZm2MaP85VtXD6dr7Rjb/P+3Q3MAZpn2CRs\nf5OxmBTmAn293vuWwAFV3eF3UCJydlo7qog0x/1uwv5B4h3zZWC9qo7NZLOIn7NQ4vLjnIlIRREp\n490/A/gz8G2GzeYCN3j3ewIL1esd9DOuDG3OXXH9NGGlqverahVVrY7rRF6oqn0ybBbx8xVKXH6c\nL++4JUQkMe0+0BnIOGoxbH+TYV15zQ8iMh03KqWCuKU+R+A63VDVf+JWgrsM2AQcAW6Mkrh6AgNF\nJBX4DegV7j8MTyvgeuArrz0aYBhQLSA2P85ZKHH5cc7OAV4TkThcEpqpqvNEZCSwUlXn4pLZ6yKy\nCTe4oFeYYwo1rsHilsJN9eLqF4G4goqC8xVKXH6drz8Bc7zvO/HAG6r6oYjcCuH/m7QyF8YYY9LF\nYvORMcaYTFhSMMYYk86SgjHGmHSWFIwxxqSzpGCMMSadJQVjPCJyPKAi5hoRGZqP+64umVTINSaa\nFLp5CsbkwW9emQhjYpZdKRiTDa+2/Wivvv1yEanlPV9dRBZ6BdM+FpFq3vN/EpE5XqG+L0XkIm9X\ncSLyorj1Dv7tzTxGRAaLWzdirYjM8OnHNAawpGBMoDMyNB9dE/DaAVVtCDyLq64Jrgjfa17BtGnA\nBO/5CcBir1BfMrDOe742MElV6wP7gau854cCTb393BquH86YUNiMZmM8InJYVUsGeX4z0EFVf/CK\n9O1U1fIishc4R1WPec/vUNUKIrIHqBJQTC2t/PdHqlrbe3wfkKCqj4rIh8BhXJXXdwLWRTAm4uxK\nwZjQaCb3c+L3gPvHOdmndzkwCXdVscKrFGqMLywpGBOaawL+/cK7/zkni7ddB3zq3f8YGAjpC9+U\nzmynIlIEqKqqi4D7cGWjT7taMSZS7BuJMSedEVCRFeBDVU0bllpWRNbivu339p67HZgiIvcCezhZ\nqfIOYLKI3Iy7IhgIZFbWOA6Y6iUOASZ46yEY4wvrUzAmG16fQoqq7vU7FmPCzZqPjDHGpLMrBWOM\nMensSsEYY0w6SwrGGGPSWVIwxhiTzpKCMcaYdJYUjDHGpPv/x97NFD08eJAAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEWCAYAAACXGLsWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBo\ndHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3XmczfX+wPHX21gG2ZcW0qgkQmhC\nF5VIWnBTieqWFOVS0tWiq5JuorpKN7mJuu0SLSgk6Uq/G0bZZclSgzT2PWbm/fvj8x1zZszMOcOc\n8z1n5v18PM7DOd/lfN/zHXPe57OLqmKMMcbkpZjfARhjjIl+liyMMcYEZcnCGGNMUJYsjDHGBGXJ\nwhhjTFCWLIwxxgRlycKETETiRGS/iNQqyGP9JCLnikiB9x8XkXYisjHg9WoRaR3KsSdwrXEi8tiJ\nnm9MKIr7HYAJHxHZH/CyDPAHkOa9vkdV38vP+6lqGnBKQR9bFKhq3YJ4HxG5G7hNVS8PeO+7C+K9\njcmLJYtCTFWPfVh731zvVtWvcjteRIqramokYjMmGPv/GF2sGqoIE5F/iMiHIvKBiOwDbhORS0Tk\nexHZLSJbReRlESnhHV9cRFREErzX73r7p4vIPhH5n4jUzu+x3v6rRWSNiOwRkX+JyHci0iOXuEOJ\n8R4RWSciu0Tk5YBz40TkRRHZISLrgQ553J+/i8iEbNtGi8hI7/ndIrLK+3l+9r715/ZeySJyufe8\njIi848W2Argo27GDRWS9974rRKSTt70h8ArQ2qvi2x5wb4cEnH+v97PvEJFPReT0UO5Nfu5zRjwi\n8pWI7BSR30Tk4YDrPO7dk70ikiQiZ+RU5Sci8zJ+z979nOtdZycwWETqiMgc7xrbvftWIeD8s7yf\nMcXbP0pE4r2Y6wUcd7qIHBSRKrn9vCYIVbVHEXgAG4F22bb9AzgCdMR9cSgNXAw0x5U6zwbWAP28\n44sDCiR4r98FtgOJQAngQ+DdEzi2OrAP6OztexA4CvTI5WcJJcbPgApAArAz42cH+gErgJpAFWCu\n+zPI8TpnA/uBsgHv/TuQ6L3u6B0jwBXAIaCRt68dsDHgvZKBy73nLwDfAJWAs4CV2Y7tCpzu/U5u\n8WI41dt3N/BNtjjfBYZ4z9t7MTYG4oFXga9DuTf5vM8VgG1Af6AUUB5o5u0bBCwB6ng/Q2OgMnBu\n9nsNzMv4PXs/WyrQB4jD/X88D2gLlPT+n3wHvBDw8yz37mdZ7/iW3r6xwDMB1/kb8Inff4ex/PA9\nAHtE6Bede7L4Osh5A4GPvOc5JYB/BxzbCVh+Asf2BL4N2CfAVnJJFiHG2CJg/8fAQO/5XFx1XMa+\na7J/gGV77++BW7znVwOr8zh2GtDXe55Xsvgl8HcB/DXw2Bzedzlwrfc8WLJ4CxgWsK88rp2qZrB7\nk8/7/BdgYS7H/ZwRb7btoSSL9UFiuDHjukBr4DcgLofjWgIbAPFeLwa6FPTfVVF6WDWU+TXwhYic\nLyKfe9UKe4GhQNU8zv8t4PlB8m7Uzu3YMwLjUPfXnZzbm4QYY0jXAjblES/A+0B37/kt3uuMOK4T\nkfleFclu3Lf6vO5VhtPzikFEeojIEq8qZTdwfojvC+7nO/Z+qroX2AXUCDgmpN9ZkPt8Ji4p5CSv\nfcFk//94mohMFJHNXgz/yRbDRnWdKbJQ1e9wpZRWItIAqAV8foIxGazNwrhvmoFew32TPVdVywNP\n4L7ph9NW3DdfAEREyPrhlt3JxLgV9yGTIVjX3olAOxGpgasme9+LsTQwCXgWV0VUEfgyxDh+yy0G\nETkbGIOriqnive9PAe8brJvvFlzVVsb7lcNVd20OIa7s8rrPvwLn5HJebvsOeDGVCdh2WrZjsv98\nI3C9+Bp6MfTIFsNZIhKXSxxvA7fhSkETVfWPXI4zIbBkYbIrB+wBDngNhPdE4JrTgKYi0lFEiuPq\nwauFKcaJwAMiUsNr7Hwkr4NV9TdcVcl/cFVQa71dpXD16ClAmohch6tbDzWGx0SkorhxKP0C9p2C\n+8BMweXNXriSRYZtQM3AhuZsPgDuEpFGIlIKl8y+VdVcS2p5yOs+TwFqiUg/ESklIuVFpJm3bxzw\nDxE5R5zGIlIZlyR/w3WkiBOR3gQktjxiOADsEZEzcVVhGf4H7ACGies0UFpEWgbsfwdXbXULLnGY\nk2DJwmT3N+AOXIPza7iG6LBS1W3AzcBI3B//OcCPuG+UBR3jGGA2sAxYiCsdBPM+rg3iWBWUqu4G\nBgCf4BqJb8QlvVA8iSvhbASmE/BBpqpLgX8BC7xj6gLzA86dBawFtolIYHVSxvkzcNVFn3jn1wJu\nDTGu7HK9z6q6B7gSuAGXwNYAl3m7nwc+xd3nvbjG5niverEX8Bius8O52X62nDwJNMMlrSnA5IAY\nUoHrgHq4UsYvuN9Dxv6NuN/zH6r6f/n82U02GY0/xkQNr1phC3Cjqn7rdzwmdonI27hG8yF+xxLr\nbFCeiQoi0gHX8+gQruvlUdy3a2NOiNf+0xlo6HcshYFVQ5lo0QpYj6urvwq43hokzYkSkWdxYz2G\nqeovfsdTGFg1lDHGmKCsZGGMMSaoQtNmUbVqVU1ISPA7DGOMiSmLFi3arqp5dVUHClGySEhIICkp\nye8wjDEmpohIsFkMAKuGMsYYEwJLFsYYY4KyZGGMMSaoQtNmkZOjR4+SnJzM4cOH/Q7F5CE+Pp6a\nNWtSokRu0x0ZY/xWqJNFcnIy5cqVIyEhATeRqYk2qsqOHTtITk6mdu3awU8wxviiUFdDHT58mCpV\nqliiiGIiQpUqVaz0Z0yUK9TJArBEEQPsd2RM9CvU1VDGGFMoHD0Ku3bB7t3u34xHxuuqVaF377CG\nYMkijHbs2EHbtm49nN9++424uDiqVXMDJRcsWEDJkiWDvsedd97Jo48+St26dXM9ZvTo0VSsWJFb\nbz3RZQuMMWGlCocO5f5hH+z1gQN5v/8ll1iyiGVVqlRh8eLFAAwZMoRTTjmFgQMHZjnm2GLoxXKu\nEXzzzTeDXqdv374nH6wxJm+qsG9f/j/oMx5HjuT9/uXKQaVKmY9zz836umLF3F+XKhX2H9+ShQ/W\nrVtHp06daNKkCT/++COzZs3iqaee4ocffuDQoUPcfPPNPPHEEwC0atWKV155hQYNGlC1alXuvfde\npk+fTpkyZfjss8+oXr06gwcPpmrVqjzwwAO0atWKVq1a8fXXX7Nnzx7efPNN/vSnP3HgwAFuv/12\nVq1aRf369dm4cSPjxo2jcePGWWJ78skn+eKLLzh06BCtWrVizJgxiAhr1qzh3nvvZceOHcTFxfHx\nxx+TkJDAsGHD+OCDDyhWrBjXXXcdzzzzjB+31JjQpKW5D/FgH+y5ffinp+f+3sWKZX6AZ/xbs2bw\nD/pKlaBCBSge3R/HYY3OW9BmFBAHjFPV4dn2nwW8gVtveSdwW8ZawSKShlsSEeAXVe10UsE88AB4\n3/ILTOPG8NJLJ3TqTz/9xNtvv01iYiIAw4cPp3LlyqSmptKmTRtuvPFG6tevn+WcPXv2cNlllzF8\n+HAefPBB3njjDR599NHj3ltVWbBgAVOmTGHo0KHMmDGDf/3rX5x22mlMnjyZJUuW0LRp0xzj6t+/\nP0899RSqyi233MKMGTO4+uqr6d69O0OGDKFjx44cPnyY9PR0pk6dyvTp01mwYAGlS5dm586dJ3Qv\njMmXI0dO7IN+1y7Yuzfv9y5RIuuHeLVqUKdO1m25ffiXK+cSRiEVtmThLY05GrdObzKwUESmqOrK\ngMNeAN5W1bdE5Arc4vJ/8fYdUtWsX3sLkXPOOedYogD44IMPGD9+PKmpqWzZsoWVK1celyxKly7N\n1VdfDcBFF13Et9/mvOJoly5djh2zceNGAObNm8cjjzwCwIUXXsgFF1yQ47mzZ8/m+eef5/Dhw2zf\nvp2LLrqIFi1asH37djp27Ai4QXQAX331FT179qR06dIAVK5c+URuhTHOwYOwaBEsWAC//JL7B/+h\nQ3m/T5kyWT/Ia9WCCy8MrTqndGmw3nk5CmfJohmwTlXXA4jIBNwSh4HJoj7woPd8Dm6R9/A4wRJA\nuJQtW/bY87Vr1zJq1CgWLFhAxYoVue2223IcdxDYIB4XF0dqamqO713Kq7/M65icHDx4kH79+vHD\nDz9Qo0YNBg8ebOMfTHikp8OqVTB/vksO8+fDsmWumghctUzgB3nduqFV51SsCCF0HDH5F85kUQP4\nNeB1MtA82zFLgC64qqrrgXIiUkVVdwDxIpIEpALDVfW4RCIivYHeALVq1Sr4nyBC9u7dS7ly5Shf\nvjxbt25l5syZdOjQoUCv0bJlSyZOnEjr1q1ZtmwZK1euPO6YQ4cOUaxYMapWrcq+ffuYPHkyt956\nK5UqVaJatWpMnTo1SzXUlVdeyYgRI+jWrduxaigrXZgcbd2aNTEsXOgai8ElhmbNYNAgaN7cPa9e\n3d94zXH8blEZCLwiIj2AucBmwPtqwVmqutlbdP1rEVmmqj8HnqyqY4GxAImJiTG7PmzTpk2pX78+\n559/PmeddRYtW7Ys8Gvcd9993H777dSvX//Yo0KFClmOqVKlCnfccQf169fn9NNPp3nzzNz+3nvv\ncc899/D3v/+dkiVLMnnyZK677jqWLFlCYmIiJUqUoGPHjjz99NMFHruJMQcOZFYnzZ/vHr963xuL\nF3dVQn/5i0sMzZu7NoFCXNdfWIRtDW4RuQQYoqpXea8HAajqs7kcfwrwk6rWzGHff4Bpqjopt+sl\nJiZq9sWPVq1aRb169U74ZyhMUlNTSU1NJT4+nrVr19K+fXvWrl1L8SjpgWG/qxiVlnZ8ddLy5ZnV\nSbVrZyaF5s1dpxCvjctEBxFZpKqJwY4L5yfFQqCOiNTGlRi6AbcEHiAiVYGdqpoODML1jEJEKgEH\nVfUP75iWwHNhjLXQ279/P23btiU1NRVV5bXXXouaRGFiyJYtmaWFBQtcddL+/W5fxYquCqlTJ5cY\nLr7YqpMKkbB9Wqhqqoj0A2bius6+oaorRGQokKSqU4DLgWdFRHHVUBmjy+oBr4lIOm7+quHZelGZ\nfKpYsSKLFi3yOwwTS/bvd9VJgaWG5GS3r0QJV510xx2Z7QxWnVSohfWrpap+AXyRbdsTAc8nAcdV\nLanq/wENwxmbMSZAWhqsXJm11LB8eeYgtLPPhtatMxNDkybgdaE2RYPVQxhTFG3enDUxJCVlVidV\nquQSwp//nFmd5M1pZoouSxbGFHb797tkEJgcNm92+0qUcI3OPXpkrU6ygWkmG0sWxhQmaWmwYkXW\nxLBiRWZ10jnnwGWXZSaGxo2tOsmExFqjwqhNmzbMnDkzy7aXXnqJPn365HneKaecAsCWLVu48cYb\nczzm8ssvJ3tX4exeeuklDh48eOz1Nddcw+7du0MJ3cSK5GSYPBkeftglgQoVXMNz797wySdQowY8\n/jh8/jmkpMC6dfDee3D//dCihSUKEzIrWYRR9+7dmTBhAlddddWxbRMmTOC550LrBXzGGWcwaVKu\nQ0uCeumll7jtttsoU6YMAF988UWQM0xU27fv+OqkLVvcvpIlXSmhZ09XYmje3E1xbdVJpoBYySKM\nbrzxRj7//HOOePPYb9y4kS1bttC6detj4x6aNm1Kw4YN+eyzz447f+PGjTRo0ABwU3F069aNevXq\ncf3113MoYDK1Pn36kJiYyAUXXMCTTz4JwMsvv8yWLVto06YNbdq0ASAhIYHt27cDMHLkSBo0aECD\nBg14yZs3a+PGjdSrV49evXpxwQUX0L59+yzXyTB16lSaN29OkyZNaNeuHdu2bQPcWI4777yThg0b\n0qhRIyZPngzAjBkzaNq0KRdeeOGxxaBMEKmpsGQJjB0Ld90FDRq4UsMVV7hpMZYvhzZt4OWX4fvv\n3Wyq8+e717fdZu0OpsAVmZKFHzOUV65cmWbNmjF9+nQ6d+7MhAkT6Nq1KyJCfHw8n3zyCeXLl2f7\n9u20aNGCTp065boe9ZgxYyhTpgyrVq1i6dKlWaYYf+aZZ6hcuTJpaWm0bduWpUuXcv/99zNy5Ejm\nzJlD1apVs7zXokWLePPNN5k/fz6qSvPmzbnsssuoVKkSa9eu5YMPPuD111+na9euTJ48mdtuuy3L\n+a1ateL7779HRBg3bhzPPfcc//znP3n66aepUKECy5a5meV37dpFSkoKvXr1Yu7cudSuXdumMc+J\nqpsOI3B6jEWL3CysAJUru5LCTTe5UkOzZlClir8xmyKnyCQLv2RURWUki/HjxwNuzYnHHnuMuXPn\nUqxYMTZv3sy2bds47bTTcnyfuXPncv/99wPQqFEjGjVqdGzfxIkTGTt2LKmpqWzdupWVK1dm2Z/d\nvHnzuP7664/NfNulSxe+/fZbOnXqRO3atY8tiBQ4xXmg5ORkbr75ZrZu3cqRI0eoXbs24KYsnzBh\nwrHjKlWqxNSpU7n00kuPHWMTDeJKAYHVSfPnw2+/uX0lS7oxDHffnVmddM45VkowvisyycKvGco7\nd+7MgAED+OGHHzh48CAXXXQR4CbmS0lJYdGiRZQoUYKEhIQTmg58w4YNvPDCCyxcuJBKlSrRo0eP\nk5pWvFTA8oxxcXE5VkPdd999PPjgg3Tq1IlvvvmGIUOGnPD1ioR162D27MzEsGqVK02Aqy5q1y6z\nd9KFF0ZkiUxj8svaLMLslFNOoU2bNvTs2ZPu3bsf275nzx6qV69OiRIlmDNnDps2bcrzfS699FLe\nf/99AJYvX87SpUsBN7152bJlqVChAtu2bWP69OnHzilXrhz7MqaBDtC6dWs+/fRTDh48yIEDB/jk\nk09o3bp1yD/Tnj17qFGjBgBvvfXWse1XXnklo0ePPvZ6165dtGjRgrlz57JhwwaAolUNtXUr9Orl\n1mK4916YMgUSEmDIEJg+HXbsgDVr4J13oF8/lywsUZgoVWRKFn7q3r07119/fZYqmltvvZWOHTvS\nsGFDEhMTOf/88/N8jz59+nDnnXdSr1496tWrd6yEcuGFF9KkSRPOP/98zjzzzCzTm/fu3ZsOHTpw\nxhlnMGfOnGPbmzZtSo8ePWjWrBkAd999N02aNMmxyiknQ4YM4aabbqJSpUpcccUVxxLB4MGD6du3\nLw0aNCAuLo4nn3ySLl26MHbsWLp06UJ6ejrVq1dn1qxZIV0nZu3fD//8Jzz3HBw96rqp9u1r1Ukm\npoVtivJIsynKY1uh+F2lpcGbb7pxDb/95hqkn33WJQljolQ0TFFuTNGgCjNmuIFxy5fDJZe4gXJ/\n+pPfkRlTYKzNwpiTsXgxtG8P11wDhw7BpEnw3XeWKEyhU+iTRWGpZivMYvJ3lJzsJt9r2hR++MF1\nt1u5Em64wdolTKFUqKuh4uPj2bFjB1WqVMl1sJvxl6qyY8cO4mNljqK9e13D9ciRro1i4EB47DG3\nSpwxhVihThY1a9YkOTmZlJQUv0MxeYiPj6dmzeOWXo8uqanw+uvw5JNuQr7u3WHYMNcV1pgiIKzJ\nQkQ6AKNwy6qOU9Xh2fafhVt3uxqwE7hNVZO9fXcAg71D/6Gqb5FPJUqUODZy2JgTogpTp8Ijj8BP\nP8Gll7oZXC++2O/IjImosLVZiEgcMBq4GqgPdBeR+tkOewF4W1UbAUOBZ71zKwNPAs2BZsCTIlIp\nXLEak6OkJDdZX+fOLml8+il8840lClMkhbOBuxmwTlXXq+oRYALQOdsx9YGvvedzAvZfBcxS1Z2q\nuguYBXQIY6zGZNq0yc3cevHFbuGg0aNh2TKXNKztyxRR4UwWNYBfA14ne9sCLQG6eM+vB8qJSJUQ\nz0VEeotIkogkWbuEOWm7d7vqprp13TiJQYPcvE5//atbftSYIszvrrMDgctE5EfgMmAzkBbqyao6\nVlUTVTWxmi0ob07UkSNuHYhzz4Xnn4ebb3ZzNg0b5taQMMaENVlsBs4MeF3T23aMqm5R1S6q2gT4\nu7dtdyjnGnPSVOHjj+GCC6B/fzfj66JF8NZbcOaZwc83pggJZ7JYCNQRkdoiUhLoBkwJPEBEqopI\nRgyDcD2jAGYC7UWkktew3d7bZkzB+P57aN3aDaIrWdL1cPrqK7eWhDHmOGFLFqqaCvTDfcivAiaq\n6goRGSoinbzDLgdWi8ga4FTgGe/cncDTuISzEBjqbTPm5Kxf76qZLrnEtUeMHeuWL73mGmu8NiYP\nhXrWWWOO2bkT/vEPeOUV11g9cCA89BCccorfkRnjK5t11hiAP/5wXV+ffhr27IGePWHoUDjjDL8j\nMyam+N0bypjwUIUPP4R69eBvf3PLli5eDOPGWaIw5gRYsjCFz7x5rk2iWzcoVw5mznTrTTRq5Hdk\nxsQsSxam8FizBrp0cb2cfv0V3njDTR/evr3fkRkT86zNwsS+lBTXDvHvf0N8vGufGDAAypYtsEuo\nwuHDcPAgHDjg/g18Hmzb6ae7pS+aNoWaNa3jlYk9lixM7Dp0yI28HjaMo/sOc7BHPw70f4yDZatx\nYF3+P9CD7c9vx8HixV2+io93+Sw93W2vWjUzcTRp4v49+2woZuV8E8Ws66wJu/R097me3w/t3Lcp\nB7ft40DKIQ6ml+KAnEKq5u97jwiUKeMeZcse/7wgtgVOJ3XwICxd6mrFMh7Ll8PRo25/+fKZiSPj\nUbcuxMUV4C/CmBxY11kTFqmpbhzb2rWhf7gfPpz/65QqlfOHcbmjOzlt0xLK7NlCmSplKHtZImXq\nVsz3B3l8fGSrgsqUgRYt3CPDkSNuUtvABPLvf7vEmnHOhRdmTSD167sB58aAK+1u2+Z6hdetG95r\nWcnChGzrVrdA3H//6zoZhesbeenSrgoni1Wr4OGHYdo0N2/TsGFwyy2Fru4mNRVWr86aQH78Efbt\nc/tLloSGDbNWYTVq5O6ZKbz27XNf0Favdv041qzJfL5vn/sS8r//ndh7h1qysGRhQvL11y5R7N/v\nvv3+5S8RuvC2bTBkiFvStGxZt971/fcXqU/H9HT4+eesCeSHH9ygdHBVVfXqZS2BNG7sErqJHUeP\nwoYNWRNBxvOtWzOPE4GzzoLzzst8NGwIl19+Yte1ZGEKRHo6PPOMW3r6/PNh0iRXFRJ2Bw/CyJEw\nYoSrx7r3XnjiCbCp6AFX/fDLL67UEZhAMj5URKBOnawJpEkTqFzZ37iLOlX3O8peOlizxk1blpqa\neWyVKq5qKSMhZDw/55yC/a5kycKctJQUt2Dcl1+6f8eMicBUSmlp8M478Pe/w5YtcP31MHy4+ysx\nQW3denwC2bQpc39CQtYqrKZN4bTTfAu30Nq7N+eEsGaNK51niI93ST17UqhTxyWLSLAGbnNS5s1z\nA6C3b3cN2nffHYEG4Vmz3AR/S5dCs2YwYYIbYGdCdvrp7nHNNZnbduw4PoF8/HHWcwJLIE2bumYh\nGwuStyNHXGkgp6Tw22+Zx4m4JH3eedCqVdakULNm7DS7WbIwWajCP/8Jjz4KtWu7ZR8aNw7zRZct\nc43XM2a4v6oJE6BrV/u0KiBVqkC7du6RYe9eNzN7YAKZPj1zLEiVKscnkKI4FkQVNm/OWjLISAob\nNriCcIZq1VwSuOaarAnh7LNdCSLWWTWUOWbXLrjjDpg61a0JNH58mFcV3bLFtUO8+aYbaDB4MPTr\n5/rNmog7eNDl7cAEsmxZ1rEgjRsfPxbkuJ5rMWj37pwTwpo17r5kKF36+DaEjEelSv7FfzKsGsrk\ny8KF7sv85s1uUHS/fmH8Yr9/v1vr+oUX3CdR//4uUVjrq6/KlHGT8zZvnrktp7Egr72WORakdOnj\nx4JccEF0jgX54w9XbZRT99Pff888rlgxV6o+7zy47LKsSaFGjaJXuspgJYsiTtUt9/Dgg67ueuLE\nrB8WBSo11ZUinnjCVep27erGS5xzTpguaMIhNdV9wGYfC7J3r9tfokTmWJCMR6TGgqSnuy88OSWE\njRszq9kATj0151LC2WcXrcJtVPSGEpEOwCggDhinqsOz7a8FvAVU9I55VFW/EJEE3FKsq71Dv1fV\ne/O6liWL/Nu7F3r1cgniuuvgrbfC9OVe1VWIP/QQrFwJf/qTK1VcckkYLmb8kJ7uvrVnHwuyY4fb\nHxfnul5nHwtSvvyJXW/XrpzHI6xdm1nqATc0J7CqKCMp1KkDFSue/M9dGPieLEQkDlgDXAkk49bS\n7q6qKwOOGQv8qKpjRKQ+8IWqJnjJYpqqNgj1epYs8mfJErjpJvcHPmyY64QUluL1jz+6JDF7Npx7\nrusG26WLNV4XAapupvjsJZAtWzKPyWksSEaX0cOH3WDE7F1PV692vfQyxMW5aqOcxiSccYb9Vwsm\nGtosmgHrVHW9F9AEoDOwMuAYBTK+W1QAtmDCStUt89CvnytFzJkTpt6pv/7q2iHeece1/I0a5QbW\nRWNltgkLEahVyz3+/OfM7b/9lrUr7/z5blHDDLVquS8umzZlnen3tNNcErj++qxJoXZt+28VCeFM\nFjWAXwNeJwPZa8OHAF+KyH1AWSCgcx+1ReRHYC8wWFW/zX4BEekN9AaoVatWwUVeSB04AH/9K7z9\ntutG+d57UL16AV9k71436nrkSPeX/tBDMGiQlfnNMaedBldf7R4Zdu7MmkDA9cwLHKR2olVWpmD4\n3RuqO/AfVf2niFwCvCMiDYCtQC1V3SEiFwGfisgFqro38GRVHQuMBVcNFengY8mqVa7aaeVKN9XS\n4MEFPP310aNu/qYhQ9zQ71tucfOEJCQU4EVMYVW5MrRt6x4mOoWzE9hm4MyA1zW9bYHuAiYCqOr/\ngHigqqr+oao7vO2LgJ8Bm+/hBL3/Plx8sese+OWXbp6nAksUqvDZZ677S9++buKohQtdscUShTGF\nRjiTxUKgjojUFpGSQDdgSrZjfgHaAohIPVyySBGRal4DOSJyNlAHWB/GWAuljPn3br3VNR4uXpx1\nFO9JW7jQTXWZUSH92WeuESQxaFuZMSbGhC1ZqGoq0A+YiesGO1FVV4jIUBHp5B32N6CXiCwBPgB6\nqOuedSmwVEQWA5OAe1V1Z7hiLYx+/tn1TH3tNXjkETfF+BlnFNCbb9zoMlCzZq5+69VX3VDfTp2s\n64kxhZQNyiuEPv4Y7rzTVTVVFhkkAAAbtElEQVS9/bYbQ1Fgtm1zHeYPH3Yj+R55xFoejYlh0dB1\n1kTYkSNuPr5Ro9yX/okT3SIpBerxx910HT/+CA1CHgZjjIlxRXSWk8Lnl1/g0ktdoujfH779NgyJ\nYulSN7tgv36WKIwpYqxkUQh8/jncfrvrvfrRR3DjjWG4iKqrdqpY0c3tZIwpUqxkEcNSU914t+uu\nc6Nef/ghTIkCYNo0N2XHkCGxOxezMeaEWckiRm3ZAt27w9y50Ls3vPRSGGf1PHLETR51/vmuL64x\npsixZBGDZs92A6T373dTL912W5gvOGaMm8Ft2jQ3/7QxpsixaqgYkpYGQ4fClVdC1apuTFzYE8WO\nHfDUU+6igQs7G2OKFCtZxIjff3eJYdYs+Mtf3Jf9smUjcOGnnoI9e9zEgDbgzpgiy5JFDJg3D26+\n2X3Jf/11uOuuCH1u//STG53dq5d1lTWmiLNqqCiWnu6Wqr78crc+8vffw913R/AL/sCBrvgydGiE\nLmiMiVZWsohSO3dCjx4wdarrDjt+fIRn1Zg1yw3gGDEiDIteGGNijSWLKLRgAXTt6rrHvvyyGzAd\n0eaC1FQ3AK92bTcc3BhT5FmyiCKq8Mor8Le/uRli581zczxF3PjxsHy5Gw5eqpQPARhjoo21WUSJ\nvXtdI/b998NVV7nR2L4kij173GSBrVvDDTf4EIAxJhpZySIKLFni2iU2bIDnnnMli2J+pfFhw9yy\nqF98YV1ljTHHBP1IEpH7RMQmAwoDVRg3Dlq0gIMH4Ztv4KGHfEwU69e7eUNuv91WuzPGZBHKx9Kp\nwEIRmSgiHUTs62ZBOHAA7rjDDWFo1cotD9Gqlc9BPfIIFC/uShfGGBMgaLJQ1cG4NbDHAz2AtSIy\nTETOCXaul1xWi8g6EXk0h/21RGSOiPwoIktF5JqAfYO881aLyFX5+qmi3KpVrj3i3XfdAOkZM6Kg\nd+q338KkSS5h1KjhczDGmGgTUoWHty72b94jFagETBKR53I7R0TigNHA1UB9oLuI1M922GDc2txN\ngG7Aq9659b3XFwAdgFe994t5773nani2b3dDGZ54wi1/6qv0dBgwAGrWdAPxjDEmm1DaLPqLyCLg\nOeA7oKGq9gEuAvLqLtMMWKeq61X1CDAB6JztGAUyhppVALZ4zzsDE1T1D1XdAKzz3i9mHT4M99zj\n5ndKTHTVTm3b+h2V5913YdEiePZZN1TcGGOyCaU3VGWgi6puCtyoqukicl0e59UAfg14nQw0z3bM\nEOBLEbkPKAu0Czj3+2znHlc3IiK9gd4AtWrVCvqD+GXdOrjpJli8GB59FJ5+2jUNRIUDB9wKShdf\n7OY9N8aYHIRSDTUd2JnxQkTKi0hzAFVddZLX7w78R1VrAtcA74hIyH2BVHWsqiaqamK1atVOMpTw\nmDwZLroINm1yy0E8+2wUJQpwk09t2QIvvuhjNyxjTLQL5dNhDLA/4PV+b1swm4EzA17X9LYFuguY\nCKCq/wPigaohnhvVjhyBBx5w4yfq1XPVTtde63dU2SQnu4EdXbtCy5Z+R2OMiWKhJAvxGrgBV/1E\naNVXC4E6IlJbREriGqynZDvmF6AtgIjUwyWLFO+4biJSSkRq43pjLQjhmlFh0yY3AHrUKJcw5s6F\ns87yO6ocPPaYa9weMcLvSIwxUS6UD/31InI/maWJvwLrg52kqqki0g+YCcQBb6jqChEZCiSp6hTg\nb8DrIjIA19jdw0tMK0RkIrAS1/uqr6qm5feH88Pnn7vFidLSXE/UqJ0xY+FCtybro49CQoLf0Rhj\nopwEFBpyPkCkOvAycAXuA3028ICq/h7+8EKXmJioSUlJvl0/NdVNqTR8ODRu7ObgO/dc38LJm6or\n+qxd6x4RnfvcGBNNRGSRqgadsiFoycJLCt0KJKpCassW6NbNjWu75x43Y0Z8vN9R5eGjj+C772Ds\nWEsUxpiQBE0WIhKPa4i+ANemAICq9gxjXDHjq69cj9ODB91whVtv9TuiIA4fdqO0GzWCnvYrNMaE\nJpQG7neA04CrgP/ieibtC2dQsSAtzU3V0b49VKvmmgCiPlGAK/Zs3AgjR0bB0HFjTKwIJVmcq6qP\nAwdU9S3gWo4fXFek/P47XH01DBniRmQvWOC6x0a9bdvcJIEdO0bR8HFjTCwIpTfUUe/f3SLSADc/\nlN/T3vnm229d+8TOnW568Z49Y2jZh8cfh0OH4IUX/I7EGBNjQilZjPXWsxiMG/+wEihyHfPT0934\ntTZtoGxZ+P57uOuuGEoUS5e65VL79oXzzvM7GmNMjMmzZOFNvbFXVXcBc4GzIxJVlNm50609MW2a\nm+Np3LgY60SkCg8+CBUquGlujTEmn/IsWXijtR+OUCxRacECaNIEZs6Ef/0LPvwwxhIFuCw3e7Zr\nZKlc2e9ojDExKJRqqK9EZKCInCkilTMeYY/MZ6rw8stu9ToRNyyhX78YqnbKcOSIW6Oibl3o08fv\naIwxMSqUBu6bvX/7BmxTCnGV1J49cPfdbrqOjh3hP/+J4S/kY8bAmjWudFGihN/RGGNiVCgjuGtH\nIpBosXixa5fYsME1aA8cGIOliQw7d7rBIFdeCddcE/x4Y4zJRSgjuG/Pabuqvl3w4fhH1TVc33cf\nVKkC33zjqqBi2lNPuWLSyJExnPGMMdEglGqoiwOex+OmFP8BKDTJYv9+V53/7rvuS/h777lR2TFt\n9Wp49VXo1QsaNPA7GmNMjAulGuq+wNciUhG3nnahsGmTG439008wdKhb4qFQzIIxcKBbT3voUL8j\nMcYUAieywOcBoNC0Y1SvDjVrum6xhWYGjK++cg3aI0a4H9AYY05SKG0WU3G9n8B1ta2PtxRqYVC6\nNHz5pd9RFKC0NDcAr3Zt6N/f72iMMYVEKCWLwImEUoFNqpocypuLSAdgFG6lvHGqOjzb/heBNt7L\nMkB1Va3o7UsDlnn7flHVTqFcs8gbPx6WLXNrVpQq5Xc0xphCIpRk8QuwVVUPA4hIaRFJUNWNeZ0k\nInHAaOBKIBlYKCJTVHVlxjGqOiDg+PuAJgFvcUhVG4f8kxjX82nwYLcKXtSu52qMiUWhjOD+CEgP\neJ3mbQumGbBOVder6hFco3jnPI7vDnwQwvua3AwbBikp1lXWGFPgQkkWxb0PewC85yVDOK8G8GvA\n62Rv23FE5Cxco/nXAZvjRSRJRL4XkT+HcL2ibf16t7DR7bdDYtDldI0xJl9CSRYpInKsvUBEOgPb\nCziObsAkVU0L2HaWt4j4LcBLInJO9pNEpLeXUJJSUlIKOKQY88gjULy4K10YY0wBCyVZ3As8JiK/\niMgvwCPAPSGctxk4M+B1TW9bTrqRrQpKVTd7/64HviFre0bGMWNVNVFVE6vF/Ci6k/Dtt24iq4cf\nhho5Ft6MMeakhDIo72eghYic4r3eH+J7LwTqiEhtXJLohislZCEi5wOVgP8FbKsEHFTVP0SkKtAS\neC7E6xYt6ekwYIBLEgMH+h2NMaaQClqyEJFhIlJRVfer6n4RqSQi/wh2nqqmAv2AmcAqYKKqrhCR\noYHVWrgkMkFVNWBbPSBJRJYAc4Dhgb2oTIB334VFi+DZZ90SfsYYEwaS9TM6hwNEflTVJtm2/aCq\nTcMaWT4lJiZqUlKS32FE1oEDbonUGjXcOq/FQqlVNMaYTCKyyGsfzlMo4yziRKSUqv7hvXFpwEZ7\nRYPnn4ctW2DiREsUxpiwCiVZvAfMFpE3AQF6AG+FMygTguRkt+BG167QsqXf0RhjCrlQGrhHeG0H\n7XBzRM0Ezgp3YCaIxx5zjdsjRvgdiTGmCAi17mIbLlHcBFyBa7A2flm4EN55x/WCSkjwOxpjTBGQ\na8lCRM7DTcHRHTcI70Ncg3ib3M4xEaDqkkT16jBokN/RGGOKiLyqoX4CvgWuU9V1ACIyII/jTSRM\nmgTffQdjx0L58n5HY4wpIvKqhuoCbAXmiMjrItIW18Bt/HL4sBul3agR9OzpdzTGmCIk15KFqn4K\nfCoiZXGzxT4AVBeRMcAnqlqYlgyKDaNGwcaNbiW8QrH2qzEmVgRt4FbVA6r6vqp2xM3v9CNufigT\nSdu2wTPPQMeOhWj9V2NMrMjXSC5V3eVN3mefVpH2xBNw6BC88ELwY40xpoDZsN9YsHQpjBsHffu6\n6T2MMSbCLFlEO1V48EGoUMGVLowxxgehTPdh/DRtGsye7Rq3K1f2OxpjTBFlJYtoduSIW6Oibl3o\n08fvaIwxRZiVLKLZmDGwZg1MnQolSvgdjTGmCLOSRbTauROeegratYNrr/U7GmNMEWfJIlo99RTs\n2QMjR4LYwHljjL/CmixEpIOIrBaRdSLyaA77XxSRxd5jjYjsDth3h4is9R53hDPOqLN6Nbz6KvTq\nBQ0b+h2NMcaEr81CROKA0cCVQDKwUESmBK6lraoDAo6/D2jiPa8MPAkk4qZGX+Sduytc8UaVgQOh\nTBkYOtTvSIwxBghvyaIZsE5V16vqEWACbo6p3HQHPvCeXwXMUtWdXoKYBXQIY6zR46uvXHfZv//d\nTUNujDFRIJzJogbwa8DrZG/bcUTkLKA28HV+zhWR3iKSJCJJKSkpBRK0r9LS3AC82rWhf3+/ozHG\nmGOipYG7GzBJVdPyc5I3T1WiqiZWq1YtTKFF0PjxsGyZW1u7VCm/ozHGmGPCmSw2A2cGvK7pbctJ\nNzKroPJ7buGwdy88/ji0bg033OB3NMYYk0U4k8VCoI6I1BaRkriEMCX7QSJyPlAJ+F/A5plAexGp\nJCKVgPbetsJr2DD4/XfrKmuMiUph6w2lqqki0g/3IR8HvKGqK0RkKJCkqhmJoxswQVU14NydIvI0\nLuEADFXVneGK1XcbNsCLL8Ltt0Niot/RGGPMcSTgMzqmJSYmalJSkt9hnJiuXeHzz93UHjVy7ANg\njDFhISKLVDXot9RoaeAuuubNg48+cmtrW6IwxkQpSxZ+Sk+HAQNckhg40O9ojDEmVzbrrJ/efReS\nkuDtt6FsWb+jMcaYXFnJwi8HDsCgQa5B+9Zb/Y7GGGPyZCULvzz/PGzZAh9+CMUsZxtjopt9Svkh\nOdmN0r7pJmjVyu9ojDEmKEsWfnjsMTcP1IgRfkdijDEhsWQRaQsXwjvvZE4YaIwxMcCSRSSpuq6y\n1au7xm1jjIkR1sAdSZMmwXffwdixUL6839EYY0zIrGQRKYcPu1HajRpBz55+R2OMMfliJYtIGTUK\nNm50K+HFxfkdjTHG5IuVLCJh2zZ45hno2BHatvU7GmOMyTdLFpHwxBNw6BC88ILfkRhjzAmxZBFu\ny5bBuHHQty+cd57f0RhjzAmxZBFOqm48RYUKrnRhjDExyhq4w+nzz12D9qhRULmy39EYY8wJC2vJ\nQkQ6iMhqEVknIo/mckxXEVkpIitE5P2A7Wkisth7HLd2d9Q7etStUVG3LvTp43c0xhhzUsJWshCR\nOGA0cCWQDCwUkSmqujLgmDrAIKClqu4SkeoBb3FIVRuHK76wGzMGVq+GqVOhRAm/ozHGmJMSzpJF\nM2Cdqq5X1SPABKBztmN6AaNVdReAqv4exngiZ+dOGDIE2rWDa6/1OxpjjDlp4UwWNYBfA14ne9sC\nnQecJyLficj3ItIhYF+8iCR52/+c0wVEpLd3TFJKSkrBRn8ynnoK9uyBkSNBxO9ojDHmpPndwF0c\nqANcDtQE5opIQ1XdDZylqptF5GzgaxFZpqo/B56sqmOBsQCJiYka2dBzsXo1vPoq3H03NGzodzTG\nGFMgwlmy2AycGfC6prctUDIwRVWPquoGYA0ueaCqm71/1wPfAE3CGGvBGTgQSpeGoUP9jsQYYwpM\nOJPFQqCOiNQWkZJANyB7r6ZPcaUKRKQqrlpqvYhUEpFSAdtbAiuJdl99BdOmweDBcOqpfkdjjDEF\nJmzVUKqaKiL9gJlAHPCGqq4QkaFAkqpO8fa1F5GVQBrwkKruEJE/Aa+JSDouoQ0P7EUVldLSMhc0\n6t/f72iMMaZAhbXNQlW/AL7Itu2JgOcKPOg9Ao/5PyC2KvzHj3dTe3z0EZQq5Xc0xhhToGy6j4Kw\ndy88/ji0bg033OB3NMYYU+D87g1VOAwbBr//7qb3sK6yxphCyEoWJ2vDBnjxRbj9dkhM9DsaY4wJ\nC0sWJ+uRR6B4cVe6MMaYQsqSxcmYN881aD/8MNTIPjjdGGMKD0sWJyo9HQYMcEli4EC/ozHGmLCy\nBu4T9d57kJQEb78NZcv6HY0xxoSVlSxOxIEDMGiQa9C+9Va/ozHGmLCzksWJeOEF2LwZJkyAYpZv\njTGFn33S5dfmzfDcc3DTTdCqld/RGGNMRFiyyK/HHoPUVBgxwu9IjDEmYixZ5EdGg/aAAW7CQGOM\nKSIsWYRKFR54AKpXd6ULY4wpQqyBO1STJsF338Frr0H58n5HY4wxEWUli1AcPuxGaTdqBHfd5Xc0\nxhgTcVayCMWoUbBxo1sJLy7O72iMMSbirGQRzLZt8Mwz0LEjtG3rdzTGGOOLsCYLEekgIqtFZJ2I\nPJrLMV1FZKWIrBCR9wO23yEia73HHeGMM09PPAGHDrmBeMYYU0SFrRpKROKA0cCVQDKwUESmBK6l\nLSJ1gEFAS1XdJSLVve2VgSeBRECBRd65u8IVb46WLYNx4+C+++C88yJ6aWOMiSbhLFk0A9ap6npV\nPQJMADpnO6YXMDojCajq7972q4BZqrrT2zcL6BDGWI+nCg8+CBUquNKFMcYUYeFMFjWAXwNeJ3vb\nAp0HnCci34nI9yLSIR/nIiK9RSRJRJJSUlIKMHTcEqlffQVDhkDlygX73sYYE2P8buAuDtQBLge6\nA6+LSMVQT1bVsaqaqKqJ1apVK7iojh51a1TUrQt9+hTc+xpjTIwKZ9fZzcCZAa9retsCJQPzVfUo\nsEFE1uCSx2ZcAgk895uwRZrdmDGwejVMnQolSkTsssYYE63CWbJYCNQRkdoiUhLoBkzJdsyneElB\nRKriqqXWAzOB9iJSSUQqAe29beG3c6eremrXDq69NiKXNMaYaBe2koWqpopIP9yHfBzwhqquEJGh\nQJKqTiEzKawE0oCHVHUHgIg8jUs4AENVdWe4Ys1i6FDYswdGjgSRiFzSGGOinaiq3zEUiMTERE1K\nSjq5N1m9Gho0gJ493RxQxhhTyInIIlVNDHac3w3c0eWhh6B0aVe6MMYYc4zNDZVh9mzXoD18OJx6\nqt/RGGNMVLGSBUBamhuAl5AA/fv7HY0xxkQdK1kAvPEGLF0KEydCfLzf0RhjTNSxksXevTB4MLRq\nBTfe6Hc0xhgTlSxZHDgALVvCiy9aV1ljjMmFVUOdfjp8/LHfURhjTFSzkoUxxpigLFkYY4wJypKF\nMcaYoCxZGGOMCcqShTHGmKAsWRhjjAnKkoUxxpigLFkYY4wJqtCsZyEiKcCmk3iLqsD2AgqnIFlc\n+WNx5Y/FlT+FMa6zVLVasIMKTbI4WSKSFMoCIJFmceWPxZU/Flf+FOW4rBrKGGNMUJYsjDHGBGXJ\nItNYvwPIhcWVPxZX/lhc+VNk47I2C2OMMUFZycIYY0xQliyMMcYEVaSShYi8ISK/i8jyXPaLiLws\nIutEZKmINI2SuC4XkT0isth7PBGhuM4UkTkislJEVohI/xyOifg9CzGuiN8zEYkXkQUissSL66kc\njiklIh9692u+iCRESVw9RCQl4H7dHe64Aq4dJyI/isi0HPZF/H6FEJOf92qjiCzzrpuUw/7w/T2q\napF5AJcCTYHluey/BpgOCNACmB8lcV0OTPPhfp0ONPWelwPWAPX9vmchxhXxe+bdg1O85yWA+UCL\nbMf8Ffi397wb8GGUxNUDeCXS/8e8az8IvJ/T78uP+xVCTH7eq41A1Tz2h+3vsUiVLFR1LrAzj0M6\nA2+r8z1QUUROj4K4fKGqW1X1B+/5PmAVUCPbYRG/ZyHGFXHePdjvvSzhPbL3IOkMvOU9nwS0FQnv\n4u8hxuULEakJXAuMy+WQiN+vEGKKZmH7eyxSySIENYBfA14nEwUfQp5LvGqE6SJyQaQv7hX/m+C+\nlQby9Z7lERf4cM+86ovFwO/ALFXN9X6paiqwB6gSBXEB3OBVXUwSkTPDHZPnJeBhID2X/X7cr2Ax\ngT/3ClyS/1JEFolI7xz2h+3v0ZJFbPgBN3/LhcC/gE8jeXEROQWYDDygqnsjee28BInLl3umqmmq\n2hioCTQTkQaRuG4wIcQ1FUhQ1UbALDK/zYeNiFwH/K6qi8J9rVCFGFPE71WAVqraFLga6Csil0bq\nwpYsstoMBH5LqOlt85Wq7s2oRlDVL4ASIlI1EtcWkRK4D+T3VPXjHA7x5Z4Fi8vPe+ZdczcwB+iQ\nbdex+yUixYEKwA6/41LVHar6h/dyHHBRBMJpCXQSkY3ABOAKEXk32zGRvl9BY/LpXmVce7P37+/A\nJ0CzbIeE7e/RkkVWU4DbvR4FLYA9qrrV76BE5LSMeloRaYb7vYX9A8a75nhglaqOzOWwiN+zUOLy\n456JSDURqeg9Lw1cCfyU7bApwB3e8xuBr9VrmfQzrmz12p1w7UBhpaqDVLWmqibgGq+/VtXbsh0W\n0fsVSkx+3CvvumVFpFzGc6A9kL0HZdj+HosXxJvEChH5ANdLpqqIJANP4hr7UNV/A1/gehOsAw4C\nd0ZJXDcCfUQkFTgEdAv3B4ynJfAXYJlX3w3wGFArIDY/7lkocflxz04H3hKROFxymqiq00RkKJCk\nqlNwSe4dEVmH69TQLcwxhRrX/SLSCUj14uoRgbhyFAX3K1hMft2rU4FPvO9AxYH3VXWGiNwL4f97\ntOk+jDHGBGXVUMYYY4KyZGGMMSYoSxbGGGOCsmRhjDEmKEsWxhhjgrJkYUwQIpIWMMPoYhF5tADf\nO0FymW3YmGhSpMZZGHOCDnlTZRhTZFnJwpgT5K0t8Jy3vsACETnX254gIl97E83NFpFa3vZTReQT\nb3LDJSLyJ++t4kTkdXFrTXzpjbJGRO4Xt2bHUhGZ4NOPaQxgycKYUJTOVg11c8C+ParaEHgFN1sp\nuIkL3/ImmnsPeNnb/jLwX29yw6bACm97HWC0ql4A7AZu8LY/CjTx3ufecP1wxoTCRnAbE4SI7FfV\nU3LYvhG4QlXXexMb/qaqVURkO3C6qh71tm9V1aoikgLUDJiELmOK9VmqWsd7/QhQQlX/ISIzgP24\nGXM/DViTwpiIs5KFMSdHc3meH38EPE8jsy3xWmA0rhSy0Jt11RhfWLIw5uTcHPDv/7zn/0fmhHe3\nAt96z2cDfeDYYkQVcntTESkGnKmqc4BHcFNzH1e6MSZS7JuKMcGVDpjdFmCGqmZ0n60kIktxpYPu\n3rb7gDdF5CEghcyZP/sDY0XkLlwJog+Q2/TRccC7XkIR4GVvLQpjfGFtFsacIK/NIlFVt/sdizHh\nZtVQxhhjgrKShTHGmKCsZGGMMSYoSxbGGGOCsmRhjDEmKEsWxhhjgrJkYYwxJqj/B9rqduWEix0q\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "acc = history_dict['acc']\n",
    "val_acc = history_dict['val_acc']\n",
    "loss = history_dict['loss']\n",
    "val_loss = history_dict['val_loss']\n",
    "\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "# \"bo\" is for \"blue dot\"\n",
    "plt.plot(epochs, loss, 'r', label='Training loss')\n",
    "# b is for \"solid blue line\"\n",
    "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
    "plt.title('Training and validation loss')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Loss')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()\n",
    "\n",
    "plt.clf()   # clear figure\n",
    "\n",
    "plt.plot(epochs, acc, 'r', label='Training acc')\n",
    "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n",
    "plt.title('Training and validation accuracy')\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend()\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2ZIwH5OYRvy7"
   },
   "outputs": [],
   "source": [
    "# Hyperparameters distributions\n",
    "from scipy.stats import randint\n",
    "from scipy.stats import uniform\n",
    "\n",
    "# Model selection\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.model_selection import cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3wVcx9wxRvy9"
   },
   "outputs": [],
   "source": [
    "def create_model(multigpu=False,\n",
    "                 vocab_size = 10000, \n",
    "                 embedding_size = 256, \n",
    "                 dropout_lvl= .3,\n",
    "                 spatial_dropout=True,\n",
    "                 gru_units=128,\n",
    "                 conv_filters=128,\n",
    "                 conv_kernel=4,\n",
    "                 fc_layers=1,\n",
    "                 fc_dense=32):\n",
    "\n",
    "    inputs = layers.Input(name='inputs',shape=[pad_length])\n",
    "    layer  = layers.Embedding(vocab_size, embedding_size, input_length=pad_length)(inputs)\n",
    "    \n",
    "    if spatial_dropout:\n",
    "        layer  = layers.SpatialDropout1D(dropout_lvl)(layer)\n",
    "        \n",
    "    layer  = layers.Bidirectional(layers.CuDNNGRU(int(gru_units), return_sequences=True))(layer)\n",
    "    \n",
    "    sent_representation_1 = attention(layer)\n",
    "    \n",
    "    conv_kernel_tuple = [conv_kernel]\n",
    "    conv = layers.Conv1D(filters=conv_filters, kernel_size=conv_kernel_tuple, \n",
    "                         padding='valid', kernel_initializer='he_uniform')(layer)\n",
    "    \n",
    "    avg_pool_conv = layers.GlobalAveragePooling1D()(conv)\n",
    "    max_pool_conv = layers.GlobalMaxPooling1D()(conv)\n",
    "\n",
    "    layer  = layers.Bidirectional(layers.CuDNNGRU(int(gru_units), return_sequences=True))(layer)\n",
    "    \n",
    "    sent_representation_2 = attention(layer)  \n",
    "\n",
    "    layer = layers.concatenate([sent_representation_1, sent_representation_2,\n",
    "                                avg_pool_conv, max_pool_conv])\n",
    "\n",
    "    layer = layers.Dropout(dropout_lvl)(layer)\n",
    "    \n",
    "    for l in range(fc_layers):\n",
    "        layer = layers.Dense(fc_dense//(l+1), name='FC'+str(l+1))(layer)\n",
    "        \n",
    "    layer  = layers.LeakyReLU()(layer)\n",
    "    layer = layers.Dropout(dropout_lvl)(layer)\n",
    "    layer  = layers.Dense(1, name='out_layer')(layer)\n",
    "    outputs  = layers.Activation('sigmoid')(layer)\n",
    "\n",
    "    model  = models.Model(inputs=inputs, outputs=outputs)\n",
    "    \n",
    "    if multigpu:\n",
    "        model = multi_gpu_model(model, gpus=2)\n",
    "\n",
    "    model.compile(optimizer=Adam(),\n",
    "                  loss='binary_crossentropy',\n",
    "                  metrics=['acc'])\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "YAUEN96LRvzA"
   },
   "outputs": [],
   "source": [
    "clf = KerasClassifier(build_fn=create_model, \n",
    "                      epochs=15, \n",
    "                      batch_size=512,\n",
    "                      multigpu=False,\n",
    "                      vocab_size = len(np.unique(train_data)),\n",
    "                      verbose=0)\n",
    "\n",
    "skf = StratifiedKFold(n_splits=3, shuffle=True, random_state=0)\n",
    "\n",
    "random_search = RandomizedSearchCV(clf, \n",
    "                                   param_distributions={\"epochs\": randint(1, 3),\n",
    "                                                        \"batch_size\": randint(64, 512),\n",
    "                                                        \"embedding_size\": randint(16, 512),\n",
    "                                                        \"dropout_lvl\": uniform(0, 0.5),\n",
    "                                                        \"spatial_dropout\" : randint(0, 1),\n",
    "                                                        \"gru_units\": randint(32, 256),\n",
    "                                                        \"conv_filters\": randint(32, 256),\n",
    "                                                        \"conv_kernel\": randint(1, 8),\n",
    "                                                        \"fc_layers\": randint(1, 3),\n",
    "                                                        \"fc_dense\" : randint(4, 128)\n",
    "                                       },\n",
    "                                   n_iter=10,\n",
    "                                   n_jobs=1,\n",
    "                                   cv=skf,\n",
    "                                   scoring='accuracy',\n",
    "                                   iid=False, # just return the average score across folds\n",
    "                                   return_train_score=False,\n",
    "                                   verbose=2,\n",
    "                                   random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1244
    },
    "colab_type": "code",
    "id": "QJltym6XRvzC",
    "outputId": "ad2aca51-c1b8-4f49-8fcc-ccb83f4846d7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n",
      "[CV] batch_size=236, conv_filters=79, conv_kernel=6, dropout_lvl=0.42213287429050866, embedding_size=267, epochs=2, fc_dense=107, fc_layers=2, gru_units=243, spatial_dropout=0 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  batch_size=236, conv_filters=79, conv_kernel=6, dropout_lvl=0.42213287429050866, embedding_size=267, epochs=2, fc_dense=107, fc_layers=2, gru_units=243, spatial_dropout=0, total= 1.2min\n",
      "[CV] batch_size=236, conv_filters=79, conv_kernel=6, dropout_lvl=0.42213287429050866, embedding_size=267, epochs=2, fc_dense=107, fc_layers=2, gru_units=243, spatial_dropout=0 \n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:  1.2min remaining:    0.0s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[CV]  batch_size=236, conv_filters=79, conv_kernel=6, dropout_lvl=0.42213287429050866, embedding_size=267, epochs=2, fc_dense=107, fc_layers=2, gru_units=243, spatial_dropout=0, total= 1.2min\n",
      "[CV] batch_size=236, conv_filters=79, conv_kernel=6, dropout_lvl=0.42213287429050866, embedding_size=267, epochs=2, fc_dense=107, fc_layers=2, gru_units=243, spatial_dropout=0 \n",
      "[CV]  batch_size=236, conv_filters=79, conv_kernel=6, dropout_lvl=0.42213287429050866, embedding_size=267, epochs=2, fc_dense=107, fc_layers=2, gru_units=243, spatial_dropout=0, total= 1.2min\n",
      "[CV] batch_size=341, conv_filters=68, conv_kernel=7, dropout_lvl=0.02835648865872159, embedding_size=412, epochs=1, fc_dense=69, fc_layers=1, gru_units=71, spatial_dropout=0 \n",
      "[CV]  batch_size=341, conv_filters=68, conv_kernel=7, dropout_lvl=0.02835648865872159, embedding_size=412, epochs=1, fc_dense=69, fc_layers=1, gru_units=71, spatial_dropout=0, total=  16.7s\n",
      "[CV] batch_size=341, conv_filters=68, conv_kernel=7, dropout_lvl=0.02835648865872159, embedding_size=412, epochs=1, fc_dense=69, fc_layers=1, gru_units=71, spatial_dropout=0 \n",
      "[CV]  batch_size=341, conv_filters=68, conv_kernel=7, dropout_lvl=0.02835648865872159, embedding_size=412, epochs=1, fc_dense=69, fc_layers=1, gru_units=71, spatial_dropout=0, total=  17.3s\n",
      "[CV] batch_size=341, conv_filters=68, conv_kernel=7, dropout_lvl=0.02835648865872159, embedding_size=412, epochs=1, fc_dense=69, fc_layers=1, gru_units=71, spatial_dropout=0 \n",
      "[CV]  batch_size=341, conv_filters=68, conv_kernel=7, dropout_lvl=0.02835648865872159, embedding_size=412, epochs=1, fc_dense=69, fc_layers=1, gru_units=71, spatial_dropout=0, total=  17.7s\n",
      "[CV] batch_size=151, conv_filters=206, conv_kernel=1, dropout_lvl=0.19639239805041486, embedding_size=41, epochs=2, fc_dense=76, fc_layers=2, gru_units=180, spatial_dropout=0 \n",
      "[CV]  batch_size=151, conv_filters=206, conv_kernel=1, dropout_lvl=0.19639239805041486, embedding_size=41, epochs=2, fc_dense=76, fc_layers=2, gru_units=180, spatial_dropout=0, total=  48.8s\n",
      "[CV] batch_size=151, conv_filters=206, conv_kernel=1, dropout_lvl=0.19639239805041486, embedding_size=41, epochs=2, fc_dense=76, fc_layers=2, gru_units=180, spatial_dropout=0 \n",
      "[CV]  batch_size=151, conv_filters=206, conv_kernel=1, dropout_lvl=0.19639239805041486, embedding_size=41, epochs=2, fc_dense=76, fc_layers=2, gru_units=180, spatial_dropout=0, total=  48.7s\n",
      "[CV] batch_size=151, conv_filters=206, conv_kernel=1, dropout_lvl=0.19639239805041486, embedding_size=41, epochs=2, fc_dense=76, fc_layers=2, gru_units=180, spatial_dropout=0 \n",
      "[CV]  batch_size=151, conv_filters=206, conv_kernel=1, dropout_lvl=0.19639239805041486, embedding_size=41, epochs=2, fc_dense=76, fc_layers=2, gru_units=180, spatial_dropout=0, total=  49.7s\n",
      "[CV] batch_size=179, conv_filters=240, conv_kernel=4, dropout_lvl=0.4785775794765232, embedding_size=351, epochs=2, fc_dense=68, fc_layers=1, gru_units=131, spatial_dropout=0 \n",
      "[CV]  batch_size=179, conv_filters=240, conv_kernel=4, dropout_lvl=0.4785775794765232, embedding_size=351, epochs=2, fc_dense=68, fc_layers=1, gru_units=131, spatial_dropout=0, total=  47.9s\n",
      "[CV] batch_size=179, conv_filters=240, conv_kernel=4, dropout_lvl=0.4785775794765232, embedding_size=351, epochs=2, fc_dense=68, fc_layers=1, gru_units=131, spatial_dropout=0 \n",
      "[CV]  batch_size=179, conv_filters=240, conv_kernel=4, dropout_lvl=0.4785775794765232, embedding_size=351, epochs=2, fc_dense=68, fc_layers=1, gru_units=131, spatial_dropout=0, total=  48.5s\n",
      "[CV] batch_size=179, conv_filters=240, conv_kernel=4, dropout_lvl=0.4785775794765232, embedding_size=351, epochs=2, fc_dense=68, fc_layers=1, gru_units=131, spatial_dropout=0 \n",
      "[CV]  batch_size=179, conv_filters=240, conv_kernel=4, dropout_lvl=0.4785775794765232, embedding_size=351, epochs=2, fc_dense=68, fc_layers=1, gru_units=131, spatial_dropout=0, total=  49.9s\n",
      "[CV] batch_size=241, conv_filters=61, conv_kernel=4, dropout_lvl=0.33943976505948015, embedding_size=439, epochs=1, fc_dense=69, fc_layers=2, gru_units=217, spatial_dropout=0 \n",
      "[CV]  batch_size=241, conv_filters=61, conv_kernel=4, dropout_lvl=0.33943976505948015, embedding_size=439, epochs=1, fc_dense=69, fc_layers=2, gru_units=217, spatial_dropout=0, total=  44.1s\n",
      "[CV] batch_size=241, conv_filters=61, conv_kernel=4, dropout_lvl=0.33943976505948015, embedding_size=439, epochs=1, fc_dense=69, fc_layers=2, gru_units=217, spatial_dropout=0 \n",
      "[CV]  batch_size=241, conv_filters=61, conv_kernel=4, dropout_lvl=0.33943976505948015, embedding_size=439, epochs=1, fc_dense=69, fc_layers=2, gru_units=217, spatial_dropout=0, total=  44.6s\n",
      "[CV] batch_size=241, conv_filters=61, conv_kernel=4, dropout_lvl=0.33943976505948015, embedding_size=439, epochs=1, fc_dense=69, fc_layers=2, gru_units=217, spatial_dropout=0 \n",
      "[CV]  batch_size=241, conv_filters=61, conv_kernel=4, dropout_lvl=0.33943976505948015, embedding_size=439, epochs=1, fc_dense=69, fc_layers=2, gru_units=217, spatial_dropout=0, total=  46.4s\n",
      "[CV] batch_size=191, conv_filters=64, conv_kernel=3, dropout_lvl=0.2073309699952618, embedding_size=179, epochs=1, fc_dense=79, fc_layers=1, gru_units=215, spatial_dropout=0 \n",
      "[CV]  batch_size=191, conv_filters=64, conv_kernel=3, dropout_lvl=0.2073309699952618, embedding_size=179, epochs=1, fc_dense=79, fc_layers=1, gru_units=215, spatial_dropout=0, total=  40.3s\n",
      "[CV] batch_size=191, conv_filters=64, conv_kernel=3, dropout_lvl=0.2073309699952618, embedding_size=179, epochs=1, fc_dense=79, fc_layers=1, gru_units=215, spatial_dropout=0 \n",
      "[CV]  batch_size=191, conv_filters=64, conv_kernel=3, dropout_lvl=0.2073309699952618, embedding_size=179, epochs=1, fc_dense=79, fc_layers=1, gru_units=215, spatial_dropout=0, total=  41.6s\n",
      "[CV] batch_size=191, conv_filters=64, conv_kernel=3, dropout_lvl=0.2073309699952618, embedding_size=179, epochs=1, fc_dense=79, fc_layers=1, gru_units=215, spatial_dropout=0 \n",
      "[CV]  batch_size=191, conv_filters=64, conv_kernel=3, dropout_lvl=0.2073309699952618, embedding_size=179, epochs=1, fc_dense=79, fc_layers=1, gru_units=215, spatial_dropout=0, total=  41.7s\n",
      "[CV] batch_size=92, conv_filters=66, conv_kernel=1, dropout_lvl=0.009394900218177571, embedding_size=69, epochs=2, fc_dense=42, fc_layers=1, gru_units=49, spatial_dropout=0 \n",
      "[CV]  batch_size=92, conv_filters=66, conv_kernel=1, dropout_lvl=0.009394900218177571, embedding_size=69, epochs=2, fc_dense=42, fc_layers=1, gru_units=49, spatial_dropout=0, total=  39.0s\n",
      "[CV] batch_size=92, conv_filters=66, conv_kernel=1, dropout_lvl=0.009394900218177571, embedding_size=69, epochs=2, fc_dense=42, fc_layers=1, gru_units=49, spatial_dropout=0 \n",
      "[CV]  batch_size=92, conv_filters=66, conv_kernel=1, dropout_lvl=0.009394900218177571, embedding_size=69, epochs=2, fc_dense=42, fc_layers=1, gru_units=49, spatial_dropout=0, total=  39.6s\n",
      "[CV] batch_size=92, conv_filters=66, conv_kernel=1, dropout_lvl=0.009394900218177571, embedding_size=69, epochs=2, fc_dense=42, fc_layers=1, gru_units=49, spatial_dropout=0 \n",
      "[CV]  batch_size=92, conv_filters=66, conv_kernel=1, dropout_lvl=0.009394900218177571, embedding_size=69, epochs=2, fc_dense=42, fc_layers=1, gru_units=49, spatial_dropout=0, total=  40.4s\n",
      "[CV] batch_size=399, conv_filters=164, conv_kernel=2, dropout_lvl=0.2249749949556138, embedding_size=47, epochs=1, fc_dense=5, fc_layers=2, gru_units=201, spatial_dropout=0 \n",
      "[CV]  batch_size=399, conv_filters=164, conv_kernel=2, dropout_lvl=0.2249749949556138, embedding_size=47, epochs=1, fc_dense=5, fc_layers=2, gru_units=201, spatial_dropout=0, total=  41.0s\n",
      "[CV] batch_size=399, conv_filters=164, conv_kernel=2, dropout_lvl=0.2249749949556138, embedding_size=47, epochs=1, fc_dense=5, fc_layers=2, gru_units=201, spatial_dropout=0 \n",
      "[CV]  batch_size=399, conv_filters=164, conv_kernel=2, dropout_lvl=0.2249749949556138, embedding_size=47, epochs=1, fc_dense=5, fc_layers=2, gru_units=201, spatial_dropout=0, total=  41.2s\n",
      "[CV] batch_size=399, conv_filters=164, conv_kernel=2, dropout_lvl=0.2249749949556138, embedding_size=47, epochs=1, fc_dense=5, fc_layers=2, gru_units=201, spatial_dropout=0 \n",
      "[CV]  batch_size=399, conv_filters=164, conv_kernel=2, dropout_lvl=0.2249749949556138, embedding_size=47, epochs=1, fc_dense=5, fc_layers=2, gru_units=201, spatial_dropout=0, total=  42.0s\n",
      "[CV] batch_size=121, conv_filters=67, conv_kernel=7, dropout_lvl=0.3353189348090797, embedding_size=446, epochs=1, fc_dense=95, fc_layers=1, gru_units=174, spatial_dropout=0 \n",
      "[CV]  batch_size=121, conv_filters=67, conv_kernel=7, dropout_lvl=0.3353189348090797, embedding_size=446, epochs=1, fc_dense=95, fc_layers=1, gru_units=174, spatial_dropout=0, total=  49.4s\n",
      "[CV] batch_size=121, conv_filters=67, conv_kernel=7, dropout_lvl=0.3353189348090797, embedding_size=446, epochs=1, fc_dense=95, fc_layers=1, gru_units=174, spatial_dropout=0 \n",
      "[CV]  batch_size=121, conv_filters=67, conv_kernel=7, dropout_lvl=0.3353189348090797, embedding_size=446, epochs=1, fc_dense=95, fc_layers=1, gru_units=174, spatial_dropout=0, total=  48.0s\n",
      "[CV] batch_size=121, conv_filters=67, conv_kernel=7, dropout_lvl=0.3353189348090797, embedding_size=446, epochs=1, fc_dense=95, fc_layers=1, gru_units=174, spatial_dropout=0 \n",
      "[CV]  batch_size=121, conv_filters=67, conv_kernel=7, dropout_lvl=0.3353189348090797, embedding_size=446, epochs=1, fc_dense=95, fc_layers=1, gru_units=174, spatial_dropout=0, total=  48.9s\n",
      "[CV] batch_size=163, conv_filters=85, conv_kernel=5, dropout_lvl=0.2850983852089398, embedding_size=100, epochs=2, fc_dense=72, fc_layers=1, gru_units=228, spatial_dropout=0 \n",
      "[CV]  batch_size=163, conv_filters=85, conv_kernel=5, dropout_lvl=0.2850983852089398, embedding_size=100, epochs=2, fc_dense=72, fc_layers=1, gru_units=228, spatial_dropout=0, total= 1.3min\n",
      "[CV] batch_size=163, conv_filters=85, conv_kernel=5, dropout_lvl=0.2850983852089398, embedding_size=100, epochs=2, fc_dense=72, fc_layers=1, gru_units=228, spatial_dropout=0 \n",
      "[CV]  batch_size=163, conv_filters=85, conv_kernel=5, dropout_lvl=0.2850983852089398, embedding_size=100, epochs=2, fc_dense=72, fc_layers=1, gru_units=228, spatial_dropout=0, total= 1.3min\n",
      "[CV] batch_size=163, conv_filters=85, conv_kernel=5, dropout_lvl=0.2850983852089398, embedding_size=100, epochs=2, fc_dense=72, fc_layers=1, gru_units=228, spatial_dropout=0 \n",
      "[CV]  batch_size=163, conv_filters=85, conv_kernel=5, dropout_lvl=0.2850983852089398, embedding_size=100, epochs=2, fc_dense=72, fc_layers=1, gru_units=228, spatial_dropout=0, total= 1.3min\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Done  30 out of  30 | elapsed: 24.1min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=StratifiedKFold(n_splits=3, random_state=0, shuffle=True),\n",
       "          error_score='raise-deprecating',\n",
       "          estimator=<keras.wrappers.scikit_learn.KerasClassifier object at 0x7f1c90ea14e0>,\n",
       "          fit_params=None, iid=False, n_iter=10, n_jobs=1,\n",
       "          param_distributions={'epochs': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f1c93181668>, 'batch_size': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f1c93e83208>, 'embedding_size': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f1c93e83390>, 'dropout_lvl': <scipy...0x7f1c93e83d68>, 'fc_dense': <scipy.stats._distn_infrastructure.rv_frozen object at 0x7f1c93e83f60>},\n",
       "          pre_dispatch='2*n_jobs', random_state=0, refit=True,\n",
       "          return_train_score=False, scoring='accuracy', verbose=2)"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.fit(train_data, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "id": "yJ2bKM7RRvzH",
    "outputId": "2e4a0903-2521-4828-ebe6-ca5d6865aff3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'batch_size': 121,\n",
       " 'conv_filters': 67,\n",
       " 'conv_kernel': 7,\n",
       " 'dropout_lvl': 0.3353189348090797,\n",
       " 'embedding_size': 446,\n",
       " 'epochs': 1,\n",
       " 'fc_dense': 95,\n",
       " 'fc_layers': 1,\n",
       " 'gru_units': 174,\n",
       " 'spatial_dropout': 0}"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UQXcwQkZRvzJ"
   },
   "outputs": [],
   "source": [
    "# Skopt functions\n",
    "from skopt import BayesSearchCV\n",
    "from skopt import gp_minimize # Bayesian optimization using Gaussian Processes\n",
    "from skopt.space import Real, Categorical, Integer\n",
    "from skopt.utils import use_named_args # decorator to convert a list of parameters to named arguments\n",
    "from skopt.callbacks import DeadlineStopper # Stop the optimization before running out of a fixed budget of time.\n",
    "from skopt.callbacks import VerboseCallback # Callback to control the verbosity\n",
    "from skopt.callbacks import DeltaXStopper # Stop the optimization If the last two positions at which the objective has been evaluated are less than delta\n",
    "\n",
    "# Metrics\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "# Parallelism\n",
    "import joblib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "LTE-Ty7FRvzM"
   },
   "outputs": [],
   "source": [
    "dimensions = [Integer(1, 3, name=\"epochs\"),\n",
    "              Integer(64, 512, name=\"batch_size\"),\n",
    "              Integer(16, 512, name=\"embedding_size\"),\n",
    "              Real(0, 0.5, name=\"dropout_lvl\"),\n",
    "              Integer(0, 1, name=\"spatial_dropout\"),\n",
    "              Integer(32, 256, name=\"gru_units\"),\n",
    "              Integer(32, 256, name=\"conv_filters\"),\n",
    "              Integer(1, 8, name=\"conv_kernel\"),\n",
    "              Integer(1, 3, name=\"fc_layers\"),\n",
    "              Integer(4, 128, name=\"fc_dense\")\n",
    "             ]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "0HAt1s83RvzO"
   },
   "outputs": [],
   "source": [
    "params = [\"epochs\", \"batch_size\", \"embedding_size\", \"dropout_lvl\", \"spatial_dropout\", \n",
    "          \"gru_units\", \"conv_filters\", \"conv_kernel\", \"fc_layers\", \"fc_dense\"]\n",
    "x0 = [[ex[p] for p in params] for ex in random_search.cv_results_['params']]\n",
    "y0 = random_search.cv_results_['mean_test_score'] * -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 921
    },
    "colab_type": "code",
    "id": "9CQrL6CqRvzR",
    "outputId": "d1ff7aaa-746b-42ed-b80c-6549b720190f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last eval:  [2, 163, 100, 0.2850983852089398, 0, 228, 85, 5, 1, 72]  - Score  -0.8712778349791547\n",
      "Current iter:  0  - Score  -0.8865601841627034  - Args:  [1, 121, 446, 0.3353189348090797, 0, 174, 67, 7, 1, 95]\n",
      "Testing parameters: {'epochs': 3, 'batch_size': 482, 'embedding_size': 80, 'dropout_lvl': 0.4995202576620725, 'spatial_dropout': 0, 'gru_units': 121, 'conv_filters': 119, 'conv_kernel': 6, 'fc_layers': 3, 'fc_dense': 109}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 80, 'dropout_lvl': 0.4995202576620725, 'spatial_dropout': 0, 'gru_units': 121, 'conv_filters': 119, 'conv_kernel': 6, 'fc_layers': 3, 'fc_dense': 109}\n",
      "CV scores: [0.8822894168466523, 0.8785697144228461, 0.8869419107057129]\n",
      "Last eval:  [3, 482, 80, 0.4995202576620725, 0, 121, 119, 6, 3, 109]  - Score  -0.8826003473250704\n",
      "Current iter:  1  - Score  -0.8865601841627034  - Args:  [1, 121, 446, 0.3353189348090797, 0, 174, 67, 7, 1, 95]\n",
      "Testing parameters: {'epochs': 2, 'batch_size': 299, 'embedding_size': 236, 'dropout_lvl': 0.1147886068649128, 'spatial_dropout': 1, 'gru_units': 237, 'conv_filters': 134, 'conv_kernel': 4, 'fc_layers': 3, 'fc_dense': 101}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 236, 'dropout_lvl': 0.1147886068649128, 'spatial_dropout': 1, 'gru_units': 237, 'conv_filters': 134, 'conv_kernel': 4, 'fc_layers': 3, 'fc_dense': 101}\n",
      "CV scores: [0.8878089752819774, 0.8879289656827454, 0.8931829092654825]\n",
      "Last eval:  [2, 299, 236, 0.1147886068649128, 1, 237, 134, 4, 3, 101]  - Score  -0.8896402834100684\n",
      "Current iter:  2  - Score  -0.8896402834100684  - Args:  [2, 299, 236, 0.1147886068649128, 1, 237, 134, 4, 3, 101]\n",
      "Testing parameters: {'epochs': 2, 'batch_size': 424, 'embedding_size': 62, 'dropout_lvl': 0.25907627447094456, 'spatial_dropout': 1, 'gru_units': 218, 'conv_filters': 218, 'conv_kernel': 3, 'fc_layers': 1, 'fc_dense': 87}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 62, 'dropout_lvl': 0.25907627447094456, 'spatial_dropout': 1, 'gru_units': 218, 'conv_filters': 218, 'conv_kernel': 3, 'fc_layers': 1, 'fc_dense': 87}\n",
      "CV scores: [0.8666906647468202, 0.8720902327813775, 0.8608977436389822]\n",
      "Last eval:  [2, 424, 62, 0.25907627447094456, 1, 218, 218, 3, 1, 87]  - Score  -0.8665595470557266\n",
      "Current iter:  3  - Score  -0.8896402834100684  - Args:  [2, 299, 236, 0.1147886068649128, 1, 237, 134, 4, 3, 101]\n",
      "Testing parameters: {'epochs': 2, 'batch_size': 365, 'embedding_size': 220, 'dropout_lvl': 0.09877544899239063, 'spatial_dropout': 0, 'gru_units': 64, 'conv_filters': 207, 'conv_kernel': 4, 'fc_layers': 1, 'fc_dense': 81}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 220, 'dropout_lvl': 0.09877544899239063, 'spatial_dropout': 0, 'gru_units': 64, 'conv_filters': 207, 'conv_kernel': 4, 'fc_layers': 1, 'fc_dense': 81}\n",
      "CV scores: [0.880729541636669, 0.880729541636669, 0.8853816610657705]\n",
      "Last eval:  [2, 365, 220, 0.09877544899239063, 0, 64, 207, 4, 1, 81]  - Score  -0.8822802481130362\n",
      "Current iter:  4  - Score  -0.8896402834100684  - Args:  [2, 299, 236, 0.1147886068649128, 1, 237, 134, 4, 3, 101]\n",
      "Testing parameters: {'epochs': 2, 'batch_size': 198, 'embedding_size': 237, 'dropout_lvl': 0.11106227376768743, 'spatial_dropout': 0, 'gru_units': 137, 'conv_filters': 54, 'conv_kernel': 7, 'fc_layers': 1, 'fc_dense': 69}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 237, 'dropout_lvl': 0.11106227376768743, 'spatial_dropout': 0, 'gru_units': 137, 'conv_filters': 54, 'conv_kernel': 7, 'fc_layers': 1, 'fc_dense': 69}\n",
      "CV scores: [0.8782097432205423, 0.8794096472282218, 0.8840614498319731]\n",
      "Last eval:  [2, 198, 237, 0.11106227376768743, 0, 137, 54, 7, 1, 69]  - Score  -0.880560280093579\n",
      "Current iter:  5  - Score  -0.8896402834100684  - Args:  [2, 299, 236, 0.1147886068649128, 1, 237, 134, 4, 3, 101]\n",
      "Testing parameters: {'epochs': 1, 'batch_size': 475, 'embedding_size': 468, 'dropout_lvl': 0.14946505610485808, 'spatial_dropout': 1, 'gru_units': 159, 'conv_filters': 170, 'conv_kernel': 8, 'fc_layers': 2, 'fc_dense': 33}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 468, 'dropout_lvl': 0.14946505610485808, 'spatial_dropout': 1, 'gru_units': 159, 'conv_filters': 170, 'conv_kernel': 8, 'fc_layers': 2, 'fc_dense': 33}\n",
      "CV scores: [0.837892968562515, 0.8404127669786418, 0.8256120979356697]\n",
      "Last eval:  [1, 475, 468, 0.14946505610485808, 1, 159, 170, 8, 2, 33]  - Score  -0.8346392778256089\n",
      "Current iter:  6  - Score  -0.8896402834100684  - Args:  [2, 299, 236, 0.1147886068649128, 1, 237, 134, 4, 3, 101]\n",
      "Testing parameters: {'epochs': 2, 'batch_size': 490, 'embedding_size': 261, 'dropout_lvl': 0.27030025441182204, 'spatial_dropout': 1, 'gru_units': 42, 'conv_filters': 63, 'conv_kernel': 7, 'fc_layers': 1, 'fc_dense': 114}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 261, 'dropout_lvl': 0.27030025441182204, 'spatial_dropout': 1, 'gru_units': 42, 'conv_filters': 63, 'conv_kernel': 7, 'fc_layers': 1, 'fc_dense': 114}\n",
      "CV scores: [0.8729301655867531, 0.841732661387089, 0.8444551128180509]\n",
      "Last eval:  [2, 490, 261, 0.27030025441182204, 1, 42, 63, 7, 1, 114]  - Score  -0.8530393132639643\n",
      "Current iter:  7  - Score  -0.8896402834100684  - Args:  [2, 299, 236, 0.1147886068649128, 1, 237, 134, 4, 3, 101]\n",
      "Testing parameters: {'epochs': 2, 'batch_size': 265, 'embedding_size': 458, 'dropout_lvl': 0.18879217057554737, 'spatial_dropout': 1, 'gru_units': 178, 'conv_filters': 113, 'conv_kernel': 5, 'fc_layers': 2, 'fc_dense': 20}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 458, 'dropout_lvl': 0.18879217057554737, 'spatial_dropout': 1, 'gru_units': 178, 'conv_filters': 113, 'conv_kernel': 5, 'fc_layers': 2, 'fc_dense': 20}\n",
      "CV scores: [0.89056875449964, 0.8874490040796736, 0.8952232357177148]\n",
      "Last eval:  [2, 265, 458, 0.18879217057554737, 1, 178, 113, 5, 2, 20]  - Score  -0.8910803314323429\n",
      "Current iter:  8  - Score  -0.8910803314323429  - Args:  [2, 265, 458, 0.18879217057554737, 1, 178, 113, 5, 2, 20]\n",
      "Testing parameters: {'epochs': 2, 'batch_size': 354, 'embedding_size': 192, 'dropout_lvl': 0.38161652635756105, 'spatial_dropout': 0, 'gru_units': 201, 'conv_filters': 229, 'conv_kernel': 1, 'fc_layers': 2, 'fc_dense': 13}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 192, 'dropout_lvl': 0.38161652635756105, 'spatial_dropout': 0, 'gru_units': 201, 'conv_filters': 229, 'conv_kernel': 1, 'fc_layers': 2, 'fc_dense': 13}\n",
      "CV scores: [0.8264938804895609, 0.876769858411327, 0.857177148343735]\n",
      "Last eval:  [2, 354, 192, 0.38161652635756105, 0, 201, 229, 1, 2, 13]  - Score  -0.8534802957482076\n",
      "Current iter:  9  - Score  -0.8910803314323429  - Args:  [2, 265, 458, 0.18879217057554737, 1, 178, 113, 5, 2, 20]\n",
      "Testing parameters: {'epochs': 3, 'batch_size': 93, 'embedding_size': 192, 'dropout_lvl': 0.47091847437671136, 'spatial_dropout': 0, 'gru_units': 203, 'conv_filters': 205, 'conv_kernel': 3, 'fc_layers': 3, 'fc_dense': 23}\n",
      "{'multigpu': False, 'vocab_size': 9999, 'verbose': 0, 'embedding_size': 192, 'dropout_lvl': 0.47091847437671136, 'spatial_dropout': 0, 'gru_units': 203, 'conv_filters': 205, 'conv_kernel': 3, 'fc_layers': 3, 'fc_dense': 23}\n",
      "CV scores: [0.8884089272858171, 0.8599712023038157, 0.8888622179548727]\n",
      "Last eval:  [3, 93, 192, 0.47091847437671136, 0, 203, 205, 3, 3, 23]  - Score  -0.8790807825148352\n",
      "Current iter:  10  - Score  -0.8910803314323429  - Args:  [2, 265, 458, 0.18879217057554737, 1, 178, 113, 5, 2, 20]\n"
     ]
    }
   ],
   "source": [
    "counter = 0\n",
    "def onstep(res):\n",
    "    global counter\n",
    "    x0 = res.x_iters   # List of input points\n",
    "    y0 = res.func_vals # Evaluation of input points\n",
    "    print('Last eval: ', x0[-1], \n",
    "          ' - Score ', y0[-1])\n",
    "    print('Current iter: ', counter, \n",
    "          ' - Score ', res.fun, \n",
    "          ' - Args: ', res.x)\n",
    "    joblib.dump((x0, y0), 'checkpoint.pkl') # Saving a checkpoint to disk\n",
    "    counter += 1\n",
    "\n",
    "def make_objective(create_model, X, y, space, cv, scoring,\n",
    "                   multigpu, vocab_size, verbose=0):\n",
    "    # This decorator converts your objective function with named arguments into one that\n",
    "    # accepts a list as argument, while doing the conversion automatically.\n",
    "    @use_named_args(space) \n",
    "    def objective(**params):\n",
    "        \n",
    "        arch_params = create_model.__code__.co_varnames\n",
    "        \n",
    "        model_params = {p:params[p] for p in params if p in arch_params}\n",
    "        fit_params = {p:params[p]for p in params if p not in arch_params}\n",
    "        \n",
    "        model = KerasClassifier(build_fn=create_model, \n",
    "                                multigpu=multigpu,\n",
    "                                vocab_size=vocab_size,\n",
    "                                verbose=verbose,\n",
    "                                **model_params)\n",
    "        \n",
    "        score = list()\n",
    "        print(\"Testing parameters:\", params)\n",
    "        print(model.sk_params)\n",
    "        for j, (train_index, test_index) in enumerate(skf.split(X, y)):      \n",
    "            model.fit(X[train_index,:], y[train_index], **fit_params)\n",
    "            val_preds = model.predict(X[test_index,:])\n",
    "            score.append(scoring(y_true=y[test_index], y_pred=val_preds))\n",
    "\n",
    "        \n",
    "        print(\"CV scores:\", score)\n",
    "        return np.mean(score) * -1\n",
    "\n",
    "    return objective\n",
    "\n",
    "objective = make_objective(create_model,\n",
    "                           train_data, train_labels,\n",
    "                           space=dimensions,\n",
    "                           cv=skf,\n",
    "                           scoring=accuracy_score,\n",
    "                           multigpu=False,\n",
    "                           vocab_size = len(np.unique(train_data)),\n",
    "                           verbose=0)\n",
    "\n",
    "\n",
    "gp_round = gp_minimize(func=objective,\n",
    "                       x0=x0,              # already examined values for x\n",
    "                       y0=y0,              # observed values for x0\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge', # Expected Improvement.\n",
    "                       n_calls=10,\n",
    "                       callback=[onstep],\n",
    "                       random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "id": "kUXB8zCtRvzU",
    "outputId": "d6698b45-8cb4-44da-9e24-7f8220bea1a8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2, 265, 458, 0.18879217057554737, 1, 178, 113, 5, 2, 20] -0.8910803314323429\n"
     ]
    }
   ],
   "source": [
    "best_parameters = gp_round.x\n",
    "best_result = gp_round.fun\n",
    "print(best_parameters, best_result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "coSzJ32OSOrR"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "hUvgEZ4ORvwF",
    "15H1-CenRvwM",
    "v6ET90r_Rvwk"
   ],
   "name": "Copy of skopt_workshop.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}