skopt_workshop_part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Gc9IfCbORvux"
   },
   "source": [
    "# Bayesian Optimization Workshop: <BR>from hyperparameters optimization to Neural architecture search (NAS) \n",
    "\n",
    "[![Open in Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/lmassaron/kaggledays-2019-gbdt/blob/master/skopt_workshop_part1.ipynb)\n",
    "\n",
    "\n",
    "## Instructors\n",
    "* Luca Massaron [@lmassaron](https://www.linkedin.com/in/lmassaron/) - Data Scientist / Author / Google Developer Expert in Machine Learning\n",
    "\n",
    "## About the workshop\n",
    "In this workshop we demonstrate how to use different optimization approaches based on [Scikit-Optimize](https://github.com/scikit-optimize/scikit-optimize), a library built on top of NumPy, SciPy and Scikit-Learn, and we present an easy and fast approach to set them ready and usable.\n",
    "\n",
    "In the first part, we start from trying to optimize Gradient Boosting Decision Trees (GBDT), which presently represent the state of the art for building predictors for flat table data. However, GBDT seldom perform the best out-of-the-box (using default values) because of the many hyper-parameters to tune. Especially in the most recent GBDT implementations, such as LightGBM, the over-sophistication of hyper-parameters renders finding the optimal settings by hand or simple grid search difficult because of high combinatorial complexity and long running times for experiments.\n",
    "\n",
    "In the second part, we try to leverage what we have learned so far and challenge the Neural Architecture Search (NAS) problem.\n",
    "\n",
    "## Prerequisites\n",
    "You should be aware of the role and importance of hyper-parameter optimization in machine learning.  \n",
    "\n",
    "\n",
    "## Dealing with the Tutorial Material\n",
    "In order to make the workshop easily accessible, we are prepared cloud access:\n",
    "* Using [Google Colab](https://colab.research.google.com/github/lmassaron/kaggledays-2019-gbdt/blob/master/skopt_workshop.ipynb)\n",
    "\n",
    "## References\n",
    "* [Random Optimization](https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf) (BERGSTRA, James; BENGIO, Yoshua. Random search for hyper-parameter optimization. Journal of Machine Learning Research, 2012, 13.Feb: 281-305.) \n",
    "* [Bayesian Optimization](https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf) (SNOEK, Jasper; LAROCHELLE, Hugo; ADAMS, Ryan P. Practical bayesian optimization of machine learning algorithms. In: Advances in neural information processing systems. 2012. p. 2951-2959)\n",
    "\n",
    "For a brief introduction about the key models we will be using during this works we suggest consulting: BOSCHETTI, Alberto; MASSARON, Luca. Python data science essentials. Packt Publishing Ltd, 3rd ed., 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qDmJ6YOLRvu0"
   },
   "source": [
    "# PART I : Optimizing hyper-parameters\n",
    "\n",
    "The first topic of this workshop aims to illustrate how to best optimize the hyperparameters of a gradient boosting model (lightGBM before all, but also XGBoost and CatBoost) in a performing and efficient way. We will also compare the strong and weak points of different tuning approaches, such grid-search, random search and bayesian optimization by Scikit-optimize.\n",
    "\n",
    "Leaving apart grid-search (feasible only when the space of experiments is limited), the usual choice for the practitioner is to apply random search optimization or try some [Bayesian Optimization](https://papers.nips.cc/paper/4522-practical-bayesian-optimization-of-machine-learning-algorithms.pdf) (BO) technique, which require a more complex setup. \n",
    "\n",
    "As for as BO, there are quite a few choices (for instance Hyperopt) but we decided for Scikit-Optimize, or skopt, because it is a simple and efficient library to minimize (very) expensive and noisy black-box functions and it works with an API similar to Scikit-learn. It can be found at https://github.com/scikit-optimize/scikit-optimize/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ngnS0MSaRvu2",
    "outputId": "ea767f9f-e59a-4a09-d268-b29597bf44d8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting git+https://github.com/scikit-optimize/scikit-optimize.git\n",
      "  Cloning https://github.com/scikit-optimize/scikit-optimize.git to c:\\users\\luca\\appdata\\local\\temp\\pip-req-build-3q7uxgvk\n",
      "Requirement already satisfied (use --upgrade to upgrade): scikit-optimize==0+unknown from git+https://github.com/scikit-optimize/scikit-optimize.git in c:\\users\\luca\\anaconda3\\lib\\site-packages\n",
      "Requirement already satisfied: pyaml in c:\\users\\luca\\anaconda3\\lib\\site-packages (from scikit-optimize==0+unknown) (18.11.0)\n",
      "Requirement already satisfied: numpy in c:\\users\\luca\\anaconda3\\lib\\site-packages (from scikit-optimize==0+unknown) (1.15.4)\n",
      "Requirement already satisfied: scipy>=0.14.0 in c:\\users\\luca\\anaconda3\\lib\\site-packages (from scikit-optimize==0+unknown) (1.1.0)\n",
      "Requirement already satisfied: scikit-learn>=0.19.1 in c:\\users\\luca\\anaconda3\\lib\\site-packages (from scikit-optimize==0+unknown) (0.20.2)\n",
      "Requirement already satisfied: PyYAML in c:\\users\\luca\\anaconda3\\lib\\site-packages (from pyaml->scikit-optimize==0+unknown) (3.13)\n",
      "Building wheels for collected packages: scikit-optimize\n",
      "  Running setup.py bdist_wheel for scikit-optimize: started\n",
      "  Running setup.py bdist_wheel for scikit-optimize: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\Luca\\AppData\\Local\\Temp\\pip-ephem-wheel-cache-verq8ue2\\wheels\\11\\6f\\86\\2b772172db85ad0b4487d67e325e535ee8e7782b2a1dfcadf5\n",
      "Successfully built scikit-optimize\n"
     ]
    }
   ],
   "source": [
    "# Installing the most recent version of skopt directly from Github\n",
    "!pip install git+https://github.com/scikit-optimize/scikit-optimize.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "S9zjvLh5RvvA",
    "outputId": "f88684a2-3e36-44d4-e309-746c259ee4cb"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already up-to-date: catboost in c:\\users\\luca\\anaconda3\\lib\\site-packages (0.13.1)\n",
      "Requirement already satisfied, skipping upgrade: numpy>=1.11.1 in c:\\users\\luca\\anaconda3\\lib\\site-packages (from catboost) (1.15.4)\n",
      "Requirement already satisfied, skipping upgrade: pandas>=0.19.1 in c:\\users\\luca\\anaconda3\\lib\\site-packages (from catboost) (0.23.4)\n",
      "Requirement already satisfied, skipping upgrade: six in c:\\users\\luca\\anaconda3\\lib\\site-packages (from catboost) (1.12.0)\n",
      "Requirement already satisfied, skipping upgrade: enum34 in c:\\users\\luca\\anaconda3\\lib\\site-packages (from catboost) (1.1.6)\n",
      "Requirement already satisfied, skipping upgrade: python-dateutil>=2.5.0 in c:\\users\\luca\\anaconda3\\lib\\site-packages (from pandas>=0.19.1->catboost) (2.7.5)\n",
      "Requirement already satisfied, skipping upgrade: pytz>=2011k in c:\\users\\luca\\anaconda3\\lib\\site-packages (from pandas>=0.19.1->catboost) (2018.9)\n"
     ]
    }
   ],
   "source": [
    "# Assuring you have the most recent CatBoost release\n",
    "!pip install catboost -U"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "VZXk6GqKRvvF"
   },
   "outputs": [],
   "source": [
    "# Importing core libraries\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from time import time\n",
    "import pprint\n",
    "import joblib\n",
    "\n",
    "# Suppressing warnings because of skopt verbosity\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "# Our example dataset\n",
    "from sklearn.datasets import load_boston\n",
    "\n",
    "# Classifiers\n",
    "from sklearn.ensemble import GradientBoostingClassifier, GradientBoostingRegressor\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.pipeline import Pipeline\n",
    "import lightgbm as lgb\n",
    "import xgboost as xgb\n",
    "from catboost import CatBoostClassifier\n",
    "\n",
    "# Hyperparameters distributions\n",
    "from scipy.stats import randint\n",
    "from scipy.stats import uniform\n",
    "\n",
    "# Model selection\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "from sklearn.model_selection import GridSearchCV, RandomizedSearchCV\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Metrics\n",
    "from sklearn.metrics import average_precision_score\n",
    "from sklearn.metrics import make_scorer\n",
    "\n",
    "# Plotting\n",
    "from matplotlib import pyplot as plt\n",
    "import seaborn\n",
    "%matplotlib inline\n",
    "\n",
    "# Skopt functions\n",
    "from skopt import BayesSearchCV\n",
    "from skopt import gp_minimize # Bayesian optimization using Gaussian Processes\n",
    "from skopt.space import Real, Categorical, Integer\n",
    "from skopt.utils import use_named_args # decorator to convert a list of parameters to named arguments\n",
    "from skopt.callbacks import DeadlineStopper # Stop the optimization before running out of a fixed budget of time.\n",
    "from skopt.callbacks import VerboseCallback # Callback to control the verbosity\n",
    "from skopt.callbacks import DeltaXStopper # Stop the optimization If the last two positions at which the objective has been evaluated are less than delta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CIfUhtncRvvK"
   },
   "source": [
    "Optimizing hyper-parameters requires time and resources. In order to speed up the demonstration we will be using a toy dataset, the Boston Houseprice dataset for a classification task, to predicted the top 10% most expensive houses.\n",
    "\n",
    "The dataset presents information collected by the U.S Census Service concerning housing prices and conditions in the area of Boston Mass. Originally found in the [StatLib archive](http://lib.stat.cmu.edu/datasets/boston), the dataset has been used extensively throughout the literature to benchmark machine learning algorithms. The data was originally published by :\n",
    "> Harrison, D. and Rubinfeld, D.L. Hedonic prices and the demand for clean air', J. Environ. Economics & Management, vol.5, 81-102, 1978.\n",
    "\n",
    "The dataset contains 14 variabile relative to 506 house that were sold in the suburbs of Boston. Among the variables, the 14th, MEDV - Median value of owner-occupied homes in $1000's - is commonly used as a target for regression problems. In our example we will use it for classification, after binarizing it at the 90th percentile (also creating an unbalanced classification problem, since the positive cases are just 10 percent of the total). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "nEj3AeKCRvvL"
   },
   "outputs": [],
   "source": [
    "# Uploading the Boston dataset\n",
    "X, y = load_boston(return_X_y=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aSnCnm7ZRvvP"
   },
   "outputs": [],
   "source": [
    "# Transforming the problem into a classification (unbalanced)\n",
    "y_bin = (y > np.percentile(y, 90)).astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2b5NmP3eRvvW"
   },
   "outputs": [],
   "source": [
    "#CRIM - per capita crime rate by town\n",
    "#ZN - proportion of residential land zoned for lots over 25,000 sq.ft.\n",
    "#INDUS - proportion of non-retail business acres per town.\n",
    "#CHAS - Charles River dummy variable (1 if tract bounds river; 0 otherwise)\n",
    "#NOX - nitric oxides concentration (parts per 10 million)\n",
    "#RM - average number of rooms per dwelling\n",
    "#AGE - proportion of owner-occupied units built prior to 1940\n",
    "#DIS - weighted distances to five Boston employment centres\n",
    "#RAD - index of accessibility to radial highways\n",
    "#TAX - full-value property-tax rate per $10,000\n",
    "#PTRATIO - pupil-teacher ratio by town\n",
    "#B - 1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town\n",
    "#LSTAT - % lower status of the population\n",
    "#MEDV - Median value of owner-occupied homes in $1000's this is our target variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Y6ydkUziRvva",
    "outputId": "b5e81498-9704-4cd1-99ff-a6c8f1f5b0c2"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEICAYAAABPgw/pAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAFSpJREFUeJzt3X+Q3PV93/HnKyAiG3CExIkRHLaErTjgNkAiKC4pxWDHmBhBZuwW6tiiyKOZhri4jccBT5OWpsmYmYwdOrSZKuBGbh1+VAELaEOsUWGwOzYgDK4hIlXMD3NIQYeCAsQBjHj3j/3KHOeTdu9u9+70vedjZue731/7fe9nbl/7uc9+97upKiRJB7+fmO0CJEn9YaBLUksY6JLUEga6JLWEgS5JLWGgS1JLGOjqiySPJjl7tuuYTUl+OcnTSV5Kcups16P5x0BXV0meTPL+ccsuTfKNffNV9Z6quqfL4yxPUkkOHVCps+33gF+rqiOq6qHxK5vn/uzY55/k0CS7ktSYZfckebl5Y9h3u6NZd3aS18csH0lyS5LTxuz/WJLLJjj+FUm29v1Za84w0NUac+CN4h3Ao1222QN8aMz8+cDzE2y3741h3+2CMet2VNURwJHAGcBjwNeTnNus3wB8YoLH/HizTi1loKsvxvbik5yeZGuSF5oe6Reaze5tpnua3uV7k/xEkn+T5Kmmp/rlJD815nE/0azbneQ3xx3n3yXZmOS/J3kBuLQ59jeT7EmyM8l1SQ4b83iV5FeTbE/yYpLfTvLOZp8Xmt7uj7Yf9xwnrDXJTyZ5CTgE+E6S7x2gqf4bbw7bTwBfnmRzA1AdI1X1W8D1wDVjjvELSd4xpvYTgZ8FbpzKsXRwMNA1CNcC11bV24B3Arc0y89qpouaXuc3gUub2/uAE4AjgOsAkpwE/GfgY8Ay4KeA48Yd60JgI7AI+AqwF/hXwNHAe4FzgV8dt895wM/T6d1+FljfHON44O8Bl+zneU1Ya1W90vSYAU6uqnfuv2n4KnBWkkVJFgH/CNh0gO17dSvwc0kOr6oR4G46PfJ9PgH8r6p6rg/H0hxloKtXX216vXuS7KETtPvzQ+BdSY6uqpeq6lsH2PZjwBeq6vGqegm4Cri4GT75CHBHVX2jql4FfgsYf/Ghb1bVV6vq9ar6u6p6sKq+VVWvVdWTwH8B/vG4fa6pqheq6lHgEeBrzfH/BvhTYH8faB6o1l69DNwB/FPgYuD2Ztl4/3Fseyf57S6PuwMInTc26AytfBw6/1k0tTvc0nIGunp1UVUt2nfjx3u9Y60Ffhp4LMkDST58gG2PBZ4aM/8UcChwTLPu6X0rquoHwO5x+z89dibJTye5M8lfNcMwv0untz7Ws2Pu/90E80cwsQPVOhlfptNjPtBwy78c295V9ZtdHvM4Om92e5r5W4FlSc4AzgbeCvzPSdapg4yBrr6rqu1VdQmwlM647sYkh/PjvWvo9CzfMWb+7cBrdEJ2JzC8b0WStwBLxh9u3Pwf0PmQcGUz5PM5Oj3XfjhQrZPxdTpDSMcA3+iyba9+Gfh2Vf0t/OjNbyOdN42PAzc1/+WoxWb7rAC1UJJfAf6sqkab4RnojG2PAq/TGX/+f83yG4HfSPKnzfrfBW6uqteSbAS+leQfAluBq+kezkcCLwAvJfkZ4F80j9sP+611Mg9SVZXkgjH3p1RMOjseC3yyua0et8kGOj31BXQ+S1DL2UPXIJwHPNqc+XEtcHFVvdz0Gn8H+D/NuPAZwJfonJVxL/AEnfHkTwE0Y9yfAm6i01t/EdgFvHKAY38G+GfNtn8I3NzH57XfWierqh5tnt/+XDfuPPQHx6w7tmnbl4AHgL8PnF1VXxv3GPcCfwM8U1UPTKVOHVziD1zoYJHkCDpjxCur6onZrkeaa+yha05LckGStzZj8L8HfBd4cnarkuYmA11z3YV0PozcAaykM3zjv5XSBBxykaSWsIcuSS0xo6ctHn300bV8+fKZPKQkHfQefPDB56pqqNt2Mxroy5cvZ+tWr94pSZOR5KnuWznkIkmtYaBLUksY6JLUEl7LRVLr/fCHP2RkZISXX57oSsVzx8KFCxkeHmbBggVT2t9Al9R6IyMjHHnkkSxfvpypXgxt0KqK3bt3MzIywooVK6b0GA65SGq9l19+mSVLlszZMAdIwpIlS6b1X4SBLmlemMthvs90azTQJaklHEOXNO9ccEF/H++OO3rb7q677uKKK65g7969fPKTn+TKK6/sax0Gug5oKn/4vf5xS/PJ3r17ufzyy9m8eTPDw8OcdtpprF69mpNOOqlvx3DIRZJmwP3338+73vUuTjjhBA477DAuvvhiNm3a1NdjGOiSNAOeeeYZjj/++B/NDw8P88wzz/T1GAa6JM2AiX57ot9n3hjokjQDhoeHefrpp380PzIywrHHHtvXYxjokjQDTjvtNLZv384TTzzBq6++yk033cTq1av7egzPcpE078zGmViHHnoo1113HR/84AfZu3cvl112Ge95z3v6e4y+Ppokab/OP/98zj///IE9vkMuktQSBroktUTXQE/y7iQPj7m9kOTTSRYn2ZxkezM9aiYKliRNrGugV9VfVNUpVXUK8PPAD4DbgCuBLVW1EtjSzEuSZslkh1zOBb5XVU8BFwIbmuUbgIv6WZgkaXImG+gXAzc294+pqp0AzXRpPwuTJE1Oz6ctJjkMWA1cNZkDJFkHrAN4+9vfPqniJGkgZuH6uZdddhl33nknS5cu5ZFHHunv8RuT6aF/CPh2VT3bzD+bZBlAM9010U5Vtb6qVlXVqqGhoelVK0kHqUsvvZS77rproMeYTKBfwhvDLQC3A2ua+2uA/l4HUpJa5KyzzmLx4sUDPUZPgZ7krcAHgFvHLP488IEk25t1n+9/eZKkXvU0hl5VPwCWjFu2m85ZL5KkOcBvikpSSxjoktQSXm1R0vwzC9fPveSSS7jnnnt47rnnGB4e5uqrr2bt2rV9PYaBLkkz4MYbb+y+0TQ55CJJLWGgS1JLGOiS5oWqmu0SuppujQa6pNZbuHAhu3fvntOhXlXs3r2bhQsXTvkx/FBUUusNDw8zMjLC6OjobJdyQAsXLmR4eHjK+xvoklpvwYIFrFixYrbLGDiHXCSpJeyhq++meqnpWfiuh9Qq9tAlqSUMdElqCQNdklrCQJekljDQJaklPMtFc8ZUzo7xzBjpDfbQJaklev2R6EVJNiZ5LMm2JO9NsjjJ5iTbm+lRgy5WkrR/vfbQrwXuqqqfAU4GtgFXAluqaiWwpZmXJM2SroGe5G3AWcANAFX1alXtAS4ENjSbbQAuGlSRkqTueumhnwCMAv81yUNJrk9yOHBMVe0EaKZLJ9o5ybokW5NsnetXOpOkg1kvgX4o8HPAH1TVqcDfMonhlapaX1WrqmrV0NDQFMuUJHXTS6CPACNVdV8zv5FOwD+bZBlAM901mBIlSb3oGuhV9VfA00ne3Sw6F/hz4HZgTbNsDbBpIBVKknrS6xeLPgV8JclhwOPAP6fzZnBLkrXA94GPDqZESVIvegr0qnoYWDXBqnP7W44kaar8pqgktYSBLkktYaBLUksY6JLUEga6JLWEgS5JLWGgS1JLGOiS1BIGuiS1hIEuSS1hoEtSSxjoktQSBroktYSBLkktYaBLUksY6JLUEga6JLWEgS5JLWGgS1JL9PSbokmeBF4E9gKvVdWqJIuBm4HlwJPAP6mq5wdTpiSpm8n00N9XVadU1b4fi74S2FJVK4EtzbwkaZZMZ8jlQmBDc38DcNH0y5EkTVWvgV7A15I8mGRds+yYqtoJ0EyXTrRjknVJtibZOjo6Ov2KJUkT6mkMHTizqnYkWQpsTvJYrweoqvXAeoBVq1bVFGqUJPWgpx56Ve1opruA24DTgWeTLANoprsGVaQkqbuugZ7k8CRH7rsP/CLwCHA7sKbZbA2waVBFSpK662XI5RjgtiT7tv/jqroryQPALUnWAt8HPjq4MiVJ3XQN9Kp6HDh5guW7gXMHUZQkafL8pqgktYSBLkktYaBLUksY6JLUEga6JLWEgS5JLWGgS1JLGOiS1BIGuiS1hIEuSS1hoEtSSxjoktQSBroktUSvv1ikg9wFF8x2BZIGzR66JLWEgS5JLWGgS1JLGOiS1BI9B3qSQ5I8lOTOZn5FkvuSbE9yc5LDBlemJKmbyfTQrwC2jZm/BvhiVa0EngfW9rMwSdLk9BToSYaBXwKub+YDnANsbDbZAFw0iAIlSb3ptYf++8Bngdeb+SXAnqp6rZkfAY6baMck65JsTbJ1dHR0WsVKkvava6An+TCwq6oeHLt4gk1rov2ran1VraqqVUNDQ1MsU5LUTS/fFD0TWJ3kfGAh8DY6PfZFSQ5teunDwI7BlSlJ6qZroFfVVcBVAEnOBj5TVR9L8j+AjwA3AWuATQOss7Wm8pX8O+7ofx2SDn7TOQ/9N4B/neQv6Yyp39CfkiRJUzGpi3NV1T3APc39x4HT+1+SJGkq/KaoJLWEgS5JLWGgS1JLGOiS1BIGuiS1hIEuSS1hoEtSSxjoktQSk/pikeaGqVwuQFL72UOXpJYw0CWpJQx0SWoJA12SWsJAl6SWMNAlqSUMdElqCQNdklrCQJekljDQJaklugZ6koVJ7k/ynSSPJrm6Wb4iyX1Jtie5Oclhgy9XkrQ/vfTQXwHOqaqTgVOA85KcAVwDfLGqVgLPA2sHV6YkqZuugV4dLzWzC5pbAecAG5vlG4CLBlKhJKknPY2hJzkkycPALmAz8D1gT1W91mwyAhy3n33XJdmaZOvo6Gg/apYkTaCnQK+qvVV1CjAMnA6cONFm+9l3fVWtqqpVQ0NDU69UknRAkzrLpar2APcAZwCLkuy7nvowsKO/pUmSJqOXs1yGkixq7r8FeD+wDbgb+Eiz2Rpg06CKlCR118svFi0DNiQ5hM4bwC1VdWeSPwduSvIfgIeAGwZYpySpi66BXlX/Fzh1guWP0xlPlyTNAX5TVJJawkCXpJYw0CWpJQx0SWoJA12SWsJAl6SWMNAlqSUMdElqCQNdklrCQJekljDQJaklDHRJagkDXZJaopfL50pz1gUXTH6fO+7ofx3SXGAPXZJawkCXpJYw0CWpJQx0SWqJXn4k+vgkdyfZluTRJFc0yxcn2ZxkezM9avDlSpL2p5ce+mvAr1fVicAZwOVJTgKuBLZU1UpgSzMvSZolXQO9qnZW1beb+y8C24DjgAuBDc1mG4CLBlWkJKm7SY2hJ1kOnArcBxxTVTuhE/rA0n4XJ0nqXc+BnuQI4E+AT1fVC5PYb12SrUm2jo6OTqVGSVIPegr0JAvohPlXqurWZvGzSZY165cBuybat6rWV9Wqqlo1NDTUj5olSRPo+tX/JAFuALZV1RfGrLodWAN8vpluGkiFUp9N5XIB4CUDNMZk/4hm6I+nl2u5nAl8HPhukoebZZ+jE+S3JFkLfB/46GBKlCT1omugV9U3gOxn9bn9LUeSNFVebbFPpvpvvCT1i1/9l6SWMNAlqSVaPeTi2QyS5hN76JLUEga6JLVEq4dcpH7y90s119lDl6SWsIcuDZAfzGsm2UOXpJYw0CWpJQx0SWoJA12SWsJAl6SW8CyXCXjlRB20PFl+XrOHLkktYaBLUks45CLNQY6caCrsoUtSS3QN9CRfSrIrySNjli1OsjnJ9mZ61GDLlCR100sP/Y+A88YtuxLYUlUrgS3NvCRpFnUN9Kq6F/jrcYsvBDY09zcAF/W5LknSJE11DP2YqtoJ0EyX7m/DJOuSbE2ydXR0dIqHkyR1M/APRatqfVWtqqpVQ0NDgz6cJM1bUw30Z5MsA2imu/pXkiRpKqYa6LcDa5r7a4BN/SlHkjRVvZy2eCPwTeDdSUaSrAU+D3wgyXbgA828JGkWdf2maFVdsp9V5/a5FknSNPhNUUlqCQNdklrCQJekljDQJaklDHRJagkDXZJawkCXpJYw0CWpJfwJOmm+m6u/dzdX65rD7KFLUksY6JLUEgfNkMtU/vuSpPnEHroktYSBLkktcdAMuUg6yM3EuOk8PzPGHroktYQ9dEmT51kKc5I9dElqCQNdklpiWkMuSc4DrgUOAa6vKn8sWtLBpUXDR1PuoSc5BPhPwIeAk4BLkpzUr8IkSZMznSGX04G/rKrHq+pV4Cbgwv6UJUmarOkMuRwHPD1mfgT4B+M3SrIOWNfMvpTkL6ZxzLngaOC52S5ijrAt3mxW2yOZrSPvl38f+yTTbYt39LLRdAJ9oj+f+rEFVeuB9dM4zpySZGtVrZrtOuYC2+LNbI83sz3eMFNtMZ0hlxHg+DHzw8CO6ZUjSZqq6QT6A8DKJCuSHAZcDNzen7IkSZM15SGXqnotya8Bf0bntMUvVdWjfats7mrN8FEf2BZvZnu8me3xhhlpi1T92LC3JOkg5DdFJaklDHRJagkD/QCSfCnJriSPjFm2OMnmJNub6VGzWeNMSXJ8kruTbEvyaJIrmuXztT0WJrk/yXea9ri6Wb4iyX1Ne9zcnDAwLyQ5JMlDSe5s5udzWzyZ5LtJHk6ytVk28NeKgX5gfwScN27ZlcCWqloJbGnm54PXgF+vqhOBM4DLm0s9zNf2eAU4p6pOBk4BzktyBnAN8MWmPZ4H1s5ijTPtCmDbmPn53BYA76uqU8acfz7w14qBfgBVdS/w1+MWXwhsaO5vAC6a0aJmSVXtrKpvN/dfpPPCPY752x5VVS81swuaWwHnABub5fOmPZIMA78EXN/Mh3naFgcw8NeKgT55x1TVTuiEHLB0luuZcUmWA6cC9zGP26MZYngY2AVsBr4H7Kmq15pNRui86c0Hvw98Fni9mV/C/G0L6Ly5fy3Jg83lT2AGXiv+YpEmJckRwJ8An66qFzIHLyAyU6pqL3BKkkXAbcCJE202s1XNvCQfBnZV1YNJzt63eIJNW98WY5xZVTuSLAU2J3lsJg5qD33ynk2yDKCZ7prlemZMkgV0wvwrVXVrs3jetsc+VbUHuIfOZwuLkuzrKM2Xy2GcCaxO8iSdq66eQ6fHPh/bAoCq2tFMd9F5sz+dGXitGOiTdzuwprm/Btg0i7XMmGZM9AZgW1V9Ycyq+doeQ03PnCRvAd5P53OFu4GPNJvNi/aoqquqariqltO5BMj/rqqPMQ/bAiDJ4UmO3Hcf+EXgEWbgteI3RQ8gyY3A2XQuA/os8G+BrwK3AG8Hvg98tKrGf3DaOkl+Afg68F3eGCf9HJ1x9PnYHj9L54OtQ+h0jG6pqn+f5AQ6vdTFwEPAr1TVK7NX6cxqhlw+U1Ufnq9t0Tzv25rZQ4E/rqrfSbKEAb9WDHRJagmHXCSpJQx0SWoJA12SWsJAl6SWMNAlqSUMdElqCQNdklri/wNZrCGTv5PdRQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light",
      "tags": []
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Histogram highlighting the top 10% we use as a target\n",
    "plt.hist(y[y <= np.percentile(y, 90)], bins='auto', alpha=0.7, label='0', color='b')\n",
    "plt.hist(y[y > np.percentile(y, 90)], bins=8, alpha=0.7, label='1', color='r')\n",
    "plt.title(\"Histogram of MEDV\")\n",
    "plt.legend(loc='upper right')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "OMDuYOK4Rvvg"
   },
   "source": [
    "# Optimizing Scikit-learn GradientBoostingClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zdxYjUFZRvvh"
   },
   "source": [
    "### Gradient Tree Boosting\n",
    "\n",
    "Gradient Tree boosting or Gradient Boosting Decision Trees (GBDT) is another improved version of boosting (fitting a sequence of weak learners on reweighted versions of the data). Like AdaBoost, GBDT is based on a gradient descent function. The algorithm has proven to be one of the most proficient ones from the ensemble, though it is characterized by an increased variance of estimates, more sensibility to noise in data (both problems could be attenuated by using sub-sampling), and significant computational costs due to nonparallel operations.\n",
    "\n",
    "Apart from deep learning, gradient boosting is actually the most developed machine learning algorithm. Since Adaboost and the following Gradient Boosting implementation as developed by Jerome Friedman, there appeared various implementations of the algorithms, the most recent ones being XGBoost, LightGBM, and CatBoost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "PQi5kzxJRvvj"
   },
   "source": [
    "GridSearchCV, RandomizedSearchCV (from Scikit-learn) and BayesSearchCV (from Scikit-optimize) all have the same API. A wrapper can just put together optimization, callbacks, best results reporting and time monitoring."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_FlloD8IRvvk"
   },
   "source": [
    "References:\n",
    "\n",
    "* [https://towardsdatascience.com/entropy-how-decision-trees-make-decisions-2946b9c18c8](https://towardsdatascience.com/entropy-how-decision-trees-make-decisions-2946b9c18c8)\n",
    "\n",
    "* [https://explained.ai/gradient-boosting/](https://explained.ai/gradient-boosting/)\n",
    "* [https://www.youtube.com/watch?v=5CWwwtEM2TA](https://www.youtube.com/watch?v=5CWwwtEM2TA)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gmlVUJfBRvvl"
   },
   "outputs": [],
   "source": [
    "# Reporting util for different optimizers\n",
    "def report_perf(optimizer, X, y, title, callbacks=None):\n",
    "    \"\"\"\n",
    "    A wrapper for measuring time and performances of different optmizers\n",
    "    \n",
    "    optimizer = a sklearn or a skopt optimizer\n",
    "    X = the training set \n",
    "    y = our target\n",
    "    title = a string label for the experiment\n",
    "    \"\"\"\n",
    "    start = time()\n",
    "    if callbacks:\n",
    "        optimizer.fit(X, y, callback=callbacks)\n",
    "    else:\n",
    "        optimizer.fit(X, y)\n",
    "    best_score = optimizer.best_score_\n",
    "    best_score_std = optimizer.cv_results_['std_test_score'][optimizer.best_index_]\n",
    "    best_params = optimizer.best_params_\n",
    "    print((title + \" took %.2f seconds,  candidates checked: %d, best CV score: %.3f \"\n",
    "           +u\"\\u00B1\"+\" %.3f\") % (time() - start, \n",
    "                                  len(optimizer.cv_results_['params']),\n",
    "                                  best_score,\n",
    "                                  best_score_std))    \n",
    "    print('Best parameters:')\n",
    "    pprint.pprint(best_params)\n",
    "    print()\n",
    "    return best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SawWXRMqRvvo"
   },
   "outputs": [],
   "source": [
    "# Converting average precision score into a scorer suitable for model selection\n",
    "avg_prec = make_scorer(average_precision_score, greater_is_better=True, needs_proba=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "SdVfQGAkRvvr"
   },
   "outputs": [],
   "source": [
    "# Setting a 5-fold stratified cross-validation (note: shuffle=True)\n",
    "skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WUPMmC1ZRvvv"
   },
   "outputs": [],
   "source": [
    "# A Scikit-learn GBM classifier\n",
    "clf = GradientBoostingClassifier(n_estimators=20, random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "eL9W2COaRvvy"
   },
   "source": [
    "Grid search exhaustively searches through the hyperparameters and is not feasible in high dimensional space\n",
    "This is a very simple algorithm and suffers from the curse of dimensionality, though it's embarrassingly parallel.\n",
    "\n",
    "Here we use, GridSearchCV, a function from Scikit-learn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "_TT8OHBLRvvz",
    "outputId": "25842825-50c0-4daf-d40b-2034ec95c7e6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GridSearchCV took 59.84 seconds,  candidates checked: 96, best CV score: 0.909 ± 0.072\n",
      "Best parameters:\n",
      "{'learning_rate': 0.01,\n",
      " 'max_features': None,\n",
      " 'min_samples_leaf': 10,\n",
      " 'min_samples_split': 2,\n",
      " 'n_estimators': 500,\n",
      " 'subsample': 0.5}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# GridSearchCV needs a predefined plan of the experiments\n",
    "grid_search = GridSearchCV(clf, \n",
    "                           param_grid={\"learning_rate\": [0.01, 1.0],\n",
    "                                       \"n_estimators\": [10, 500],\n",
    "                                       \"subsample\": [1.0, 0.5],\n",
    "                                       \"min_samples_split\": [2, 10],\n",
    "                                       \"min_samples_leaf\": [1, 10],\n",
    "                                       \"max_features\": ['sqrt', 'log2', None]\n",
    "                                       },\n",
    "                           n_jobs=-1,\n",
    "                           cv=skf,\n",
    "                           scoring=avg_prec,\n",
    "                           iid=False, # just return the average score across folds\n",
    "                           return_train_score=False)\n",
    "\n",
    "best_params = report_perf(grid_search, X, y_bin,'GridSearchCV')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "RMqbcieURvv3"
   },
   "source": [
    "Random search, which simply samples the search space randomly, is feasible in high dimensional spaces, and is widely used in practice. The downside of random search, however, is that it doesn’t use information from prior experiments to select the next setting.\n",
    "\n",
    "You simply need to be lucky to catch the right hyper-parameters, or just try as much as you can ;-).\n",
    "\n",
    "In fact, the 2×Random Search is the Random Search algorithm when it was allowed to sample two points for each point the other algorithms evaluated. While some authors have claimed that 2×Random Search is highly competitive with Bayesian Optimization methods, a [study by Google](http://delivery.acm.org/10.1145/3100000/3098043/p1487-golovin.pdf) (GOLOVIN, Daniel, et al. Google vizier: A service for black-box optimization. In: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 2017. p. 1487-1495) suggests that this is only true when the dimensionality of the problem is sufficiently high (e.g., over 16)\n",
    "\n",
    "RandomizedSearchCV is a function from Scikit-learn, though skopt has it own random optimizer, *[dummy_minimize](https://scikit-optimize.github.io/#skopt.dummy_minimize)*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "R0UKhviMRvv4",
    "outputId": "293d7933-8dc0-4795-e14f-b9fe6345bbbc"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RandomizedSearchCV took 22.92 seconds,  candidates checked: 40, best CV score: 0.927 ± 0.097\n",
      "Best parameters:\n",
      "{'learning_rate': 0.10237388946089819,\n",
      " 'max_features': 'log2',\n",
      " 'min_samples_leaf': 2,\n",
      " 'min_samples_split': 9,\n",
      " 'n_estimators': 384,\n",
      " 'subsample': 0.8266004099285669}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# RandomizedSearchCV needs the distribution of the experiments to be tested\n",
    "# If you can provide the right distribution, the sampling will lead to faster and better results.\n",
    "\n",
    "random_search = RandomizedSearchCV(clf, \n",
    "                                   param_distributions={\"learning_rate\": uniform(0.01, 1.0),\n",
    "                                                        \"n_estimators\": randint(10, 500),\n",
    "                                                        \"subsample\": uniform(0.5, 0.5),\n",
    "                                                        \"min_samples_split\": randint(2, 10),\n",
    "                                                        \"min_samples_leaf\": randint(1, 10),\n",
    "                                                        \"max_features\": ['sqrt', 'log2', None]\n",
    "                                       },\n",
    "                                   n_iter=40,\n",
    "                                   n_jobs=-1,\n",
    "                                   cv=skf,\n",
    "                                   scoring=avg_prec,\n",
    "                                   iid=False, # just return the average score across folds\n",
    "                                   return_train_score=False,\n",
    "                                   random_state=0)\n",
    "\n",
    "best_params = report_perf(random_search, X, y_bin, 'RandomizedSearchCV')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nMCLW6v-Rvv9"
   },
   "source": [
    "After examining the classical and most known approaches, it is time to dwelve into Bayesian optimization.\n",
    "\n",
    "Bayesian optimization is behind [Google Cloud Machine Learning Engine](https://cloud.google.com/blog/products/gcp/hyperparameter-tuning-cloud-machine-learning-engine-using-bayesian-optimization) services.\n",
    "\n",
    "The key idea behind Bayesian optimization is that we optimize a proxy function instead than the true objective function (what actually grid search and random search both do). This holds if testing the true objective function is costly (if it is not, then we simply go for random search :-))\n",
    "\n",
    "Bayesian search balances exploration against exploitation. At start it randomly explores, doing so it builds up a surrogate function of the objective. Based on that surrogate function it exploits an initial approximate knowledge of how the predictor works in order to sample more useful examples and minimize the cost function at a global level, not a local one.\n",
    "\n",
    "As the Bayesian part of the title suggests, we use priors in order to make smarter decisions about sampling during optimizing in order to reach a minimization faster by limiting the number of evaluations we need to make.\n",
    "\n",
    "Bayesian Optimization uses an acquisition function to tell us how promising an observation will be.\n",
    "In fact, to rule the tradeoff between exploration and exploitation, the algorithm defines an acquisition function that provides a single measure of how useful it would be to try any given point.\n",
    "\n",
    "From the figure taken from [Skopt API documentation](https://scikit-optimize.github.io/notebooks/bayesian-optimization.html), you can figure out that the surrogate function (the green dotted line, whose error band is represented by the light green area) has somehow approximated the true cost function (the red dotted line):\n",
    "\n",
    "![figure_1](https://scikit-optimize.github.io/notebooks/bayesian-optimization_files/bayesian-optimization_21_0.png)\n",
    "\n",
    "The observations supporting the construction of the surrogate function are not randomly sparse around, because, through an acquisition function (in a gaussian processes it is a function guiding the selection of the next evaluation points), they have been picked as the most useful examples in order to guess how to minimize the cost function.\n",
    "\n",
    "In respect of a random optimization, a bayesian optimization is more of an educated guess, then, first sampling randomly, but then focussing on the most important combination of hyper-parameters in order to figure out, first the surrogate function of the cost function, then the global minimum of the cost function:\n",
    "\n",
    "![figure_2](https://scikit-optimize.github.io/notebooks/bayesian-optimization_files/bayesian-optimization_18_1.png)\n",
    "\n",
    "Gaussian process (GP) is one of the possible ways to build a surrogate function: it consists of a distribution on functions.\n",
    "Originally GPs were developed to help search for gold ([kriging](https://en.wikipedia.org/wiki/Kriging)). Please note that the approach is closely related to the statistical ideas in the optimal design of experiments.\n",
    "In a gaussian process, based on a distribution of functions resembling the true cost function, the alogorithm operates in:\n",
    "\n",
    "* Exploration -> seeking points and areas on the optimization surface with high variance\n",
    "* Exploitation -> seeking points with low mean\n",
    "\n",
    "This is done by a second, specialized function, the acquisition function.\n",
    "\n",
    "Other approaches are 1) ensembles of decision trees 2) Tree of Parzen Estimators (TPE used by [Hyperopt](http://hyperopt.github.io/hyperopt/) \n",
    "another Bayesian optimization package package) \n",
    "\n",
    "Gaussian Processes are just models, and they're much more like k-nearest neighbors and linear regression than may at first be apparent. If you want to understand more of GPs, you can read the post: [https://planspace.org/20181226-gaussian_processes_are_not_so_fancy/](https://planspace.org/20181226-gaussian_processes_are_not_so_fancy)by Aaron Schumacher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "719HE3ykRvv-",
    "outputId": "0959ee6d-3f4a-4c7b-9eea-3f2689fd8648"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BayesSearchCV_GP took 138.81 seconds,  candidates checked: 40, best CV score: 0.928 ± 0.062\n",
      "Best parameters:\n",
      "{'learning_rate': 0.7184419933944712,\n",
      " 'max_features': 'log2',\n",
      " 'min_samples_leaf': 10,\n",
      " 'min_samples_split': 3,\n",
      " 'n_estimators': 347,\n",
      " 'subsample': 1.0}\n",
      "\n",
      "BayesSearchCV_RF took 124.22 seconds,  candidates checked: 40, best CV score: 0.945 ± 0.051\n",
      "Best parameters:\n",
      "{'learning_rate': 0.6049627162690525,\n",
      " 'max_features': 'sqrt',\n",
      " 'min_samples_leaf': 9,\n",
      " 'min_samples_split': 10,\n",
      " 'n_estimators': 485,\n",
      " 'subsample': 0.6024816853254407}\n",
      "\n",
      "BayesSearchCV_ET took 101.51 seconds,  candidates checked: 40, best CV score: 0.933 ± 0.059\n",
      "Best parameters:\n",
      "{'learning_rate': 0.39752021916372565,\n",
      " 'max_features': None,\n",
      " 'min_samples_leaf': 6,\n",
      " 'min_samples_split': 9,\n",
      " 'n_estimators': 78,\n",
      " 'subsample': 0.7687211202072857}\n",
      "\n",
      "BayesSearchCV_GBRT took 95.73 seconds,  candidates checked: 40, best CV score: 0.933 ± 0.068\n",
      "Best parameters:\n",
      "{'learning_rate': 0.42127830500668484,\n",
      " 'max_features': 'log2',\n",
      " 'min_samples_leaf': 4,\n",
      " 'min_samples_split': 8,\n",
      " 'n_estimators': 377,\n",
      " 'subsample': 0.5657650066445118}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# also BayesSearchCV needs to work on the distributions of the experiments but it is less sensible to them\n",
    "\n",
    "search_spaces = {\"learning_rate\": Real(0.01, 1.0),\n",
    "                 \"n_estimators\": Integer(10, 500),\n",
    "                 \"subsample\": Real(0.5, 1.0),\n",
    "                 \"min_samples_split\": Integer(2, 10),\n",
    "                 \"min_samples_leaf\": Integer(1, 10),\n",
    "                 \"max_features\": Categorical(categories=['sqrt', 'log2', None])}\n",
    "\n",
    "for baseEstimator in ['GP', 'RF', 'ET', 'GBRT']:\n",
    "    opt = BayesSearchCV(clf,\n",
    "                        search_spaces,\n",
    "                        scoring=avg_prec,\n",
    "                        cv=skf,\n",
    "                        n_iter=40,\n",
    "                        n_jobs=-1,\n",
    "                        return_train_score=False,\n",
    "                        optimizer_kwargs={'base_estimator': baseEstimator},\n",
    "                        random_state=4)\n",
    "    \n",
    "    best_params = report_perf(opt, X, y_bin,'BayesSearchCV_'+baseEstimator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hUvgEZ4ORvwF"
   },
   "source": [
    "## Searching more complex spaces\n",
    "If you have multiple models to optimize, you can leverage the *Pipeline* command in order to search different search spaces based on different models. That's requires to access to the hyper-parameters accordingly to *Pipeline* specifications, anyway."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ryetw5AqRvwG",
    "outputId": "b6145df6-3ae2-4e1d-b599-638d2137996b"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Luca\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n",
      "C:\\Users\\Luca\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n",
      "C:\\Users\\Luca\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n",
      "C:\\Users\\Luca\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n",
      "C:\\Users\\Luca\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n",
      "C:\\Users\\Luca\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n",
      "C:\\Users\\Luca\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n",
      "C:\\Users\\Luca\\Anaconda3\\lib\\site-packages\\skopt\\optimizer\\optimizer.py:399: UserWarning: The objective has been evaluated at this point before.\n",
      "  warnings.warn(\"The objective has been evaluated \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BayesSearchCV_GP took 106.81 seconds,  candidates checked: 40, best CV score: 0.920 ± 0.077\n",
      "Best parameters:\n",
      "{'model': GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
      "              learning_rate=0.2554001338861434, loss='deviance',\n",
      "              max_depth=3, max_features='sqrt', max_leaf_nodes=None,\n",
      "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
      "              min_samples_leaf=6, min_samples_split=7,\n",
      "              min_weight_fraction_leaf=0.0, n_estimators=225,\n",
      "              n_iter_no_change=None, presort='auto', random_state=0,\n",
      "              subsample=0.7900947461882674, tol=0.0001,\n",
      "              validation_fraction=0.1, verbose=0, warm_start=False),\n",
      " 'model__learning_rate': 0.2554001338861434,\n",
      " 'model__max_features': 'sqrt',\n",
      " 'model__min_samples_leaf': 6,\n",
      " 'model__min_samples_split': 7,\n",
      " 'model__n_estimators': 225,\n",
      " 'model__subsample': 0.7900947461882674}\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Initialize a pipeline with a model\n",
    "pipe = Pipeline([('model', GradientBoostingClassifier(n_estimators=20, random_state=0))])\n",
    "\n",
    "# Define search space for GBM;\n",
    "search_space_GBM = {\"model\": Categorical([GradientBoostingClassifier(n_estimators=20, random_state=0)]),\n",
    "                    \"model__learning_rate\": Real(0.01, 1.0),\n",
    "                    \"model__n_estimators\": Integer(10, 500),\n",
    "                    \"model__subsample\": Real(0.5, 1.0),\n",
    "                    \"model__min_samples_split\": Integer(2, 10),\n",
    "                    \"model__min_samples_leaf\": Integer(1, 10),\n",
    "                    \"model__max_features\": Categorical(categories=['sqrt', 'log2', None])}\n",
    "\n",
    "# Define search space for RF\n",
    "search_space_RF  = {\"model\": Categorical([RandomForestClassifier(n_estimators=20, random_state=0)]),\n",
    "                    \"model__n_estimators\": Integer(10, 200),\n",
    "                    \"model__min_samples_split\": Integer(2, 10),\n",
    "                    \"model__min_samples_leaf\": Integer(1, 10),\n",
    "                    \"model__max_features\": Categorical(categories=['sqrt', 'log2', None])}\n",
    "\n",
    "opt = BayesSearchCV(pipe,\n",
    "                        search_spaces=[(search_space_GBM, 20), (search_space_RF, 20)],\n",
    "                        scoring=avg_prec,\n",
    "                        cv=skf,\n",
    "                        n_jobs=-1,\n",
    "                        return_train_score=False,\n",
    "                        optimizer_kwargs={'base_estimator': 'GP'},\n",
    "                        random_state=4)\n",
    "    \n",
    "best_params = report_perf(opt, X, y_bin,'BayesSearchCV_GP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "15H1-CenRvwM"
   },
   "source": [
    "## Controlling the time cost of Bayesian optimization\n",
    "\n",
    "Running a single LightGBM model could take long time and in a Kaggle competition time is often a luxury. \n",
    "\n",
    "*DeadlineStopper* and *DeltaXStopper* are skopt callbacks that control the total time spent and the improvement of a BayesSearchCV (in our implementation to be called with *report_perf*, using the parameter *callbacks=[]*). \n",
    "\n",
    "Anyway, sometimes it is easier to control manually the optimization steps, hence the usage of low-level optimizers. \n",
    "\n",
    "We start defining a custom callback, using a different approach to search spaces (a list instead of a dictionary), and to manually create our objective function to be minimized.\n",
    "\n",
    "In our custom callback, we print the last evaluation point (so you know what's happening) and the best score and parameters foudn up so far. We also record the list of explored points (*x0*) and their relative results (*y0*). This will help us to reprise the learning at a later time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aDI5sUthRvwO"
   },
   "outputs": [],
   "source": [
    "counter = 0\n",
    "def onstep(res):\n",
    "    global counter\n",
    "    x0 = res.x_iters   # List of input points\n",
    "    y0 = res.func_vals # Evaluation of input points\n",
    "    print('Last eval: ', x0[-1], \n",
    "          ' - Score ', y0[-1])\n",
    "    print('Current iter: ', counter, \n",
    "          ' - Score ', res.fun, \n",
    "          ' - Args: ', res.x)\n",
    "    joblib.dump((x0, y0), 'checkpoint.pkl') # Saving a checkpoint to disk\n",
    "    counter += 1\n",
    "\n",
    "# Our search space\n",
    "dimensions = [Real(0.01, 1.0, name=\"learning_rate\"),\n",
    "              Integer(10, 500, name=\"n_estimators\"),\n",
    "              Real(0.5, 1.0, name=\"subsample\"),\n",
    "              Integer(2, 10, name=\"min_samples_split\"),\n",
    "              Integer(1, 10, name=\"min_samples_leaf\"),\n",
    "              Categorical(categories=['sqrt', 'log2', None], name=\"max_features\")]\n",
    "\n",
    "# The objective function to be minimized\n",
    "def make_objective(model, X, y, space, cv, scoring):\n",
    "    # This decorator converts your objective function with named arguments into one that\n",
    "    # accepts a list as argument, while doing the conversion automatically.\n",
    "    @use_named_args(space) \n",
    "    def objective(**params):\n",
    "        model.set_params(**params)\n",
    "        return -np.mean(cross_val_score(model, \n",
    "                                        X, y, \n",
    "                                        cv=cv, \n",
    "                                        n_jobs=-1,\n",
    "                                        scoring=scoring))\n",
    "\n",
    "    return objective\n",
    "\n",
    "objective = make_objective(clf,\n",
    "                           X, y_bin,\n",
    "                           space=dimensions,\n",
    "                           cv=skf,\n",
    "                           scoring=avg_prec)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5sPMx6Z8RvwV"
   },
   "source": [
    "There are different low-level optimizers that can be used for the purpose:\n",
    "* **gp_minimize** Bayesian optimization using Gaussian Processes.\n",
    "* **forest_minimize** Sequential optimisation using decision trees\n",
    "* **gbrt_minimize** Sequential optimization using gradient boosted trees\n",
    "* **dummy_minimize** Random search by uniform sampling within the given bounds (a replacement for Scikit-learn's RandomSearch)\n",
    "\n",
    "Each optimizer has its own parameters, so they cannot be just automatically switched, though they share most of the key parameters.\n",
    "\n",
    "Here we encounter also a new parameter, **acq_func**, useful for defining how the acquisition function should behave, that is, if to take as minimum the lower confidence bound, the minimum expected value or probability (the suggested default is usually a good choice)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Ry4T-E5lRvwW",
    "outputId": "c313824e-140f-4ff9-b987-a1e0235cff71"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last eval:  [0.7049433514242192, 368, 0.8187395776032731, 8, 9, None]  - Score  -0.9053516521792384\n",
      "Current iter:  0  - Score  -0.9053516521792384  - Args:  [0.7049433514242192, 368, 0.8187395776032731, 8, 9, None]\n",
      "Last eval:  [0.24229127648570348, 16, 0.7320268705932458, 8, 2, 'sqrt']  - Score  -0.8892368868276309\n",
      "Current iter:  1  - Score  -0.9053516521792384  - Args:  [0.7049433514242192, 368, 0.8187395776032731, 8, 9, None]\n",
      "Last eval:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']  - Score  -0.9234068462086364\n",
      "Current iter:  2  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.010263121534902437, 442, 0.972884851277688, 9, 2, 'sqrt']  - Score  -0.9049683094683093\n",
      "Current iter:  3  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.8875384524348883, 205, 0.9771110531385809, 8, 8, None]  - Score  -0.8977127929278959\n",
      "Current iter:  4  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.9408844987095382, 278, 0.7733463688856329, 7, 2, None]  - Score  -0.7844031794538096\n",
      "Current iter:  5  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.3343579995412422, 367, 0.5956932474515344, 8, 9, 'log2']  - Score  -0.9143859278549682\n",
      "Current iter:  6  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.7879021133849456, 252, 0.75143818812381, 8, 5, 'log2']  - Score  -0.5935980587554553\n",
      "Current iter:  7  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.8164027897144066, 346, 0.6236194997341288, 6, 7, 'sqrt']  - Score  -0.8692463647463649\n",
      "Current iter:  8  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.7856253596534273, 448, 0.7012180692236435, 9, 8, 'sqrt']  - Score  -0.753150122881838\n",
      "Current iter:  9  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n"
     ]
    }
   ],
   "source": [
    "gp_round = gp_minimize(func=objective,\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge', # Defining what to minimize \n",
    "                       n_calls=10,\n",
    "                       callback=[onstep],\n",
    "                       random_state=22)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xAZPhsdURvwa",
    "outputId": "b1fb48ed-a7ad-4d1c-e4bb-f6f4512a31b2"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last eval:  [0.7856253596534273, 448, 0.7012180692236435, 9, 8, 'sqrt']  - Score  -0.753150122881838\n",
      "Current iter:  10  - Score  -0.9234068462086364  - Args:  [0.09808017080760961, 436, 0.721036739670265, 3, 4, 'log2']\n",
      "Last eval:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']  - Score  -0.9433474858474857\n",
      "Current iter:  11  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.3045592604790276, 38, 0.6363281472900566, 6, 8, 'log2']  - Score  -0.9160759254036765\n",
      "Current iter:  12  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.39885694813982153, 420, 0.6686980802086342, 7, 4, None]  - Score  -0.8992387417077818\n",
      "Current iter:  13  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.14894727260851873, 436, 0.7368040226368553, 8, 6, None]  - Score  -0.9172835775335775\n",
      "Current iter:  14  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.7234263281786577, 295, 0.7686866147245054, 8, 2, 'log2']  - Score  -0.9011155635306421\n",
      "Current iter:  15  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.1944690198934924, 371, 0.6082751772121859, 3, 4, 'sqrt']  - Score  -0.9163157411434923\n",
      "Current iter:  16  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.23009817436907185, 199, 0.9512992377647025, 6, 7, None]  - Score  -0.901292475498358\n",
      "Current iter:  17  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.10828754685538415, 485, 0.8265700178989689, 3, 4, None]  - Score  -0.9187638287638288\n",
      "Current iter:  18  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.6117523620283132, 169, 0.5192127132363674, 7, 10, 'log2']  - Score  -0.8649301045063078\n",
      "Current iter:  19  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.6387082848675283, 498, 0.7909251647192672, 5, 5, 'log2']  - Score  -0.7693444638581505\n",
      "Current iter:  20  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n"
     ]
    }
   ],
   "source": [
    "x0, y0 = joblib.load('checkpoint.pkl')\n",
    "\n",
    "gp_round = gp_minimize(func=objective,\n",
    "                       x0=x0,              # already examined values for x\n",
    "                       y0=y0,              # observed values for x0\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge', # Expected Improvement.\n",
    "                       n_calls=10,\n",
    "                       callback=[onstep],\n",
    "                       random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "4MlCsu12Rvwe",
    "outputId": "85d43294-7a66-47ea-c751-a9c788fff61b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2'] -0.9433474858474857\n"
     ]
    }
   ],
   "source": [
    "best_parameters = gp_round.x\n",
    "best_result = gp_round.fun\n",
    "print(best_parameters, best_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "TQ7I1aeiRvwj"
   },
   "source": [
    "In conclusion, just keep in mind a few points from the workshop:\n",
    "\n",
    "* Bayesian Optimization has its own hyper-parameters (therefore use defaults, [unless you know what you doing](https://i0.kym-cdn.com/entries/icons/original/000/008/342/ihave.jpg))\n",
    "\n",
    "* Experiments are run sequentially (skopt can leverage some parallelism, though), having multiple cores is helpful for your learning algorithm,but Bayesian Optimization will always be slower than Random Search. Use it only when needed.\n",
    "\n",
    "* Packages are not all that friendly (hence the workshop :-)) but you can reuse some simple wrappers re-adaptable to being used in different Kaggle competitions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "v6ET90r_Rvwk"
   },
   "source": [
    "## Hacking Bayesian optimization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "aXLBCm86Rvwl"
   },
   "outputs": [],
   "source": [
    "# Our search space\n",
    "\n",
    "dimensions = [Real(0.01, 1.0, name=\"learning_rate\"),\n",
    "              Integer(10, 500, name=\"n_estimators\"),\n",
    "              Real(0.5, 1.0, name=\"subsample\"),\n",
    "              Integer(2, 10, name=\"min_samples_split\"),\n",
    "              Integer(1, 10, name=\"min_samples_leaf\"),\n",
    "              Categorical(categories=['sqrt', 'log2', None], name=\"max_features\")]\n",
    "\n",
    "rand_opt_dims = {\"learning_rate\": uniform(0.01, 1.0),\n",
    "                 \"n_estimators\": randint(10, 500),\n",
    "                 \"subsample\": uniform(0.5, 0.5),\n",
    "                 \"min_samples_split\": randint(2, 10),\n",
    "                 \"min_samples_leaf\": randint(1, 10),\n",
    "                 \"max_features\": ['sqrt', 'log2', None]\n",
    "                }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zsEDBBGRRvwp",
    "outputId": "4113e7c9-fe63-46fb-c538-7a2b8b3d9ecc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=StratifiedKFold(n_splits=5, random_state=0, shuffle=True),\n",
       "          error_score='raise-deprecating',\n",
       "          estimator=GradientBoostingClassifier(criterion='friedman_mse', init=None,\n",
       "              learning_rate=0.1, loss='deviance', max_depth=3,\n",
       "              max_features=None, max_leaf_nodes=None,\n",
       "              min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "              min_samples_leaf=1, min_sampl...      subsample=1.0, tol=0.0001, validation_fraction=0.1,\n",
       "              verbose=0, warm_start=False),\n",
       "          fit_params=None, iid=False, n_iter=60, n_jobs=-1,\n",
       "          param_distributions={'learning_rate': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000000000C578400>, 'n_estimators': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000000000C5584A8>, 'subsample': <scipy.stats._distn_infrastructure.rv_frozen object at 0x000000000C558668>, 'min_...istn_infrastructure.rv_frozen object at 0x000000000C1323C8>, 'max_features': ['sqrt', 'log2', None]},\n",
       "          pre_dispatch='2*n_jobs', random_state=0, refit=True,\n",
       "          return_train_score=False,\n",
       "          scoring=make_scorer(average_precision_score, needs_proba=True),\n",
       "          verbose=0)"
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Converting average precision score into a scorer suitable for model selection\n",
    "avg_prec = make_scorer(average_precision_score, greater_is_better=True, needs_proba=True)\n",
    "\n",
    "# Setting a 5-fold stratified cross-validation (note: shuffle=True)\n",
    "skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=0)\n",
    "\n",
    "# A Scikit-learn GBM classifier\n",
    "clf = GradientBoostingClassifier(n_estimators=20, random_state=0)\n",
    "\n",
    "random_search = RandomizedSearchCV(clf, \n",
    "                                   param_distributions=rand_opt_dims, \n",
    "                                   n_iter=60,\n",
    "                                   n_jobs=-1,\n",
    "                                   cv=skf,\n",
    "                                   scoring=avg_prec,\n",
    "                                   iid=False, # just return the average score across folds\n",
    "                                   return_train_score=False,\n",
    "                                   random_state=0)\n",
    "\n",
    "random_search.fit(X, y_bin)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jjh6zPm5Rvwt",
    "outputId": "f5bbb6a1-a032-4455-99fd-7d2b22f58393"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best result by random search: 0.93595\n"
     ]
    }
   ],
   "source": [
    "print(\"Best result by random search: %0.5f\" % random_search.best_score_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "904p4nH7Rvwx",
    "outputId": "dcbf55b3-af2c-4643-8415-f7164e368569"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['mean_fit_time', 'std_fit_time', 'mean_score_time', 'std_score_time', 'param_learning_rate', 'param_max_features', 'param_min_samples_leaf', 'param_min_samples_split', 'param_n_estimators', 'param_subsample', 'params', 'split0_test_score', 'split1_test_score', 'split2_test_score', 'split3_test_score', 'split4_test_score', 'mean_test_score', 'std_test_score', 'rank_test_score'])"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "random_search.cv_results_.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "f-gRAx_pRvw2"
   },
   "outputs": [],
   "source": [
    "params = [\"learning_rate\", \"n_estimators\", \"subsample\", \"min_samples_split\", \"min_samples_leaf\", \"max_features\"]\n",
    "x0 = [[ex[p] for p in params] for ex in random_search.cv_results_['params']]\n",
    "y0 = random_search.cv_results_['mean_test_score'] * -1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UCU11Hq6Rvw5",
    "outputId": "dea8b710-0065-4c79-b268-53daf7cae840"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Last eval:  [0.11029394226549781, 177, 0.834958273295455, 8, 7, 'log2']  - Score  -0.922229074846722\n",
      "Current iter:  21  - Score  -0.9359512756354189  - Args:  [0.2223904988903086, 406, 0.784786767287369, 7, 6, None]\n",
      "Last eval:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']  - Score  -0.9433474858474857\n",
      "Current iter:  22  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.3045592604790276, 38, 0.6363281472900566, 6, 8, 'log2']  - Score  -0.9160759254036765\n",
      "Current iter:  23  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.39885694813982153, 420, 0.6686980802086342, 7, 4, None]  - Score  -0.8992387417077818\n",
      "Current iter:  24  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.14894727260851873, 436, 0.7368040226368553, 8, 6, None]  - Score  -0.9172835775335775\n",
      "Current iter:  25  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.7234263281786577, 295, 0.7686866147245054, 8, 2, 'log2']  - Score  -0.9011155635306421\n",
      "Current iter:  26  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.1944690198934924, 371, 0.6082751772121859, 3, 4, 'sqrt']  - Score  -0.9163157411434923\n",
      "Current iter:  27  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.23009817436907185, 199, 0.9512992377647025, 6, 7, None]  - Score  -0.901292475498358\n",
      "Current iter:  28  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.10828754685538415, 485, 0.8265700178989689, 3, 4, None]  - Score  -0.9187638287638288\n",
      "Current iter:  29  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.6117523620283132, 169, 0.5192127132363674, 7, 10, 'log2']  - Score  -0.8649301045063078\n",
      "Current iter:  30  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n",
      "Last eval:  [0.6387082848675283, 498, 0.7909251647192672, 5, 5, 'log2']  - Score  -0.7693444638581505\n",
      "Current iter:  31  - Score  -0.9433474858474857  - Args:  [0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2']\n"
     ]
    }
   ],
   "source": [
    "# The objective function to be minimized\n",
    "def make_objective(model, X, y, space, cv, scoring):\n",
    "    # This decorator converts your objective function with named arguments into one that\n",
    "    # accepts a list as argument, while doing the conversion automatically.\n",
    "    @use_named_args(space) \n",
    "    def objective(**params):\n",
    "        model.set_params(**params)\n",
    "        return -np.mean(cross_val_score(model, \n",
    "                                        X, y, \n",
    "                                        cv=cv, \n",
    "                                        n_jobs=-1,\n",
    "                                        scoring=scoring))\n",
    "\n",
    "    return objective\n",
    "\n",
    "objective = make_objective(clf,\n",
    "                           X, y_bin,\n",
    "                           space=dimensions,\n",
    "                           cv=skf,\n",
    "                           scoring=avg_prec)\n",
    "\n",
    "\n",
    "gp_round = gp_minimize(func=objective,\n",
    "                       x0=x0,              # already examined values for x\n",
    "                       y0=y0,              # observed values for x0\n",
    "                       dimensions=dimensions,\n",
    "                       acq_func='gp_hedge', # Expected Improvement.\n",
    "                       n_calls=10,\n",
    "                       callback=[onstep],\n",
    "                       random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 0,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "C3KFiuCNRvw9",
    "outputId": "612dc1cd-a8e0-40f7-b78b-cb6982012727"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.5969161720427683, 424, 0.9289728088113784, 9, 7, 'log2'] -0.9433474858474857\n"
     ]
    }
   ],
   "source": [
    "best_parameters = gp_round.x\n",
    "best_result = gp_round.fun\n",
    "print(best_parameters, best_result)"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "hUvgEZ4ORvwF",
    "15H1-CenRvwM",
    "v6ET90r_Rvwk"
   ],
   "name": "Copy of skopt_workshop.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}