experimental_feature_and_ensemble_models.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Modeling Process With Experimental Feature Database"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook we use our experimental database to identify key features and tune the hyperparameters of several high preforming models. For model selection, we where looking for models where we could best improve the precision(reduce false positives) while not drastically reducing recall (false negatives).  To evaluate the models we used sklearn metrics and also viewed the confusion matrix to see how the models predictions on the test set preformed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T02:57:13.515882Z",
     "start_time": "2021-03-17T02:57:11.716434Z"
    }
   },
   "outputs": [],
   "source": [
    "# Import Packages\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "# Sklearn Packages\n",
    "from sklearn.model_selection import train_test_split, GridSearchCV, cross_val_score\n",
    "\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier, BaggingClassifier\n",
    "\n",
    "\n",
    "from sklearn import set_config\n",
    "from sklearn import metrics\n",
    "from sklearn.metrics import mean_squared_error, precision_score, confusion_matrix, accuracy_score\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.exceptions import ConvergenceWarning\n",
    "from imblearn.over_sampling import SMOTE\n",
    "from xgboost import XGBClassifier\n",
    "from xgboost import plot_importance\n",
    "\n",
    "set_config(print_changed_only=False)\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings(\"ignore\")\n",
    "\n",
    "pd.set_option('display.max_columns', 300)\n",
    "% matplotlib inline\n",
    "\n",
    "plt.style.use('seaborn')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T02:57:13.931529Z",
     "start_time": "2021-03-17T02:57:13.522425Z"
    }
   },
   "outputs": [],
   "source": [
    "# Importing all functions used in this notebook\n",
    "from functions import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T02:57:13.989004Z",
     "start_time": "2021-03-17T02:57:13.933670Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>baseline value</th>\n",
       "      <th>accelerations</th>\n",
       "      <th>fetal_movement</th>\n",
       "      <th>uterine_contractions</th>\n",
       "      <th>light_decelerations</th>\n",
       "      <th>prolongued_decelerations</th>\n",
       "      <th>abnormal_short_term_variability</th>\n",
       "      <th>mean_value_of_short_term_variability</th>\n",
       "      <th>percentage_of_time_with_abnormal_long_term_variability</th>\n",
       "      <th>mean_value_of_long_term_variability</th>\n",
       "      <th>histogram_width</th>\n",
       "      <th>histogram_min</th>\n",
       "      <th>histogram_max</th>\n",
       "      <th>histogram_number_of_peaks</th>\n",
       "      <th>histogram_mode</th>\n",
       "      <th>histogram_mean</th>\n",
       "      <th>histogram_median</th>\n",
       "      <th>histogram_variance</th>\n",
       "      <th>fetal_health</th>\n",
       "      <th>uterine_cont_per_min</th>\n",
       "      <th>total_change</th>\n",
       "      <th>sqrt_total_change</th>\n",
       "      <th>hist_zeros_1.0</th>\n",
       "      <th>hist_zeros_2.0</th>\n",
       "      <th>hist_zeros_3.0</th>\n",
       "      <th>hist_zeros_4.0</th>\n",
       "      <th>hist_zeros_5.0</th>\n",
       "      <th>hist_zeros_7.0</th>\n",
       "      <th>hist_zeros_8.0</th>\n",
       "      <th>hist_zeros_10.0</th>\n",
       "      <th>hist_tendancy_0.0</th>\n",
       "      <th>hist_tendancy_1.0</th>\n",
       "      <th>sev_decel_0.001</th>\n",
       "      <th>quant_acc_1</th>\n",
       "      <th>quant_light_dec_1</th>\n",
       "      <th>quant_hist_mean_1</th>\n",
       "      <th>quant_hist_mean_2</th>\n",
       "      <th>quant_hist_mean_3</th>\n",
       "      <th>quant_hist_mean_4</th>\n",
       "      <th>quant_hist_mean_5</th>\n",
       "      <th>quant_hist_mean_6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>120.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>43.0</td>\n",
       "      <td>2.4</td>\n",
       "      <td>64.0</td>\n",
       "      <td>62.0</td>\n",
       "      <td>126.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>121.0</td>\n",
       "      <td>73.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>132.0</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.006</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10.4</td>\n",
       "      <td>130.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>6.0</td>\n",
       "      <td>141.0</td>\n",
       "      <td>136.0</td>\n",
       "      <td>140.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.36</td>\n",
       "      <td>1.980</td>\n",
       "      <td>0.122474</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>133.0</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>13.4</td>\n",
       "      <td>130.0</td>\n",
       "      <td>68.0</td>\n",
       "      <td>198.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>141.0</td>\n",
       "      <td>135.0</td>\n",
       "      <td>138.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.48</td>\n",
       "      <td>1.862</td>\n",
       "      <td>0.118322</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>134.0</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008</td>\n",
       "      <td>0.003</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>23.0</td>\n",
       "      <td>117.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>134.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.48</td>\n",
       "      <td>1.876</td>\n",
       "      <td>0.118322</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>132.0</td>\n",
       "      <td>0.007</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.008</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16.0</td>\n",
       "      <td>2.4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>19.9</td>\n",
       "      <td>117.0</td>\n",
       "      <td>53.0</td>\n",
       "      <td>170.0</td>\n",
       "      <td>9.0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>136.0</td>\n",
       "      <td>138.0</td>\n",
       "      <td>11.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.48</td>\n",
       "      <td>1.980</td>\n",
       "      <td>0.122474</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   baseline value  accelerations  fetal_movement  uterine_contractions  \\\n",
       "0           120.0          0.000             0.0                 0.000   \n",
       "1           132.0          0.006             0.0                 0.006   \n",
       "2           133.0          0.003             0.0                 0.008   \n",
       "3           134.0          0.003             0.0                 0.008   \n",
       "4           132.0          0.007             0.0                 0.008   \n",
       "\n",
       "   light_decelerations  prolongued_decelerations  \\\n",
       "0                0.000                       0.0   \n",
       "1                0.003                       0.0   \n",
       "2                0.003                       0.0   \n",
       "3                0.003                       0.0   \n",
       "4                0.000                       0.0   \n",
       "\n",
       "   abnormal_short_term_variability  mean_value_of_short_term_variability  \\\n",
       "0                             73.0                                   0.5   \n",
       "1                             17.0                                   2.1   \n",
       "2                             16.0                                   2.1   \n",
       "3                             16.0                                   2.4   \n",
       "4                             16.0                                   2.4   \n",
       "\n",
       "   percentage_of_time_with_abnormal_long_term_variability  \\\n",
       "0                                               43.0        \n",
       "1                                                0.0        \n",
       "2                                                0.0        \n",
       "3                                                0.0        \n",
       "4                                                0.0        \n",
       "\n",
       "   mean_value_of_long_term_variability  histogram_width  histogram_min  \\\n",
       "0                                  2.4             64.0           62.0   \n",
       "1                                 10.4            130.0           68.0   \n",
       "2                                 13.4            130.0           68.0   \n",
       "3                                 23.0            117.0           53.0   \n",
       "4                                 19.9            117.0           53.0   \n",
       "\n",
       "   histogram_max  histogram_number_of_peaks  histogram_mode  histogram_mean  \\\n",
       "0          126.0                        2.0           120.0           137.0   \n",
       "1          198.0                        6.0           141.0           136.0   \n",
       "2          198.0                        5.0           141.0           135.0   \n",
       "3          170.0                       11.0           137.0           134.0   \n",
       "4          170.0                        9.0           137.0           136.0   \n",
       "\n",
       "   histogram_median  histogram_variance  fetal_health  uterine_cont_per_min  \\\n",
       "0             121.0                73.0           2.0                  0.00   \n",
       "1             140.0                12.0           1.0                  0.36   \n",
       "2             138.0                13.0           1.0                  0.48   \n",
       "3             137.0                13.0           1.0                  0.48   \n",
       "4             138.0                11.0           1.0                  0.48   \n",
       "\n",
       "   total_change  sqrt_total_change  hist_zeros_1.0  hist_zeros_2.0  \\\n",
       "0         0.000           0.000000               0               0   \n",
       "1         1.980           0.122474               1               0   \n",
       "2         1.862           0.118322               1               0   \n",
       "3         1.876           0.118322               0               0   \n",
       "4         1.980           0.122474               0               0   \n",
       "\n",
       "   hist_zeros_3.0  hist_zeros_4.0  hist_zeros_5.0  hist_zeros_7.0  \\\n",
       "0               0               0               0               0   \n",
       "1               0               0               0               0   \n",
       "2               0               0               0               0   \n",
       "3               0               0               0               0   \n",
       "4               0               0               0               0   \n",
       "\n",
       "   hist_zeros_8.0  hist_zeros_10.0  hist_tendancy_0.0  hist_tendancy_1.0  \\\n",
       "0               0                0                  0                  1   \n",
       "1               0                0                  1                  0   \n",
       "2               0                0                  1                  0   \n",
       "3               0                0                  0                  1   \n",
       "4               0                0                  0                  1   \n",
       "\n",
       "   sev_decel_0.001  quant_acc_1  quant_light_dec_1  quant_hist_mean_1  \\\n",
       "0                0            0                  0                  0   \n",
       "1                0            0                  0                  0   \n",
       "2                0            0                  0                  0   \n",
       "3                0            0                  0                  0   \n",
       "4                0            1                  0                  0   \n",
       "\n",
       "   quant_hist_mean_2  quant_hist_mean_3  quant_hist_mean_4  quant_hist_mean_5  \\\n",
       "0                  0                  1                  0                  0   \n",
       "1                  0                  1                  0                  0   \n",
       "2                  0                  1                  0                  0   \n",
       "3                  0                  1                  0                  0   \n",
       "4                  0                  1                  0                  0   \n",
       "\n",
       "   quant_hist_mean_6  \n",
       "0                  0  \n",
       "1                  0  \n",
       "2                  0  \n",
       "3                  0  \n",
       "4                  0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Read in dataframe\n",
    "df = pd.read_csv('experiment_features.csv',index_col=0)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T02:57:13.995178Z",
     "start_time": "2021-03-17T02:57:13.991131Z"
    }
   },
   "outputs": [],
   "source": [
    "# # Function Prints best parameters for GridSearchCV\n",
    "# def print_results(results):\n",
    "#     print('Best Parameters: {}\\n'.format(results.best_params_))   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T02:59:22.870182Z",
     "start_time": "2021-03-17T02:59:22.861873Z"
    }
   },
   "outputs": [],
   "source": [
    "#train test split of data\n",
    "X = df.drop('fetal_health', axis =1)\n",
    "y = df.fetal_health\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= 0.25, random_state=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T02:59:23.128757Z",
     "start_time": "2021-03-17T02:59:23.117657Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "StandardScaler(copy=True, with_mean=True, with_std=True)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#standared scaler for predicting features\n",
    "scaler = StandardScaler()  \n",
    "scaler.fit(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Class imbalance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- To handle class imbalance we oversampled the minority class using SMOTE(Synthetic Minority Oversampling Technique), this balanced the minority class by sampling the nearest neighboors and adding points between the neighbors.  We used both a SMOTE sampled database and unbalanced database to compare the effect of the metrics for each experimental baseline model.  The Smote Database preformed better on every baseline model, we chose to use the SMOTE database for our final models.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T02:59:24.944931Z",
     "start_time": "2021-03-17T02:59:24.919328Z"
    }
   },
   "outputs": [],
   "source": [
    "#Used smote to oversample minority class\n",
    "smote = SMOTE(random_state=25)\n",
    "X_train_smote, y_train_smote = smote.fit_sample(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KNN "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Here we ran a basline model of knn on the experimental database and compared the metrics results to the database treated with smote.  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Baseline KNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:01:15.536713Z",
     "start_time": "2021-03-17T03:01:15.486928Z"
    }
   },
   "outputs": [],
   "source": [
    "#baseline KNN with class imbalance\n",
    "knn = KNeighborsClassifier()\n",
    "knn.fit(X_train, y_train)\n",
    "y_pred = knn.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:01:16.365593Z",
     "start_time": "2021-03-17T03:01:16.350534Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.9191729323308271\n",
      "Precision: 0.9236111111111112\n",
      "Recall: 0.9755501222493888\n",
      "F1 Score: 0.9488703923900119\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[399  10  33  90]\n"
     ]
    }
   ],
   "source": [
    "#prediction metrics and confusion matrix of base KNN with class imbalance\n",
    "evaluation(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:01:43.351650Z",
     "start_time": "2021-03-17T03:01:43.299615Z"
    }
   },
   "outputs": [],
   "source": [
    "#Base KNN model with Smote oversampled class\n",
    "sm_knn = KNeighborsClassifier()\n",
    "sm_knn.fit(X_train_smote, y_train_smote)\n",
    "y_pred_smote_knn = sm_knn.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:01:44.059581Z",
     "start_time": "2021-03-17T03:01:44.045551Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.9078947368421053\n",
      "Precision: 0.9477611940298507\n",
      "Recall: 0.9315403422982885\n",
      "F1 Score: 0.9395807644882861\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[381  28  21 102]\n"
     ]
    }
   ],
   "source": [
    "#Prediction metrics and confusion matrix of smote base KNN\n",
    "evaluation(y_test, y_pred_smote_knn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SMOTE didn't improve the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Logistic Regression Baseline Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Here are two Logistic regression Models comparing the effect of SMOTE on the metric scores.  SMOTE greatly improved the precision of the model, however Recall was greatly reduced.  While one of the aims for our project was to improve precision as best we can, too many false negatives are not ideal for the overal fetal mortality rate.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:02:24.441601Z",
     "start_time": "2021-03-17T03:02:23.950560Z"
    }
   },
   "outputs": [],
   "source": [
    "#Fit Train set with Logistic Regression model\n",
    "lr = LogisticRegression()\n",
    "lr.fit(X_train,y_train)\n",
    "y_pred = lr.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:02:24.605835Z",
     "start_time": "2021-03-17T03:02:24.590843Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.8778195488721805\n",
      "Precision: 0.9056603773584906\n",
      "Recall: 0.9388753056234719\n",
      "F1 Score: 0.921968787515006\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[384  25  40  83]\n"
     ]
    }
   ],
   "source": [
    "#Regression model evaluation metrics and confusion matrix\n",
    "evaluation(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:02:38.669074Z",
     "start_time": "2021-03-17T03:02:38.613441Z"
    }
   },
   "outputs": [],
   "source": [
    "smlr = LogisticRegression(solver='liblinear')\n",
    "smlr.fit(X_train_smote,y_train_smote)\n",
    "y_predsm = smlr.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:02:39.794128Z",
     "start_time": "2021-03-17T03:02:39.774538Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.8853383458646616\n",
      "Precision: 0.967741935483871\n",
      "Recall: 0.8801955990220048\n",
      "F1 Score: 0.9218950064020486\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[360  49  12 111]\n"
     ]
    }
   ],
   "source": [
    "evaluation(y_test, y_predsm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Below is a table of feature coefficients for the Logistic Regression.  The greatest feature Coefficients are sqrt_total_change, quant_acc_1, quant_hist_mean(1 2 and 3) and tendancy_1.  All of these features are ones that we engineered.  Looking at what these features represent, for this model, the change in the rate of FHR and the average change in that rate seem to be the biggest factors for classifying fetal health.  We wanted to try and improve the Logistic regression by tuning the hyperparameters to improve both the recall and precision.  To attempt this we used a bagging classifer to train the logistic regression on multiple random samples and aggrigate the predictions to see how that effected the evaluation metrics.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:03:03.721573Z",
     "start_time": "2021-03-17T03:03:03.681229Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>features</th>\n",
       "      <td>baseline value</td>\n",
       "      <td>accelerations</td>\n",
       "      <td>fetal_movement</td>\n",
       "      <td>uterine_contractions</td>\n",
       "      <td>light_decelerations</td>\n",
       "      <td>prolongued_decelerations</td>\n",
       "      <td>abnormal_short_term_variability</td>\n",
       "      <td>mean_value_of_short_term_variability</td>\n",
       "      <td>percentage_of_time_with_abnormal_long_term_var...</td>\n",
       "      <td>mean_value_of_long_term_variability</td>\n",
       "      <td>histogram_width</td>\n",
       "      <td>histogram_min</td>\n",
       "      <td>histogram_max</td>\n",
       "      <td>histogram_number_of_peaks</td>\n",
       "      <td>histogram_mode</td>\n",
       "      <td>histogram_mean</td>\n",
       "      <td>histogram_median</td>\n",
       "      <td>histogram_variance</td>\n",
       "      <td>uterine_cont_per_min</td>\n",
       "      <td>total_change</td>\n",
       "      <td>sqrt_total_change</td>\n",
       "      <td>hist_zeros_1.0</td>\n",
       "      <td>hist_zeros_2.0</td>\n",
       "      <td>hist_zeros_3.0</td>\n",
       "      <td>hist_zeros_4.0</td>\n",
       "      <td>hist_zeros_5.0</td>\n",
       "      <td>hist_zeros_7.0</td>\n",
       "      <td>hist_zeros_8.0</td>\n",
       "      <td>hist_zeros_10.0</td>\n",
       "      <td>hist_tendancy_0.0</td>\n",
       "      <td>hist_tendancy_1.0</td>\n",
       "      <td>sev_decel_0.001</td>\n",
       "      <td>quant_acc_1</td>\n",
       "      <td>quant_light_dec_1</td>\n",
       "      <td>quant_hist_mean_1</td>\n",
       "      <td>quant_hist_mean_2</td>\n",
       "      <td>quant_hist_mean_3</td>\n",
       "      <td>quant_hist_mean_4</td>\n",
       "      <td>quant_hist_mean_5</td>\n",
       "      <td>quant_hist_mean_6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Coefs</th>\n",
       "      <td>0.117174</td>\n",
       "      <td>-0.0040917</td>\n",
       "      <td>0.0108432</td>\n",
       "      <td>-0.0053203</td>\n",
       "      <td>-0.00221984</td>\n",
       "      <td>0.000806171</td>\n",
       "      <td>0.0699097</td>\n",
       "      <td>-0.715013</td>\n",
       "      <td>0.0372611</td>\n",
       "      <td>0.0319318</td>\n",
       "      <td>-0.0144669</td>\n",
       "      <td>0.00639082</td>\n",
       "      <td>-0.00807605</td>\n",
       "      <td>0.251932</td>\n",
       "      <td>-0.0210527</td>\n",
       "      <td>0.0316949</td>\n",
       "      <td>-0.157476</td>\n",
       "      <td>0.032656</td>\n",
       "      <td>-0.319218</td>\n",
       "      <td>0.0406248</td>\n",
       "      <td>-0.0494468</td>\n",
       "      <td>0.0646365</td>\n",
       "      <td>0.0201561</td>\n",
       "      <td>-0.00272192</td>\n",
       "      <td>0.0176237</td>\n",
       "      <td>-0.00225233</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0110208</td>\n",
       "      <td>-0.000777236</td>\n",
       "      <td>-0.0765569</td>\n",
       "      <td>0.0919297</td>\n",
       "      <td>-0.00379458</td>\n",
       "      <td>-0.235999</td>\n",
       "      <td>-0.34457</td>\n",
       "      <td>-0.467502</td>\n",
       "      <td>-0.255169</td>\n",
       "      <td>-0.356305</td>\n",
       "      <td>0.0562109</td>\n",
       "      <td>0.42863</td>\n",
       "      <td>0.598199</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sm_Coefs</th>\n",
       "      <td>0.112113</td>\n",
       "      <td>-0.224252</td>\n",
       "      <td>0.313262</td>\n",
       "      <td>-0.055485</td>\n",
       "      <td>-0.0363894</td>\n",
       "      <td>0.0445865</td>\n",
       "      <td>0.08435</td>\n",
       "      <td>-0.626913</td>\n",
       "      <td>0.0396073</td>\n",
       "      <td>0.0459468</td>\n",
       "      <td>-0.00961589</td>\n",
       "      <td>-0.0010398</td>\n",
       "      <td>-0.0106557</td>\n",
       "      <td>0.200837</td>\n",
       "      <td>-0.00977889</td>\n",
       "      <td>-0.0597073</td>\n",
       "      <td>-0.0313393</td>\n",
       "      <td>0.0454627</td>\n",
       "      <td>-3.3291</td>\n",
       "      <td>0.0622807</td>\n",
       "      <td>-1.24574</td>\n",
       "      <td>-0.514483</td>\n",
       "      <td>-0.583633</td>\n",
       "      <td>-1.0854</td>\n",
       "      <td>0.384853</td>\n",
       "      <td>-0.626686</td>\n",
       "      <td>0</td>\n",
       "      <td>0.195844</td>\n",
       "      <td>-0.629847</td>\n",
       "      <td>-0.752577</td>\n",
       "      <td>-1.25432</td>\n",
       "      <td>-0.238835</td>\n",
       "      <td>-3.31225</td>\n",
       "      <td>-1.66982</td>\n",
       "      <td>-2.73523</td>\n",
       "      <td>-2.34835</td>\n",
       "      <td>-2.62443</td>\n",
       "      <td>-1.19418</td>\n",
       "      <td>-0.1252</td>\n",
       "      <td>0.816205</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       0              1               2                     3  \\\n",
       "features  baseline value  accelerations  fetal_movement  uterine_contractions   \n",
       "Coefs           0.117174     -0.0040917       0.0108432            -0.0053203   \n",
       "sm_Coefs        0.112113      -0.224252        0.313262             -0.055485   \n",
       "\n",
       "                            4                         5  \\\n",
       "features  light_decelerations  prolongued_decelerations   \n",
       "Coefs             -0.00221984               0.000806171   \n",
       "sm_Coefs           -0.0363894                 0.0445865   \n",
       "\n",
       "                                        6  \\\n",
       "features  abnormal_short_term_variability   \n",
       "Coefs                           0.0699097   \n",
       "sm_Coefs                          0.08435   \n",
       "\n",
       "                                             7  \\\n",
       "features  mean_value_of_short_term_variability   \n",
       "Coefs                                -0.715013   \n",
       "sm_Coefs                             -0.626913   \n",
       "\n",
       "                                                          8  \\\n",
       "features  percentage_of_time_with_abnormal_long_term_var...   \n",
       "Coefs                                             0.0372611   \n",
       "sm_Coefs                                          0.0396073   \n",
       "\n",
       "                                            9               10             11  \\\n",
       "features  mean_value_of_long_term_variability  histogram_width  histogram_min   \n",
       "Coefs                               0.0319318       -0.0144669     0.00639082   \n",
       "sm_Coefs                            0.0459468      -0.00961589     -0.0010398   \n",
       "\n",
       "                     12                         13              14  \\\n",
       "features  histogram_max  histogram_number_of_peaks  histogram_mode   \n",
       "Coefs       -0.00807605                   0.251932      -0.0210527   \n",
       "sm_Coefs     -0.0106557                   0.200837     -0.00977889   \n",
       "\n",
       "                      15                16                  17  \\\n",
       "features  histogram_mean  histogram_median  histogram_variance   \n",
       "Coefs          0.0316949         -0.157476            0.032656   \n",
       "sm_Coefs      -0.0597073        -0.0313393           0.0454627   \n",
       "\n",
       "                            18            19                 20  \\\n",
       "features  uterine_cont_per_min  total_change  sqrt_total_change   \n",
       "Coefs                -0.319218     0.0406248         -0.0494468   \n",
       "sm_Coefs               -3.3291     0.0622807           -1.24574   \n",
       "\n",
       "                      21              22              23              24  \\\n",
       "features  hist_zeros_1.0  hist_zeros_2.0  hist_zeros_3.0  hist_zeros_4.0   \n",
       "Coefs          0.0646365       0.0201561     -0.00272192       0.0176237   \n",
       "sm_Coefs       -0.514483       -0.583633         -1.0854        0.384853   \n",
       "\n",
       "                      25              26              27               28  \\\n",
       "features  hist_zeros_5.0  hist_zeros_7.0  hist_zeros_8.0  hist_zeros_10.0   \n",
       "Coefs        -0.00225233               0       0.0110208     -0.000777236   \n",
       "sm_Coefs       -0.626686               0        0.195844        -0.629847   \n",
       "\n",
       "                         29                 30               31           32  \\\n",
       "features  hist_tendancy_0.0  hist_tendancy_1.0  sev_decel_0.001  quant_acc_1   \n",
       "Coefs            -0.0765569          0.0919297      -0.00379458    -0.235999   \n",
       "sm_Coefs          -0.752577           -1.25432        -0.238835     -3.31225   \n",
       "\n",
       "                         33                 34                 35  \\\n",
       "features  quant_light_dec_1  quant_hist_mean_1  quant_hist_mean_2   \n",
       "Coefs              -0.34457          -0.467502          -0.255169   \n",
       "sm_Coefs           -1.66982           -2.73523           -2.34835   \n",
       "\n",
       "                         36                 37                 38  \\\n",
       "features  quant_hist_mean_3  quant_hist_mean_4  quant_hist_mean_5   \n",
       "Coefs             -0.356305          0.0562109            0.42863   \n",
       "sm_Coefs           -2.62443           -1.19418            -0.1252   \n",
       "\n",
       "                         39  \n",
       "features  quant_hist_mean_6  \n",
       "Coefs              0.598199  \n",
       "sm_Coefs           0.816205  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Create a table of logistic regression coefficients and comparing the coefficients \n",
    "#of the SMOTE and inbalanced datasets\n",
    "coef_table = pd.DataFrame(list(X_train.columns)).copy()\n",
    "coef_table.insert(len(coef_table.columns),\"Coefs\",lr.coef_.transpose())\n",
    "\n",
    "coef_table_2 = pd.DataFrame(list(X_train_smote.columns)).copy()\n",
    "coef_table_2.insert(len(coef_table_2.columns),'sm_Coefs',smlr.coef_.transpose())\n",
    "\n",
    "smote_vs_coef = pd.concat([coef_table,coef_table_2],axis=1)\n",
    "smote_vs_coef.columns = ['features','Coefs','del','sm_Coefs']\n",
    "del smote_vs_coef['del']\n",
    "smote_vs_coef.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:03:11.235250Z",
     "start_time": "2021-03-17T03:03:11.231204Z"
    }
   },
   "outputs": [],
   "source": [
    "# Bagging classifier for legistic regression\n",
    "bag_log = BaggingClassifier(\n",
    "    base_estimator=LogisticRegression(\n",
    "        random_state=1),n_estimators=200,\n",
    "    max_samples=.85,\n",
    "    max_features=10,oob_score=True,\n",
    "    n_jobs=-1,verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:03:25.863885Z",
     "start_time": "2021-03-17T03:03:21.040168Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=8)]: Done   2 out of   8 | elapsed:    4.2s remaining:   12.6s\n",
      "[Parallel(n_jobs=8)]: Done   8 out of   8 | elapsed:    4.2s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.8759398496240601\n",
      "Precision: 0.9549071618037135\n",
      "Recall: 0.8801955990220048\n",
      "F1 Score: 0.9160305343511449\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[360  49  17 106]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=8)]: Done   2 out of   8 | elapsed:    0.0s remaining:    0.1s\n",
      "[Parallel(n_jobs=8)]: Done   8 out of   8 | elapsed:    0.0s finished\n"
     ]
    }
   ],
   "source": [
    "# Fit bagging classifier\n",
    "bag_log.fit(X_train_smote, y_train_smote)\n",
    "y_pred = bag_log.predict(X_test)\n",
    "\n",
    "# Evaluation Metrics\n",
    "evaluation(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The Bagging classifer made only minute changes to the evaluation metrics.  We decided to look at some other models and compare the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decision Tree"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The next model we tried was a decision tree.  Again we compared the imbalanced and SMOTE data sets on baseline models to observe the effect on the metrics.  With the SMOTE data set, decision tree produced very good metrics with all default hyperparamers.  The most important features where abnormal_short_term_variability, and mean_value_of_short_term_variability\t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:03:31.318304Z",
     "start_time": "2021-03-17T03:03:31.295289Z"
    }
   },
   "outputs": [],
   "source": [
    "#Train decision tree with train set and predict on the test set\n",
    "tree = DecisionTreeClassifier()\n",
    "tree = tree.fit(X_train,y_train)\n",
    "y_pred = tree.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:03:38.705240Z",
     "start_time": "2021-03-17T03:03:38.689538Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.9229323308270677\n",
      "Precision: 0.9577114427860697\n",
      "Recall: 0.941320293398533\n",
      "F1 Score: 0.9494451294697904\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[385  24  17 106]\n"
     ]
    }
   ],
   "source": [
    "#Evaluation metrics\n",
    "evaluation(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:03:48.702826Z",
     "start_time": "2021-03-17T03:03:48.665307Z"
    }
   },
   "outputs": [],
   "source": [
    "#Decision tree with smote dataset\n",
    "smtree = DecisionTreeClassifier()\n",
    "smtree.fit(X_train_smote,y_train_smote)\n",
    "y_pred = smtree.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:03:50.833264Z",
     "start_time": "2021-03-17T03:03:50.805511Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.9342105263157895\n",
      "Precision: 0.9583333333333334\n",
      "Recall: 0.9559902200488998\n",
      "F1 Score: 0.9571603427172583\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[391  18  17 106]\n"
     ]
    }
   ],
   "source": [
    "#evaluation Metrics\n",
    "evaluation(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:04:02.054486Z",
     "start_time": "2021-03-17T03:04:02.011941Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>features</th>\n",
       "      <td>baseline value</td>\n",
       "      <td>accelerations</td>\n",
       "      <td>fetal_movement</td>\n",
       "      <td>uterine_contractions</td>\n",
       "      <td>light_decelerations</td>\n",
       "      <td>prolongued_decelerations</td>\n",
       "      <td>abnormal_short_term_variability</td>\n",
       "      <td>mean_value_of_short_term_variability</td>\n",
       "      <td>percentage_of_time_with_abnormal_long_term_var...</td>\n",
       "      <td>mean_value_of_long_term_variability</td>\n",
       "      <td>histogram_width</td>\n",
       "      <td>histogram_min</td>\n",
       "      <td>histogram_max</td>\n",
       "      <td>histogram_number_of_peaks</td>\n",
       "      <td>histogram_mode</td>\n",
       "      <td>histogram_mean</td>\n",
       "      <td>histogram_median</td>\n",
       "      <td>histogram_variance</td>\n",
       "      <td>uterine_cont_per_min</td>\n",
       "      <td>total_change</td>\n",
       "      <td>sqrt_total_change</td>\n",
       "      <td>hist_zeros_1.0</td>\n",
       "      <td>hist_zeros_2.0</td>\n",
       "      <td>hist_zeros_3.0</td>\n",
       "      <td>hist_zeros_4.0</td>\n",
       "      <td>hist_zeros_5.0</td>\n",
       "      <td>hist_zeros_7.0</td>\n",
       "      <td>hist_zeros_8.0</td>\n",
       "      <td>hist_zeros_10.0</td>\n",
       "      <td>hist_tendancy_0.0</td>\n",
       "      <td>hist_tendancy_1.0</td>\n",
       "      <td>sev_decel_0.001</td>\n",
       "      <td>quant_acc_1</td>\n",
       "      <td>quant_light_dec_1</td>\n",
       "      <td>quant_hist_mean_1</td>\n",
       "      <td>quant_hist_mean_2</td>\n",
       "      <td>quant_hist_mean_3</td>\n",
       "      <td>quant_hist_mean_4</td>\n",
       "      <td>quant_hist_mean_5</td>\n",
       "      <td>quant_hist_mean_6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>importance</th>\n",
       "      <td>0.0417007</td>\n",
       "      <td>0.0196927</td>\n",
       "      <td>0.0129526</td>\n",
       "      <td>0.00915159</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0285847</td>\n",
       "      <td>0.342457</td>\n",
       "      <td>0.117288</td>\n",
       "      <td>0.0467433</td>\n",
       "      <td>0.0120976</td>\n",
       "      <td>0.0193758</td>\n",
       "      <td>0.0326662</td>\n",
       "      <td>0.0351468</td>\n",
       "      <td>0.0132994</td>\n",
       "      <td>0.0228727</td>\n",
       "      <td>0.168744</td>\n",
       "      <td>0.00889406</td>\n",
       "      <td>0.00628032</td>\n",
       "      <td>0.00526129</td>\n",
       "      <td>0.0384527</td>\n",
       "      <td>0.0126355</td>\n",
       "      <td>0.000779075</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00361421</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00130948</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sm_importance</th>\n",
       "      <td>0.0417007</td>\n",
       "      <td>0.0196927</td>\n",
       "      <td>0.0129526</td>\n",
       "      <td>0.00915159</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0285847</td>\n",
       "      <td>0.342457</td>\n",
       "      <td>0.117288</td>\n",
       "      <td>0.0467433</td>\n",
       "      <td>0.0120976</td>\n",
       "      <td>0.0193758</td>\n",
       "      <td>0.0326662</td>\n",
       "      <td>0.0351468</td>\n",
       "      <td>0.0132994</td>\n",
       "      <td>0.0228727</td>\n",
       "      <td>0.168744</td>\n",
       "      <td>0.00889406</td>\n",
       "      <td>0.00628032</td>\n",
       "      <td>0.00526129</td>\n",
       "      <td>0.0384527</td>\n",
       "      <td>0.0126355</td>\n",
       "      <td>0.000779075</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00361421</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00130948</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                            0              1               2  \\\n",
       "features       baseline value  accelerations  fetal_movement   \n",
       "importance          0.0417007      0.0196927       0.0129526   \n",
       "sm_importance       0.0417007      0.0196927       0.0129526   \n",
       "\n",
       "                                  3                    4  \\\n",
       "features       uterine_contractions  light_decelerations   \n",
       "importance               0.00915159                    0   \n",
       "sm_importance            0.00915159                    0   \n",
       "\n",
       "                                      5                                6  \\\n",
       "features       prolongued_decelerations  abnormal_short_term_variability   \n",
       "importance                    0.0285847                         0.342457   \n",
       "sm_importance                 0.0285847                         0.342457   \n",
       "\n",
       "                                                  7  \\\n",
       "features       mean_value_of_short_term_variability   \n",
       "importance                                 0.117288   \n",
       "sm_importance                              0.117288   \n",
       "\n",
       "                                                               8  \\\n",
       "features       percentage_of_time_with_abnormal_long_term_var...   \n",
       "importance                                             0.0467433   \n",
       "sm_importance                                          0.0467433   \n",
       "\n",
       "                                                 9               10  \\\n",
       "features       mean_value_of_long_term_variability  histogram_width   \n",
       "importance                               0.0120976        0.0193758   \n",
       "sm_importance                            0.0120976        0.0193758   \n",
       "\n",
       "                          11             12                         13  \\\n",
       "features       histogram_min  histogram_max  histogram_number_of_peaks   \n",
       "importance         0.0326662      0.0351468                  0.0132994   \n",
       "sm_importance      0.0326662      0.0351468                  0.0132994   \n",
       "\n",
       "                           14              15                16  \\\n",
       "features       histogram_mode  histogram_mean  histogram_median   \n",
       "importance          0.0228727        0.168744        0.00889406   \n",
       "sm_importance       0.0228727        0.168744        0.00889406   \n",
       "\n",
       "                               17                    18            19  \\\n",
       "features       histogram_variance  uterine_cont_per_min  total_change   \n",
       "importance             0.00628032            0.00526129     0.0384527   \n",
       "sm_importance          0.00628032            0.00526129     0.0384527   \n",
       "\n",
       "                              20              21              22  \\\n",
       "features       sqrt_total_change  hist_zeros_1.0  hist_zeros_2.0   \n",
       "importance             0.0126355     0.000779075               0   \n",
       "sm_importance          0.0126355     0.000779075               0   \n",
       "\n",
       "                           23              24              25              26  \\\n",
       "features       hist_zeros_3.0  hist_zeros_4.0  hist_zeros_5.0  hist_zeros_7.0   \n",
       "importance                  0      0.00361421               0               0   \n",
       "sm_importance               0      0.00361421               0               0   \n",
       "\n",
       "                           27               28                 29  \\\n",
       "features       hist_zeros_8.0  hist_zeros_10.0  hist_tendancy_0.0   \n",
       "importance                  0                0                  0   \n",
       "sm_importance               0                0                  0   \n",
       "\n",
       "                              30               31           32  \\\n",
       "features       hist_tendancy_1.0  sev_decel_0.001  quant_acc_1   \n",
       "importance            0.00130948                0            0   \n",
       "sm_importance         0.00130948                0            0   \n",
       "\n",
       "                              33                 34                 35  \\\n",
       "features       quant_light_dec_1  quant_hist_mean_1  quant_hist_mean_2   \n",
       "importance                     0                  0                  0   \n",
       "sm_importance                  0                  0                  0   \n",
       "\n",
       "                              36                 37                 38  \\\n",
       "features       quant_hist_mean_3  quant_hist_mean_4  quant_hist_mean_5   \n",
       "importance                     0                  0                  0   \n",
       "sm_importance                  0                  0                  0   \n",
       "\n",
       "                              39  \n",
       "features       quant_hist_mean_6  \n",
       "importance                     0  \n",
       "sm_importance                  0  "
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Table for decision tree feature coefficients\n",
    "coef_table = pd.DataFrame(list(X_train.columns)).copy()\n",
    "coef_table.insert(len(coef_table.columns),\"Coefs\",tree.feature_importances_.transpose())\n",
    "\n",
    "coef_table_2 = pd.DataFrame(list(X_train_smote.columns)).copy()\n",
    "coef_table_2.insert(len(coef_table_2.columns),'sm_Coefs',tree.feature_importances_.transpose())\n",
    "\n",
    "smote_vs_coef = pd.concat([coef_table,coef_table_2],axis=1)\n",
    "smote_vs_coef.columns = ['features','importance','del','sm_importance']\n",
    "del smote_vs_coef['del']\n",
    "smote_vs_coef.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- As we did above we used the SMOTE and imbalanced data and this time trained a random forest classifer.  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:04:05.905695Z",
     "start_time": "2021-03-17T03:04:05.658981Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done  50 out of  50 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=8)]: Done  34 tasks      | elapsed:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  50 out of  50 | elapsed:    0.0s finished\n"
     ]
    }
   ],
   "source": [
    "#Random Forest classifier using 50 estimators and a max depth of 3\n",
    "rfc = RandomForestClassifier(random_state =1, n_estimators= 50, max_depth = 3, n_jobs =-1,verbose=1)\n",
    "rfc.fit(X_train,y_train)\n",
    "y_pred = rfc.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:04:07.540373Z",
     "start_time": "2021-03-17T03:04:07.525165Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.900375939849624\n",
      "Precision: 0.8973214285714286\n",
      "Recall: 0.9828850855745721\n",
      "F1 Score: 0.9381563593932323\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[402   7  46  77]\n"
     ]
    }
   ],
   "source": [
    "#Evaluation metrics for random forest\n",
    "evaluation(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:04:33.575024Z",
     "start_time": "2021-03-17T03:04:33.311748Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.9116541353383458\n",
      "Precision: 0.9547738693467337\n",
      "Recall: 0.9290953545232273\n",
      "F1 Score: 0.9417596034696406\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[380  29  18 105]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done  50 out of  50 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=8)]: Done  34 tasks      | elapsed:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  50 out of  50 | elapsed:    0.0s finished\n"
     ]
    }
   ],
   "source": [
    "#Random Forest classifier using 50 estimators and a max depth of 3 using SMOTE dataset\n",
    "smrfc = RandomForestClassifier(random_state =1, n_estimators= 50, max_depth = 3, n_jobs =-1,verbose=1)\n",
    "smrfc.fit(X_train_smote,y_train_smote)\n",
    "y_pred = smrfc.predict(X_test)\n",
    "\n",
    "evaluation(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:04:45.920185Z",
     "start_time": "2021-03-17T03:04:45.665380Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>features</th>\n",
       "      <td>baseline value</td>\n",
       "      <td>accelerations</td>\n",
       "      <td>fetal_movement</td>\n",
       "      <td>uterine_contractions</td>\n",
       "      <td>light_decelerations</td>\n",
       "      <td>prolongued_decelerations</td>\n",
       "      <td>abnormal_short_term_variability</td>\n",
       "      <td>mean_value_of_short_term_variability</td>\n",
       "      <td>percentage_of_time_with_abnormal_long_term_var...</td>\n",
       "      <td>mean_value_of_long_term_variability</td>\n",
       "      <td>histogram_width</td>\n",
       "      <td>histogram_min</td>\n",
       "      <td>histogram_max</td>\n",
       "      <td>histogram_number_of_peaks</td>\n",
       "      <td>histogram_mode</td>\n",
       "      <td>histogram_mean</td>\n",
       "      <td>histogram_median</td>\n",
       "      <td>histogram_variance</td>\n",
       "      <td>uterine_cont_per_min</td>\n",
       "      <td>total_change</td>\n",
       "      <td>sqrt_total_change</td>\n",
       "      <td>hist_zeros_1.0</td>\n",
       "      <td>hist_zeros_2.0</td>\n",
       "      <td>hist_zeros_3.0</td>\n",
       "      <td>hist_zeros_4.0</td>\n",
       "      <td>hist_zeros_5.0</td>\n",
       "      <td>hist_zeros_7.0</td>\n",
       "      <td>hist_zeros_8.0</td>\n",
       "      <td>hist_zeros_10.0</td>\n",
       "      <td>hist_tendancy_0.0</td>\n",
       "      <td>hist_tendancy_1.0</td>\n",
       "      <td>sev_decel_0.001</td>\n",
       "      <td>quant_acc_1</td>\n",
       "      <td>quant_light_dec_1</td>\n",
       "      <td>quant_hist_mean_1</td>\n",
       "      <td>quant_hist_mean_2</td>\n",
       "      <td>quant_hist_mean_3</td>\n",
       "      <td>quant_hist_mean_4</td>\n",
       "      <td>quant_hist_mean_5</td>\n",
       "      <td>quant_hist_mean_6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>importance_baseline</th>\n",
       "      <td>0.00992676</td>\n",
       "      <td>0.103532</td>\n",
       "      <td>0.00470882</td>\n",
       "      <td>0.0105892</td>\n",
       "      <td>0.00061708</td>\n",
       "      <td>0.0876404</td>\n",
       "      <td>0.157685</td>\n",
       "      <td>0.149226</td>\n",
       "      <td>0.0898387</td>\n",
       "      <td>0.0122778</td>\n",
       "      <td>0.0241374</td>\n",
       "      <td>0.0345421</td>\n",
       "      <td>0.00347558</td>\n",
       "      <td>0.00129207</td>\n",
       "      <td>0.0299309</td>\n",
       "      <td>0.100239</td>\n",
       "      <td>0.0523094</td>\n",
       "      <td>0.0412711</td>\n",
       "      <td>0.0316257</td>\n",
       "      <td>0.0267606</td>\n",
       "      <td>0.0168846</td>\n",
       "      <td>0.000475167</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000412919</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0050735</td>\n",
       "      <td>0.00202451</td>\n",
       "      <td>6.36107e-07</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000271607</td>\n",
       "      <td>0.000183496</td>\n",
       "      <td>0.000962509</td>\n",
       "      <td>0.00208523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sm_importance_baseline</th>\n",
       "      <td>0.0172741</td>\n",
       "      <td>0.20107</td>\n",
       "      <td>0.00249253</td>\n",
       "      <td>0.0203137</td>\n",
       "      <td>0.00140923</td>\n",
       "      <td>0.0858243</td>\n",
       "      <td>0.173725</td>\n",
       "      <td>0.104371</td>\n",
       "      <td>0.0807028</td>\n",
       "      <td>0.00967133</td>\n",
       "      <td>0.0139368</td>\n",
       "      <td>0.0450373</td>\n",
       "      <td>0.007946</td>\n",
       "      <td>0.000497999</td>\n",
       "      <td>0.0139928</td>\n",
       "      <td>0.0629823</td>\n",
       "      <td>0.0220239</td>\n",
       "      <td>0.0390045</td>\n",
       "      <td>0.0415174</td>\n",
       "      <td>0.0158197</td>\n",
       "      <td>0.00860188</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0241652</td>\n",
       "      <td>0.000976957</td>\n",
       "      <td>0.000289048</td>\n",
       "      <td>0.000687007</td>\n",
       "      <td>0.00428605</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000333931</td>\n",
       "      <td>0.00104688</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     0              1               2  \\\n",
       "features                baseline value  accelerations  fetal_movement   \n",
       "importance_baseline         0.00992676       0.103532      0.00470882   \n",
       "sm_importance_baseline       0.0172741        0.20107      0.00249253   \n",
       "\n",
       "                                           3                    4  \\\n",
       "features                uterine_contractions  light_decelerations   \n",
       "importance_baseline                0.0105892           0.00061708   \n",
       "sm_importance_baseline             0.0203137           0.00140923   \n",
       "\n",
       "                                               5  \\\n",
       "features                prolongued_decelerations   \n",
       "importance_baseline                    0.0876404   \n",
       "sm_importance_baseline                 0.0858243   \n",
       "\n",
       "                                                      6  \\\n",
       "features                abnormal_short_term_variability   \n",
       "importance_baseline                            0.157685   \n",
       "sm_importance_baseline                         0.173725   \n",
       "\n",
       "                                                           7  \\\n",
       "features                mean_value_of_short_term_variability   \n",
       "importance_baseline                                 0.149226   \n",
       "sm_importance_baseline                              0.104371   \n",
       "\n",
       "                                                                        8  \\\n",
       "features                percentage_of_time_with_abnormal_long_term_var...   \n",
       "importance_baseline                                             0.0898387   \n",
       "sm_importance_baseline                                          0.0807028   \n",
       "\n",
       "                                                          9               10  \\\n",
       "features                mean_value_of_long_term_variability  histogram_width   \n",
       "importance_baseline                               0.0122778        0.0241374   \n",
       "sm_importance_baseline                           0.00967133        0.0139368   \n",
       "\n",
       "                                   11             12  \\\n",
       "features                histogram_min  histogram_max   \n",
       "importance_baseline         0.0345421     0.00347558   \n",
       "sm_importance_baseline      0.0450373       0.007946   \n",
       "\n",
       "                                               13              14  \\\n",
       "features                histogram_number_of_peaks  histogram_mode   \n",
       "importance_baseline                    0.00129207       0.0299309   \n",
       "sm_importance_baseline                0.000497999       0.0139928   \n",
       "\n",
       "                                    15                16                  17  \\\n",
       "features                histogram_mean  histogram_median  histogram_variance   \n",
       "importance_baseline           0.100239         0.0523094           0.0412711   \n",
       "sm_importance_baseline       0.0629823         0.0220239           0.0390045   \n",
       "\n",
       "                                          18            19                 20  \\\n",
       "features                uterine_cont_per_min  total_change  sqrt_total_change   \n",
       "importance_baseline                0.0316257     0.0267606          0.0168846   \n",
       "sm_importance_baseline             0.0415174     0.0158197         0.00860188   \n",
       "\n",
       "                                    21              22              23  \\\n",
       "features                hist_zeros_1.0  hist_zeros_2.0  hist_zeros_3.0   \n",
       "importance_baseline        0.000475167               0               0   \n",
       "sm_importance_baseline               0               0               0   \n",
       "\n",
       "                                    24              25              26  \\\n",
       "features                hist_zeros_4.0  hist_zeros_5.0  hist_zeros_7.0   \n",
       "importance_baseline                  0               0               0   \n",
       "sm_importance_baseline               0               0               0   \n",
       "\n",
       "                                    27               28                 29  \\\n",
       "features                hist_zeros_8.0  hist_zeros_10.0  hist_tendancy_0.0   \n",
       "importance_baseline                  0                0                  0   \n",
       "sm_importance_baseline               0                0                  0   \n",
       "\n",
       "                                       30               31           32  \\\n",
       "features                hist_tendancy_1.0  sev_decel_0.001  quant_acc_1   \n",
       "importance_baseline           0.000412919                0    0.0050735   \n",
       "sm_importance_baseline                  0                0    0.0241652   \n",
       "\n",
       "                                       33                 34  \\\n",
       "features                quant_light_dec_1  quant_hist_mean_1   \n",
       "importance_baseline            0.00202451        6.36107e-07   \n",
       "sm_importance_baseline        0.000976957        0.000289048   \n",
       "\n",
       "                                       35                 36  \\\n",
       "features                quant_hist_mean_2  quant_hist_mean_3   \n",
       "importance_baseline                     0        0.000271607   \n",
       "sm_importance_baseline        0.000687007         0.00428605   \n",
       "\n",
       "                                       37                 38  \\\n",
       "features                quant_hist_mean_4  quant_hist_mean_5   \n",
       "importance_baseline           0.000183496        0.000962509   \n",
       "sm_importance_baseline                  0        0.000333931   \n",
       "\n",
       "                                       39  \n",
       "features                quant_hist_mean_6  \n",
       "importance_baseline            0.00208523  \n",
       "sm_importance_baseline         0.00104688  "
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Create table of feature coefficients\n",
    "coef_table = pd.DataFrame(list(X_train.columns)).copy()\n",
    "coef_table.insert(len(coef_table.columns),\"Coefs\",rfc.feature_importances_.transpose())\n",
    "\n",
    "coef_table_2 = pd.DataFrame(list(X_train_smote.columns)).copy()\n",
    "coef_table_2.insert(len(coef_table_2.columns),'sm_Coefs',smrfc.feature_importances_.transpose())\n",
    "\n",
    "smote_vs_coef = pd.concat([coef_table,coef_table_2],axis=1)\n",
    "smote_vs_coef.columns = ['features','importance_baseline','del','sm_importance_baseline']\n",
    "del smote_vs_coef['del']\n",
    "smote_vs_coef.transpose()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grid Search Random Forest\n",
    "- We ran a gridsearch on the random forest to identify what the best hyperparamters where for the model.  We checked several estimator sizes, max depth and min weight fraction leaf to find the ideal parameters.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:04:49.933375Z",
     "start_time": "2021-03-17T03:04:49.929136Z"
    }
   },
   "outputs": [],
   "source": [
    "#parameter grid for grid search with lists of estimators, both criterion, a list of max depths sqrt max features \n",
    "#and a list of min weight fraction leaf\n",
    "parameters = {\n",
    "    'n_estimators': [25,50,100,300,500],\n",
    "    'criterion' : ['gini','entropy'],\n",
    "    'max_depth' : [8,9,10,11,12],\n",
    "    'max_features' : ['sqrt'],\n",
    "    'min_weight_fraction_leaf' : [0,0.1,0.3,0.5],\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:04:51.197265Z",
     "start_time": "2021-03-17T03:04:51.194646Z"
    }
   },
   "outputs": [],
   "source": [
    "#Gridsearch with random forest\n",
    "grid_tree=GridSearchCV(RandomForestClassifier(), parameters, cv=15, scoring='f1', verbose=1, n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.242912Z",
     "start_time": "2021-03-17T03:05:16.231283Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 15 folds for each of 200 candidates, totalling 3000 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:    2.0s\n",
      "[Parallel(n_jobs=-1)]: Done 184 tasks      | elapsed:   25.2s\n",
      "[Parallel(n_jobs=-1)]: Done 434 tasks      | elapsed:  1.2min\n",
      "[Parallel(n_jobs=-1)]: Done 784 tasks      | elapsed:  2.2min\n",
      "[Parallel(n_jobs=-1)]: Done 1234 tasks      | elapsed:  3.2min\n",
      "[Parallel(n_jobs=-1)]: Done 1784 tasks      | elapsed:  4.8min\n",
      "[Parallel(n_jobs=-1)]: Done 2434 tasks      | elapsed:  6.8min\n",
      "[Parallel(n_jobs=-1)]: Done 3000 out of 3000 | elapsed:  8.6min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=15, error_score=nan,\n",
       "             estimator=RandomForestClassifier(bootstrap=True, ccp_alpha=0.0,\n",
       "                                              class_weight=None,\n",
       "                                              criterion='gini', max_depth=None,\n",
       "                                              max_features='auto',\n",
       "                                              max_leaf_nodes=None,\n",
       "                                              max_samples=None,\n",
       "                                              min_impurity_decrease=0.0,\n",
       "                                              min_impurity_split=None,\n",
       "                                              min_samples_leaf=1,\n",
       "                                              min_samples_split=2,\n",
       "                                              min_weight_fraction_leaf=0.0,\n",
       "                                              n_estimators=100, n_jobs=None,\n",
       "                                              oob_score=False,\n",
       "                                              random_state=None, verbose=0,\n",
       "                                              warm_start=False),\n",
       "             iid='deprecated', n_jobs=-1,\n",
       "             param_grid={'criterion': ['gini', 'entropy'],\n",
       "                         'max_depth': [8, 9, 10, 11, 12],\n",
       "                         'max_features': ['sqrt'],\n",
       "                         'min_weight_fraction_leaf': [0, 0.1, 0.3, 0.5],\n",
       "                         'n_estimators': [25, 50, 100, 300, 500]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring='f1', verbose=1)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Fit random forest grid serch to SMOTE train set\n",
    "grid_tree.fit(X_train_smote, y_train_smote)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.255301Z",
     "start_time": "2021-03-17T03:13:54.246635Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9753981661884176\n",
      "\n",
      "{'criterion': 'gini', 'max_depth': 12, 'max_features': 'sqrt', 'min_weight_fraction_leaf': 0, 'n_estimators': 50}\n",
      "\n",
      "RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,\n",
      "                       criterion='gini', max_depth=12, max_features='sqrt',\n",
      "                       max_leaf_nodes=None, max_samples=None,\n",
      "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
      "                       min_samples_leaf=1, min_samples_split=2,\n",
      "                       min_weight_fraction_leaf=0, n_estimators=50, n_jobs=None,\n",
      "                       oob_score=False, random_state=None, verbose=0,\n",
      "                       warm_start=False)\n"
     ]
    }
   ],
   "source": [
    "# Find the best score, best parameters and best estimator for grid search\n",
    "print(grid_tree.best_score_)\n",
    "print()\n",
    "print(grid_tree.best_params_)\n",
    "print()\n",
    "print(grid_tree.best_estimator_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.296160Z",
     "start_time": "2021-03-17T03:13:54.260266Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Evaluation Metrics:\n",
      "Accuracy: 0.9586466165413534\n",
      "Precision: 0.9662650602409638\n",
      "Recall: 0.980440097799511\n",
      "F1 Score: 0.9733009708737864\n",
      "\n",
      "Confusion Matrix:\n",
      " TN,  FP, FN, TP\n",
      "[401   8  14 109]\n"
     ]
    }
   ],
   "source": [
    "# Evaluation metrics for random forest gridsearch best parameters\n",
    "y_pred = grid_tree.best_estimator_.predict(X_test)\n",
    "evaluation(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Findings:** \n",
    "- The best parameters for the random forest was a max depth of 11, min weight fraction leave of 0, and 500 estimators.  The Evaluation metrics where our highest score with the experimental set so far.  Below we compared feature importance for the three random forest models we ran and then graphed the top ten important features to visualize there importance for fetal health classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.365807Z",
     "start_time": "2021-03-17T03:13:54.299918Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>baseline value</td>\n",
       "      <td>accelerations</td>\n",
       "      <td>fetal_movement</td>\n",
       "      <td>uterine_contractions</td>\n",
       "      <td>light_decelerations</td>\n",
       "      <td>prolongued_decelerations</td>\n",
       "      <td>abnormal_short_term_variability</td>\n",
       "      <td>mean_value_of_short_term_variability</td>\n",
       "      <td>percentage_of_time_with_abnormal_long_term_var...</td>\n",
       "      <td>mean_value_of_long_term_variability</td>\n",
       "      <td>histogram_width</td>\n",
       "      <td>histogram_min</td>\n",
       "      <td>histogram_max</td>\n",
       "      <td>histogram_number_of_peaks</td>\n",
       "      <td>histogram_mode</td>\n",
       "      <td>histogram_mean</td>\n",
       "      <td>histogram_median</td>\n",
       "      <td>histogram_variance</td>\n",
       "      <td>uterine_cont_per_min</td>\n",
       "      <td>total_change</td>\n",
       "      <td>sqrt_total_change</td>\n",
       "      <td>hist_zeros_1.0</td>\n",
       "      <td>hist_zeros_2.0</td>\n",
       "      <td>hist_zeros_3.0</td>\n",
       "      <td>hist_zeros_4.0</td>\n",
       "      <td>hist_zeros_5.0</td>\n",
       "      <td>hist_zeros_7.0</td>\n",
       "      <td>hist_zeros_8.0</td>\n",
       "      <td>hist_zeros_10.0</td>\n",
       "      <td>hist_tendancy_0.0</td>\n",
       "      <td>hist_tendancy_1.0</td>\n",
       "      <td>sev_decel_0.001</td>\n",
       "      <td>quant_acc_1</td>\n",
       "      <td>quant_light_dec_1</td>\n",
       "      <td>quant_hist_mean_1</td>\n",
       "      <td>quant_hist_mean_2</td>\n",
       "      <td>quant_hist_mean_3</td>\n",
       "      <td>quant_hist_mean_4</td>\n",
       "      <td>quant_hist_mean_5</td>\n",
       "      <td>quant_hist_mean_6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Coefs</th>\n",
       "      <td>0.0253198</td>\n",
       "      <td>0.111056</td>\n",
       "      <td>0.0133262</td>\n",
       "      <td>0.0272228</td>\n",
       "      <td>0.00884189</td>\n",
       "      <td>0.0499036</td>\n",
       "      <td>0.167461</td>\n",
       "      <td>0.090732</td>\n",
       "      <td>0.0614142</td>\n",
       "      <td>0.0280673</td>\n",
       "      <td>0.0213656</td>\n",
       "      <td>0.0337478</td>\n",
       "      <td>0.0202811</td>\n",
       "      <td>0.0143864</td>\n",
       "      <td>0.0390797</td>\n",
       "      <td>0.0669102</td>\n",
       "      <td>0.0514727</td>\n",
       "      <td>0.0465844</td>\n",
       "      <td>0.036311</td>\n",
       "      <td>0.0209053</td>\n",
       "      <td>0.0227478</td>\n",
       "      <td>0.00083542</td>\n",
       "      <td>0.000683196</td>\n",
       "      <td>0.000339523</td>\n",
       "      <td>0.000314223</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00280929</td>\n",
       "      <td>0.00294722</td>\n",
       "      <td>0.000320958</td>\n",
       "      <td>0.0175078</td>\n",
       "      <td>0.00272703</td>\n",
       "      <td>0.00347671</td>\n",
       "      <td>0.00196887</td>\n",
       "      <td>0.0027366</td>\n",
       "      <td>0.00110818</td>\n",
       "      <td>0.0016362</td>\n",
       "      <td>0.00345193</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    0              1               2                     3  \\\n",
       "0      baseline value  accelerations  fetal_movement  uterine_contractions   \n",
       "Coefs       0.0253198       0.111056       0.0133262             0.0272228   \n",
       "\n",
       "                         4                         5  \\\n",
       "0      light_decelerations  prolongued_decelerations   \n",
       "Coefs           0.00884189                 0.0499036   \n",
       "\n",
       "                                     6                                     7  \\\n",
       "0      abnormal_short_term_variability  mean_value_of_short_term_variability   \n",
       "Coefs                         0.167461                              0.090732   \n",
       "\n",
       "                                                       8  \\\n",
       "0      percentage_of_time_with_abnormal_long_term_var...   \n",
       "Coefs                                          0.0614142   \n",
       "\n",
       "                                         9               10             11  \\\n",
       "0      mean_value_of_long_term_variability  histogram_width  histogram_min   \n",
       "Coefs                            0.0280673        0.0213656      0.0337478   \n",
       "\n",
       "                  12                         13              14  \\\n",
       "0      histogram_max  histogram_number_of_peaks  histogram_mode   \n",
       "Coefs      0.0202811                  0.0143864       0.0390797   \n",
       "\n",
       "                   15                16                  17  \\\n",
       "0      histogram_mean  histogram_median  histogram_variance   \n",
       "Coefs       0.0669102         0.0514727           0.0465844   \n",
       "\n",
       "                         18            19                 20              21  \\\n",
       "0      uterine_cont_per_min  total_change  sqrt_total_change  hist_zeros_1.0   \n",
       "Coefs              0.036311     0.0209053          0.0227478      0.00083542   \n",
       "\n",
       "                   22              23              24              25  \\\n",
       "0      hist_zeros_2.0  hist_zeros_3.0  hist_zeros_4.0  hist_zeros_5.0   \n",
       "Coefs     0.000683196     0.000339523     0.000314223               0   \n",
       "\n",
       "                   26              27               28                 29  \\\n",
       "0      hist_zeros_7.0  hist_zeros_8.0  hist_zeros_10.0  hist_tendancy_0.0   \n",
       "Coefs               0               0                0         0.00280929   \n",
       "\n",
       "                      30               31           32                 33  \\\n",
       "0      hist_tendancy_1.0  sev_decel_0.001  quant_acc_1  quant_light_dec_1   \n",
       "Coefs         0.00294722      0.000320958    0.0175078         0.00272703   \n",
       "\n",
       "                      34                 35                 36  \\\n",
       "0      quant_hist_mean_1  quant_hist_mean_2  quant_hist_mean_3   \n",
       "Coefs         0.00347671         0.00196887          0.0027366   \n",
       "\n",
       "                      37                 38                 39  \n",
       "0      quant_hist_mean_4  quant_hist_mean_5  quant_hist_mean_6  \n",
       "Coefs         0.00110818          0.0016362         0.00345193  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Best feature table for random forest gridsearch best parameters\n",
    "coef_table = pd.DataFrame(list(X_train.columns)).copy()\n",
    "coef_table.insert(len(coef_table.columns),\"Coefs\",grid_tree.best_estimator_.feature_importances_.transpose())\n",
    "coef_table.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.377089Z",
     "start_time": "2021-03-17T03:13:54.370096Z"
    }
   },
   "outputs": [],
   "source": [
    "# Table comparing random forest best features between imbalanced data set, smote and smote with gridsearch best features\n",
    "coef_table.columns = ['features','grid_search_importance']\n",
    "del coef_table['features']\n",
    "best_features_rfc = pd.concat([smote_vs_coef,coef_table],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.438276Z",
     "start_time": "2021-03-17T03:13:54.380282Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "      <th>13</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "      <th>24</th>\n",
       "      <th>25</th>\n",
       "      <th>26</th>\n",
       "      <th>27</th>\n",
       "      <th>28</th>\n",
       "      <th>29</th>\n",
       "      <th>30</th>\n",
       "      <th>31</th>\n",
       "      <th>32</th>\n",
       "      <th>33</th>\n",
       "      <th>34</th>\n",
       "      <th>35</th>\n",
       "      <th>36</th>\n",
       "      <th>37</th>\n",
       "      <th>38</th>\n",
       "      <th>39</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>features</th>\n",
       "      <td>baseline value</td>\n",
       "      <td>accelerations</td>\n",
       "      <td>fetal_movement</td>\n",
       "      <td>uterine_contractions</td>\n",
       "      <td>light_decelerations</td>\n",
       "      <td>prolongued_decelerations</td>\n",
       "      <td>abnormal_short_term_variability</td>\n",
       "      <td>mean_value_of_short_term_variability</td>\n",
       "      <td>percentage_of_time_with_abnormal_long_term_var...</td>\n",
       "      <td>mean_value_of_long_term_variability</td>\n",
       "      <td>histogram_width</td>\n",
       "      <td>histogram_min</td>\n",
       "      <td>histogram_max</td>\n",
       "      <td>histogram_number_of_peaks</td>\n",
       "      <td>histogram_mode</td>\n",
       "      <td>histogram_mean</td>\n",
       "      <td>histogram_median</td>\n",
       "      <td>histogram_variance</td>\n",
       "      <td>uterine_cont_per_min</td>\n",
       "      <td>total_change</td>\n",
       "      <td>sqrt_total_change</td>\n",
       "      <td>hist_zeros_1.0</td>\n",
       "      <td>hist_zeros_2.0</td>\n",
       "      <td>hist_zeros_3.0</td>\n",
       "      <td>hist_zeros_4.0</td>\n",
       "      <td>hist_zeros_5.0</td>\n",
       "      <td>hist_zeros_7.0</td>\n",
       "      <td>hist_zeros_8.0</td>\n",
       "      <td>hist_zeros_10.0</td>\n",
       "      <td>hist_tendancy_0.0</td>\n",
       "      <td>hist_tendancy_1.0</td>\n",
       "      <td>sev_decel_0.001</td>\n",
       "      <td>quant_acc_1</td>\n",
       "      <td>quant_light_dec_1</td>\n",
       "      <td>quant_hist_mean_1</td>\n",
       "      <td>quant_hist_mean_2</td>\n",
       "      <td>quant_hist_mean_3</td>\n",
       "      <td>quant_hist_mean_4</td>\n",
       "      <td>quant_hist_mean_5</td>\n",
       "      <td>quant_hist_mean_6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>importance_baseline</th>\n",
       "      <td>0.00992676</td>\n",
       "      <td>0.103532</td>\n",
       "      <td>0.00470882</td>\n",
       "      <td>0.0105892</td>\n",
       "      <td>0.00061708</td>\n",
       "      <td>0.0876404</td>\n",
       "      <td>0.157685</td>\n",
       "      <td>0.149226</td>\n",
       "      <td>0.0898387</td>\n",
       "      <td>0.0122778</td>\n",
       "      <td>0.0241374</td>\n",
       "      <td>0.0345421</td>\n",
       "      <td>0.00347558</td>\n",
       "      <td>0.00129207</td>\n",
       "      <td>0.0299309</td>\n",
       "      <td>0.100239</td>\n",
       "      <td>0.0523094</td>\n",
       "      <td>0.0412711</td>\n",
       "      <td>0.0316257</td>\n",
       "      <td>0.0267606</td>\n",
       "      <td>0.0168846</td>\n",
       "      <td>0.000475167</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000412919</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0050735</td>\n",
       "      <td>0.00202451</td>\n",
       "      <td>6.36107e-07</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000271607</td>\n",
       "      <td>0.000183496</td>\n",
       "      <td>0.000962509</td>\n",
       "      <td>0.00208523</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sm_importance_baseline</th>\n",
       "      <td>0.0172741</td>\n",
       "      <td>0.20107</td>\n",
       "      <td>0.00249253</td>\n",
       "      <td>0.0203137</td>\n",
       "      <td>0.00140923</td>\n",
       "      <td>0.0858243</td>\n",
       "      <td>0.173725</td>\n",
       "      <td>0.104371</td>\n",
       "      <td>0.0807028</td>\n",
       "      <td>0.00967133</td>\n",
       "      <td>0.0139368</td>\n",
       "      <td>0.0450373</td>\n",
       "      <td>0.007946</td>\n",
       "      <td>0.000497999</td>\n",
       "      <td>0.0139928</td>\n",
       "      <td>0.0629823</td>\n",
       "      <td>0.0220239</td>\n",
       "      <td>0.0390045</td>\n",
       "      <td>0.0415174</td>\n",
       "      <td>0.0158197</td>\n",
       "      <td>0.00860188</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0241652</td>\n",
       "      <td>0.000976957</td>\n",
       "      <td>0.000289048</td>\n",
       "      <td>0.000687007</td>\n",
       "      <td>0.00428605</td>\n",
       "      <td>0</td>\n",
       "      <td>0.000333931</td>\n",
       "      <td>0.00104688</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>grid_search_importance</th>\n",
       "      <td>0.0253198</td>\n",
       "      <td>0.111056</td>\n",
       "      <td>0.0133262</td>\n",
       "      <td>0.0272228</td>\n",
       "      <td>0.00884189</td>\n",
       "      <td>0.0499036</td>\n",
       "      <td>0.167461</td>\n",
       "      <td>0.090732</td>\n",
       "      <td>0.0614142</td>\n",
       "      <td>0.0280673</td>\n",
       "      <td>0.0213656</td>\n",
       "      <td>0.0337478</td>\n",
       "      <td>0.0202811</td>\n",
       "      <td>0.0143864</td>\n",
       "      <td>0.0390797</td>\n",
       "      <td>0.0669102</td>\n",
       "      <td>0.0514727</td>\n",
       "      <td>0.0465844</td>\n",
       "      <td>0.036311</td>\n",
       "      <td>0.0209053</td>\n",
       "      <td>0.0227478</td>\n",
       "      <td>0.00083542</td>\n",
       "      <td>0.000683196</td>\n",
       "      <td>0.000339523</td>\n",
       "      <td>0.000314223</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.00280929</td>\n",
       "      <td>0.00294722</td>\n",
       "      <td>0.000320958</td>\n",
       "      <td>0.0175078</td>\n",
       "      <td>0.00272703</td>\n",
       "      <td>0.00347671</td>\n",
       "      <td>0.00196887</td>\n",
       "      <td>0.0027366</td>\n",
       "      <td>0.00110818</td>\n",
       "      <td>0.0016362</td>\n",
       "      <td>0.00345193</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                     0              1               2  \\\n",
       "features                baseline value  accelerations  fetal_movement   \n",
       "importance_baseline         0.00992676       0.103532      0.00470882   \n",
       "sm_importance_baseline       0.0172741        0.20107      0.00249253   \n",
       "grid_search_importance       0.0253198       0.111056       0.0133262   \n",
       "\n",
       "                                           3                    4  \\\n",
       "features                uterine_contractions  light_decelerations   \n",
       "importance_baseline                0.0105892           0.00061708   \n",
       "sm_importance_baseline             0.0203137           0.00140923   \n",
       "grid_search_importance             0.0272228           0.00884189   \n",
       "\n",
       "                                               5  \\\n",
       "features                prolongued_decelerations   \n",
       "importance_baseline                    0.0876404   \n",
       "sm_importance_baseline                 0.0858243   \n",
       "grid_search_importance                 0.0499036   \n",
       "\n",
       "                                                      6  \\\n",
       "features                abnormal_short_term_variability   \n",
       "importance_baseline                            0.157685   \n",
       "sm_importance_baseline                         0.173725   \n",
       "grid_search_importance                         0.167461   \n",
       "\n",
       "                                                           7  \\\n",
       "features                mean_value_of_short_term_variability   \n",
       "importance_baseline                                 0.149226   \n",
       "sm_importance_baseline                              0.104371   \n",
       "grid_search_importance                              0.090732   \n",
       "\n",
       "                                                                        8  \\\n",
       "features                percentage_of_time_with_abnormal_long_term_var...   \n",
       "importance_baseline                                             0.0898387   \n",
       "sm_importance_baseline                                          0.0807028   \n",
       "grid_search_importance                                          0.0614142   \n",
       "\n",
       "                                                          9               10  \\\n",
       "features                mean_value_of_long_term_variability  histogram_width   \n",
       "importance_baseline                               0.0122778        0.0241374   \n",
       "sm_importance_baseline                           0.00967133        0.0139368   \n",
       "grid_search_importance                            0.0280673        0.0213656   \n",
       "\n",
       "                                   11             12  \\\n",
       "features                histogram_min  histogram_max   \n",
       "importance_baseline         0.0345421     0.00347558   \n",
       "sm_importance_baseline      0.0450373       0.007946   \n",
       "grid_search_importance      0.0337478      0.0202811   \n",
       "\n",
       "                                               13              14  \\\n",
       "features                histogram_number_of_peaks  histogram_mode   \n",
       "importance_baseline                    0.00129207       0.0299309   \n",
       "sm_importance_baseline                0.000497999       0.0139928   \n",
       "grid_search_importance                  0.0143864       0.0390797   \n",
       "\n",
       "                                    15                16                  17  \\\n",
       "features                histogram_mean  histogram_median  histogram_variance   \n",
       "importance_baseline           0.100239         0.0523094           0.0412711   \n",
       "sm_importance_baseline       0.0629823         0.0220239           0.0390045   \n",
       "grid_search_importance       0.0669102         0.0514727           0.0465844   \n",
       "\n",
       "                                          18            19                 20  \\\n",
       "features                uterine_cont_per_min  total_change  sqrt_total_change   \n",
       "importance_baseline                0.0316257     0.0267606          0.0168846   \n",
       "sm_importance_baseline             0.0415174     0.0158197         0.00860188   \n",
       "grid_search_importance              0.036311     0.0209053          0.0227478   \n",
       "\n",
       "                                    21              22              23  \\\n",
       "features                hist_zeros_1.0  hist_zeros_2.0  hist_zeros_3.0   \n",
       "importance_baseline        0.000475167               0               0   \n",
       "sm_importance_baseline               0               0               0   \n",
       "grid_search_importance      0.00083542     0.000683196     0.000339523   \n",
       "\n",
       "                                    24              25              26  \\\n",
       "features                hist_zeros_4.0  hist_zeros_5.0  hist_zeros_7.0   \n",
       "importance_baseline                  0               0               0   \n",
       "sm_importance_baseline               0               0               0   \n",
       "grid_search_importance     0.000314223               0               0   \n",
       "\n",
       "                                    27               28                 29  \\\n",
       "features                hist_zeros_8.0  hist_zeros_10.0  hist_tendancy_0.0   \n",
       "importance_baseline                  0                0                  0   \n",
       "sm_importance_baseline               0                0                  0   \n",
       "grid_search_importance               0                0         0.00280929   \n",
       "\n",
       "                                       30               31           32  \\\n",
       "features                hist_tendancy_1.0  sev_decel_0.001  quant_acc_1   \n",
       "importance_baseline           0.000412919                0    0.0050735   \n",
       "sm_importance_baseline                  0                0    0.0241652   \n",
       "grid_search_importance         0.00294722      0.000320958    0.0175078   \n",
       "\n",
       "                                       33                 34  \\\n",
       "features                quant_light_dec_1  quant_hist_mean_1   \n",
       "importance_baseline            0.00202451        6.36107e-07   \n",
       "sm_importance_baseline        0.000976957        0.000289048   \n",
       "grid_search_importance         0.00272703         0.00347671   \n",
       "\n",
       "                                       35                 36  \\\n",
       "features                quant_hist_mean_2  quant_hist_mean_3   \n",
       "importance_baseline                     0        0.000271607   \n",
       "sm_importance_baseline        0.000687007         0.00428605   \n",
       "grid_search_importance         0.00196887          0.0027366   \n",
       "\n",
       "                                       37                 38  \\\n",
       "features                quant_hist_mean_4  quant_hist_mean_5   \n",
       "importance_baseline           0.000183496        0.000962509   \n",
       "sm_importance_baseline                  0        0.000333931   \n",
       "grid_search_importance         0.00110818          0.0016362   \n",
       "\n",
       "                                       39  \n",
       "features                quant_hist_mean_6  \n",
       "importance_baseline            0.00208523  \n",
       "sm_importance_baseline         0.00104688  \n",
       "grid_search_importance         0.00345193  "
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_features_rfc.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.804983Z",
     "start_time": "2021-03-17T03:13:54.441303Z"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAv8AAAESCAYAAABqybrjAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/Il7ecAAAACXBIWXMAAAsTAAALEwEAmpwYAABruElEQVR4nO3dd5wV1fnH8Q9iFxTUtcXYoj5oNJoYoxhFsBIUFRvxZ1TQaIzdYEtQwF6xxBorJoooSlQQsEWi2LFFYnwskSgWigVRLAj7++M5V4bh7u692+7Cft+vF6+7d+bMmWfmzi7PnDnn3DbV1dWIiIiIiMiib7FKByAiIiIiIs1Dyb+IiIiISCuh5F9EREREpJVQ8i8iIiIi0koo+RcRERERaSWU/IuIiIiItBKLVzoAERERaVpm1ge4pYSivdz93iaOZWngaHcf3JT7qS8zGwdsD6zr7pMqG03DtPRzLZWh5F9ERKT1+Ccwrpb1rzdTDAYoIW16OteyACX/IiIircc4dx9U4RhWrfD+WxOda1mA+vyLiIiIiLQSavkXERGRBZhZG+B3wBHARsDXwBPAQHd/KVe2HXAisA/wI2AJ4D3g78CZ7v6lma0DvJPZphq41d37ZPrZd3T3zzJlCtvc5+57pWVDgEOAXwC3AusBLwK/dPdqM1sfGATsDHQA/gv8FbjE3WfX81yMA9YEdgQuBnZJqx4GjgJmAxcBexMNq+OB47JjBtLx3g7cBFwIbAp8lJad4+5f5/a5JdAf2A5YLh3HbcBgd/8mU24SMCmdiwtS2QdTLNl93+rufdL7TYBTga7E04GvgVeBS939nsx2g4CBxOd/MPCbVP4t4Ep3vy4Xc1vgeOLzWR/4hOhmNsDds599ydeWND61/IuIiEgxtwLXAksC1wHDgS7AU2a2Q6GQmS0OPAKcCXwIXAPcDCwDnJzqAfgslZkBfJN+vrcB8Y0kktDrgH+kxP9nwARgP+AfwGVEAnoecH9KTutreeBJYC3geuA/wL7AXcTxb0Mc63NAT+DulORmbQaMBWYBVwOfEgn+A2b2fU5mZnsBTwHdiRuM64A5wLnAw2a2ZK7eH6f67iU+pzuo4Vyb2S9SjLsTNwmD0+svUsy7Fzn224DDgdHADcAPgGvN7PBMzIsBo1J9iwM3Ao8DvwbGm9kPMvWVdG1J01DLv4iISOvRNbXmFjOk0FJtZvsBBwFDgUPc/bu0/Hwiuf6rma3n7t8SCfBWwLnufnqhMjM7FXgT2MvMlk0t+oPSzEMdGmHswZPuvk9mf22IpHIpYBt3fyGz7lLiycTviJuT+qginmTsk240FgfeJlrPn0r7/Dbt77G0vBNxk1CwCXC1ux+Tyi1O3Dz0IlrWh5jZ8sTN0yygm7u/mCk7BDiQaLU/O1PvysSThiszy+6u4VyfRTyZ2cLdv4/NzPYH7gT+j0jis1YCNnb3aansUOJG6DDiZgCgD3GzMhz4TeZcjCKuo1OB48q8tqQJqOVfRESk9die6MZR7N86mXKHpdcTCskZQOq6cS3R8rtzWvwi8Fvg8uyO3H1mWtcWWLFxDwOAe3LvtyKS65uyiX9yBvAt0LeB+7zC3asB0nl5Li2/MpesPpte18lt/0WKhUwdJ6e3B6bXPYGOaV8v5sqeCHzFvM8nK38+anIZcGA28U/GpddVimxzcyHxT7E8RTzJWSdT5oD0emLuXAwjnryMT+/LubakCajlX0REpPU4s8QW9y2IfthHm1l+Xaf0ujnwgLu/AbxhZkub2VbAhkR/7y2I1m+IG4DG9k7u/Rbp9Uc1PN2YCWxmZm0KCXw9vJV7/2UNsRT67y+VW/4vd/80u8Dd3zazT4guQRDnFaLLDLmy08zMgc3NbAV3n5FWfevuH5RyAO7+IICZrZb2+SPiM902FSn2Wb1RZNnnRFeogs2Ad939/dz+qomuTQUlX1t1HIrUk5J/ERERyetA5AgDaymzInzf1/uPQD+ixRpgKtEVZhIxoDPf970xfJV73yG9dk//atKOuBGojy9rWP5NDcvz3q9h+UfEDRPMS6hn1FD2AyI5XjZTJn8uamRmawF/BvYgPpe5RHI/HvgpxT+rYsdXnSvbEZhSQggdKPHakqah5F9ERETyvgBmuvtaJZTtB5xDdBu5EHjZ3T8CMLMxRPJfl0JLfL478rIlRRu+SK+HufvNZWzXnJapYXkHYHr6uXBj8gPg+SJlCzdYH5e78zQu4gFgY6Irzr3Av939KzNblei+VV9fAO1r2O9y7v5lplyp15Y0AfX5FxERkbx/AWumriHzMbPdzOwcMyt0U/k/YiaaPd19bCbxb8O8bhzZFuJiXW4KfcSXyy3/UZkxA/y8SMxLmNlgMzu2jPqawhbZWX0AzGxtYA3mjRN4Ob1uS04aDLw58FaJA2Lz5/onxLiIEe5+urtPcPfCU4PCTVp9n9K8CqxV7JoBXjazQtehcq4taQJK/kVERCRvCJEEXpWdVtLMViemZvwj81qovyb6iVfl6jiDeQNCl8gsn517D/B6ev1+mkkzW5p5g2FL8TjR9/4wM+ucW3ca8AfmjQuolNXJHJOZLQFcmt4WnlbcS3TnOSpNXVoouzhwBfH04K8l7i9/rgtjEeYb1GtmKxLfXwALfjaluo24Zi7MTqmaZvdZn5gOFcq7tqQJqNuPiIiI5A0h+oTvA7xqZg8SOcP+xLSPp7n7f1PZ24CtgSfN7C6iFb8b8DOi7/8qaZuC94ENzOw24CF3/yuR+B4NXGFmWxNdYPYkkuCa+tnPx93nmNnBxDz6j5vZfcRUnD8HdiBuDP5Y/qloVF8A56W57F8jvjRsU+Bv7j4KwN0/N7NDiWk3nzKzvxN96XdIZZ8guleVYr5zTXxWzwFdzOwJYrrOlYG9gKWJ6UVXKl5VnW4irpeDgZ+k6U5/kJa9w7xBv0Mo/dqSJqCWfxEREZlPmqFlX+LbWmcRfcF7EwlrL3fPJp/XAMcSfdB/S3QDmklM/XhEKtMjU/5U4N/EF3EdlPb3SiozgUgCf0O0FO8IfEeJ3H088WVVw4lvxj0eWJsY4NrZ3T8sta4m8jZxU7MacCSR9J5AfCPu99x9BNHt52Fi8HLhPJ4M7FjGHPjznWt3n5v2PwRYFziO+HKtMcRTkYeADc2snO5WhZjnEF9udjrxdOJo4obldmC7wixHZV5b0gTaVFfXd7YrERERESmFmVUDr7j75pWORVo3tfyLiIiIiLQSSv5FRERERFoJJf8iIiIiIq2E+vyLiIiIiLQSmupTRERarO++m1P96aezKh1GUR07LktLjE1xlUdxlUdxladScVVVta/xy9rU7UdERFqsxRdvW3ehCmmpsSmu8iiu8iiu8rTEuJT8i4iIiIi0Ekr+RURERERaCSX/IiIiIiKthJJ/EWkUZlbj4CKZn86ViIhUimb7EZEGM7PDgbWB0xtYTx/gFqDK3ac3QmgtipktBVwEPAbcW9lompaZDQJOcvd2DamnZ7/7GiegRczNp+1Q6RBEZCGlln8RaQz9gQ6NUM8DQGfgs0aoqyVaHTiO1tHwciPQrdJBiIjI/FrDf0AispBw92nAtErHIQ3n7pOByZWOQ0RE5qfkX0RqZWbVwMnufklm2b1AB3fvamaTiC4/R5vZ0e7eJpXZguji0hn4EhgGnOrus9L6ccAbadsuREvxC2S6/aS6rwHWBXoTf7P+Dhzj7jMz8RwHHAusBbwFnOXud5Z5nMsAZwO/Jp5i/CvF+0Ra3w4YCOwLrAa8Cpzu7g+l9V2J7jxdgAuALYAPgPPc/UYzWwd4J+1uuJn90927lhjbJOCmdB72Bz4HbgDOdPe5pZyHzP5PAE4EOgK7ufv4EvY/DngZ+AY4FFgKuB04CTgfOBj4Cvizu5+fthlEpttPuo76At2B3VJdt6Uy35VyHkREpOHU7UdEGqoX8BFwN5HoY2YbA48D1USyeiqRvN+V27Yv8DqwJ/DXGur/E5Go/poYU3AAmbEFZjYQGEzcXPQEHgbuMLP9yjyOO4HDiRuWvYApwBgzW9/MFgPGpngvAPYG3gVGm9muuXqGAfcAPYCXgBvS+fgwbVc4pqPKjK8f8APifF6T6ji7sLKM83AG8XkcCzxfxv4PBTYmzv9g4Eji+FYgbojGAueZWeda6riceLKzF3A1cDxxzkVEpJmo5V9EGsTdXzKzb4Ap7v5MWnwGcUPQw92/BTCzN4HHzayLuz+eys0ETiy0XpvZj4vsYjJwgLtXAw+lFvYewKlm1gE4DbjQ3c9I5R8ys/ZEkj68lGMws82IhPlgd/9bWvY4kdz+EtgovXZ39wfTZmPM7GngPODBTHVXuPulqY4XiZujX7n7YDN7KZV5091fKyW2jM+Bnul8jjazFYATzOwcoiW+1PNwe7lPRZLvgP3d/SvgETP7HdGA1Nfd55rZE8BBwFbA0zXU8ZS7H5t+ftTMehKf5bX1iKdVq6pq36D1laK4yqO4yqO4SqPkX0SaQjdiNpu5Zlb4O/M0kcDuSDwVAHgr222lBs+lxL9gMrB5+nlrYGnggcx+AMYAh5rZuu7+DnXbJr2OLCxISfaPAczsYmBmJvEvGAZclpLsgsINEO7+mZl9ASxXQgx1GVG4kUruI7rdbAG0o47zQDyFAfB67v/VlPgXTAXeKHx+7v5NOtYOtdTxTO795BS7lGnatJk1rquqal/r+kpRXOVRXOVRXAvutyZK/kWkKawE/C79y1s98/PUEuqalXs/l3ldFldKr0/VsO3qzOtnX5sVgdnu/lkN6zsS3YDypgBtgOxf2dribYgPc+8LA6NXzOy/tvPwQfq5lHNeTLH/vfLHWpemOjciIlIiJf8iUop8glZXa+0MomW6WHeOxpy/f0Z67UXxmWVKbeWeASxhZiu4e6FOUv/1T4FPgFWLbLdaev2kxP00xEq596uk16lEEg21n4f89iIi0gqpxUVE6vI5sEbhjZktB/w0V2ZO7v14oBPwgrtPcPcJwHtE//NNGjG2Z4HZwCqF/aR9bQIMIFrlS1FoMd+9sMDMliQGKB9MHE/7IoN7exPH+HWJ+8mfp3L0yH0z8F7EZ/MijXceRERkEaeWfxGpyxigbxq8OhU4hXn9xws+A7Yws+2J/vxnEwn1XWZ2M9Ef/Qzgh8Qg2kbh7tPM7M/AYDPrCDxHjAc4F7jP3T8vsZ4XzWwUcJWZLU9Mk3kk0Vf/L0Rr+rPAbWbWn5jppy8xuHWPMkIuPFXYyczedPdXyth2Y2BYOp9bE18WdloaB1DneTCzFcvYl4iILKKU/ItIXU4kkvfriJbmq4kEfotMmfPS+jHAhu7+gpntQCSf9wBfA08CB7n7+40c3ynETckRwFlE3/jLgTPLrKc3MWf9QKJb0/PAju7+PwAz6w5cSBzTcsS89z3cfWypO0hJ+IXENJvbAD8pI74hxKw+fyeO8Xh3vyazvrHOQ4sycvCeLXIQH7TcAYYiIrVpU12db8ATEZGWJH3J1yh3P6bSsVRAdUtNsFtq8q+4yqO4yqO4ylPB2X5q7O6pln8RWWSlPvJt6ypXqW+YzU3LWZOGjBNolP3nploVEZGFmAb8isii7BBiIGxd/5qdma1TYmzbN2EYpez/kCbcv4iINDO1/IvIomwksGWlg6jBB5QWm7v7Ok0UQyn7L+V7EkREZCGh5F9EFlnu/jHwcaXjKCbN0jOhwjFUdP8iItL81O1HRERERKSVUPIvIiIiItJKKPkXEREREWkllPyLiIiIiLQSSv5FpNVL3wcgIiKyyNNsPyKy0DCzauBkd7+khvXrEFNT7ufud5dQ35rAjcBvgOmNGGqrZmZdgceALRs6o1DPfvc1Skytwc2n7VDpEERkIaDkX0QWJR8CnYE3Siy/E7Br04UjIiLSsij5F5FFhrt/AzxT6ThERERaKiX/IrKwWcnM7gB2B74B/gac4u6z891+zGw54ApgN6AD8B/gHHcfYWZ9gFtSndPM7Ex3H2Rm7YCBwL7AasCrwOnu/lAhgLSfK4BuwBfApUB3YLK798l0ezkSOBNYAvg5MBk4HTgAWBuYlcod7+7vpbonAdcC6wP7A7OBq9M+rgb2Ir64bKC7Dyn1pJnZEKAdcXN0AtARGA0clt4fDbQFhgInuPvczLFeBHQFlgH+AZzk7m9m6u4BnAt0Al4Cbiiy/y1SPZ2BL4FhwKnuPqvUYxARkYbTgF8RWdicQvTP3xO4jkhcj6yh7BXADsBxQA/gNWC4mW0EPACck8p1B240s8WAsUBf4AJgb+BdYLSZ7QpgZssAjwKWyp0MHA9sW2T/pwKHE8n0O8BlwLGp7l2A/sCOwOW57foTiXgv4C5gAPAc8FE67onA9Wa2Vi3nqZhd0jEdnuLem/iW4a2Bg4mk/VigdzrWNdN+NwB+n453XWC8ma2RymwN3A94ivcR4uble2a2MfA4UE3c0Jya9nFXmfGLiEgDqeVfRBY2D7v7sennf5jZHkQL/JVFym6byg8HMLMngSnA4u4+zczeTuVecPfpZtYT+CXQ3d0fTOvGmNnTwHnAg8Tg4LWBTu7+Vqr3dSKJzrvS3Udm3lcRreY3p/f/NDMDDsxtNxk4zN2r075/B7zv7iel/U0C3gJ+RtyclKodsI+7f5jqOQjYGNjC3WcCD5rZb4CtgDuAE4nW/p3dfXraZhzwX6Bf+ncKMcbiAHevBsaaWQfiJqLgDOLGpYe7f5vqeRN43My6uPvjZRyD1KCqqn3Rn1sSxVUexVUexVUaJf8isrB5Kvd+EtGlp5gngMPNbHVgFDDK3fvVUncXYGYm8S8YBlxmZu2J7i8TC4k/gLu/YGbvFKnP53vjXmhR/wHx5GAj4gZlqdx2z6dEGnf/ysxmMv/NxcfptUMtx1LMe4XEP5kCtE2Jf7buQr1dgMcKiX+KZ7qZPQpsnxb9Eri9EG9yD/Mn/92Ae4G5Zlb4f+dp4HPiyYeS/0YwbVp8jFVV7b//uSVRXOVRXOVRXAvutyZK/kVkYZPvIz6XmrswHgd8ABwE9CSSz9FA32xCm9GRSIjzpgBtgPbAysC0GsrkTc2+MbNtiC4xPwFmEP3jv0p1ZxX7n6Ix+saXW29H4OUiy6cAP86UyZ/Lj3LvVyKeXvyuSF2r17J/ERFpZOrzLyKLLHf/yt0Huvt6xGDUAcDOwNk1bPIJsGqR5atl1r9PdN/JK7bse2a2AvH04X/ABu7ewd27seCTjJaktvNRePrwMbBKbv1KufczgJuBLYv8O6+xghURkbop+ReRRZKZtTWziWZ2AoCHc4nuJoWBsnNym40H2hcG92b0JsYFfE10JdrEzNbN7GsTYL06QupEtJJfnhkrsBhxM9JSv2F4PNDNzFYuLEg/7wg8mRY9BvTMdOeBGFydr6cTcQ4npC/+eo8Y+LxJUwUvIiILUrcfEVkkufscM3sWGGhmXwOvE7PabMe87iefpde9zewhYgagZ4HbzKw/MZi2LzEAdo9U9jZiNp5RZjaAmJXnXGImm7m1hPQ60e3mDDNrSwykPRrYDKg2sza5fvMtwWVAH+BhMyvMjHQ68C3zZig6lxiPcK+ZXU10aTomV8/ZxBOOu8zsZmBpYhDwD4muTzUaOXjPFtmPF1puH2MRkdqo5V9EFmXHEd8D0J+YqedQoJ+735TWP5qWX0nMwjOHmPZzBJHUjiAS1B7uPgrA3WcT3wr8Xqr78rT9e8Sc/0W5+wxgH6L1/35izv7pwH7E3+KtGumYG0367oHtiHETtwI3EQOsO7v75FTm38TTi5WJ83UAcVOTrecFYsrVKmIw8E1E96mu7v5+cxyLiIiENtXVLa2hSUSk5TKzTYF13f3+zLLlicG9p7j7nysW3KKpuqW2rrfUln/FVR7FVR7FVZ4KzvZTY3dSdfsRESlPB+A+MzsPeBhYnpgPfyYxJWizSl2I6hozMLfwjb0iItK6qduPiEgZ3P0J4ou+ehBjBG4l5qvfzt2n1rZtE3kbmF3HvwEViEtERFogtfyLiJTJ3W8Hbq90HElPFvySsLwPmiMQERFp+ZT8i4gsxNz91UrHICIiCw91+xERERERaSWU/IuIiIiItBJK/kVEREREWgkl/yIiIiIirYQG/IrIQsXM2ri7vp2wlejZ775Kh7DIu/m0HSodgog0I7X8i0iTMLNqMzuplvXrpDL7lljfmmY2Flip0YJsZcysazrnP690LCIiUhlq+ReRSvkQ6Ay8UWL5nYBdmy6cVuFF4pz/p9KBiIhIZSj5F5GKcPdvgGcqHUdr4u6fo3MuItKqKfkXkaa0kpndAewOfAP8DTjF3Web2TrAO8B+7n63mS0HXAHsBnQgWqfPcfcRZtYHuCXVOc3MznT3QWbWDhgI7AusBrwKnO7uDxUCSPu5AugGfAFcCnQHJrt7HzPrCjwGHAmcCSwB/ByYDJwOHACsDcxK5Y539/dS3ZOAa4H1gf2B2cDVaR9XA3sBHwMD3X1IKSfMzLYHxgG/dPenMsuPBi4GVnH3L8zsQOAE4MepyMvAae7+eCo/JJ3HWcCewCPAZekYtnT3CWbWBjgO+C2wQYr/GeAPhS8PM7NxxBODr4DDgOWBh4Cj3P37bw42syOA44H1gP8Bg939hsz6A4A/ARumc3u5u19ZyjkREZHGoz7/ItKUTgGmE8nndUSyemQNZa8AdiCS0R7Aa8BwM9sIeAA4J5XrDtxoZosBY4G+wAXA3sC7wGgz2xXAzJYBHgUslTuZSFC3LbL/U4HDgRPc/R0iUT421b0L0B/YEbg8t11/oC3QC7gLGAA8B3yUjnsicL2ZrVXLecp6AnifuKHJ6g2MTIn/vsSN1APEuepLJPp3mtmSmW12I/7O71EkboB+wIXAjUSXqmOBjYEhuXKHAlul198TN1KXFVaa2R+Iz3cs0BMYno7512n9IcBQ4J9p/a3AZWZ2ct2nQ0REGpNa/kWkKT3s7semn/9hZnsQiWOxFt9tU/nhAGb2JDAFWNzdp5nZ26ncC+4+3cx6Ar8Eurv7g2ndGDN7GjgPeBD4DdFq38nd30r1vg5MKLL/K919ZOZ9FXCSu9+c3v/TzAw4MLfdZOAwd69O+/4d8L67n5T2Nwl4C/gZcXNSK3efa2bDgP3NrF+qd410rPukYusDV7v7oMJ2ZvYtMIJoWZ+YFi8O/N7dP01luuZ290PgbHe/InOMHYFLzaydu3+Rls8Bdnf3r1M9mxE3SqSbsD8Bt7h7v1T+ETNbD9jOzO4iPo/b3f2YtP4hM6sGzjCza9z9y7rOizSdqqr2i+S+yqG4yqO4ytPS4lLyLyJN6anc+0lEC3UxTwCHm9nqwChgVCaZLKYLMDOT+BcMI1qV2wNdgYmFxB/A3V8ws3eK1OfzvXHvDWBmPyCeHGxE3KAsldvu+cLUo+7+lZnNZP6bi4/Ta4dajiVvKNEqvxXRDWdf4HNgTNrPBSm2DkCnFN8eadtsfNMKiX8x7n58qqcq1dOJaJkv1FNI/l8pJP7JZGC59LMRMzBlb5xw9wNT3Z2ANYAHzCz7f84Y4CzgF0RXJKmQadNmNst+qqraN9u+yqG4yqO4ylOpuGq74VDyLyJNaVbu/Vxq7m54HPABcBCRgM41s9FAX3efXqR8R+LJQN4UoA3QHlgZmFZDmbyp2Tdmtg3Rn/8nwAzgJaLfe5vcdsX+quePuyzu/qKZObAfkfzvD4xIg6Qxs9WAm4BfAd8C/yZurMjFN98x5aXE/AbipmYW8Apxk5Gvp9jnWFi/Yh37KkzNOjT9y1u9thhFRKRxqc+/iLQI7v6Vuw909/WIFugBwM7A2TVs8gmwapHlq2XWv09038krtux7ZrYC8fThf8AG7t7B3bux4JOMpnQHsE968rBNel8wlBig2xlo5+5bEDcDJUvddUYSA5w3Bdq7+zbA/WXGOSO9zndOzWzDdANVWH80sGWRf/knNyIi0oSU/ItIxZlZWzObaGYnAHg4F3gaKAyUnZPbbDzQvjC4N6M3MS7ga6Ir0SZmtm5mX5sQM9LUphPxZOHyzFiBxYibkXzLf1MZSoxX6E88qch2jekMDHP3Z939u7Sse3otNb4qYuzA9e4+0d3n1rOe14kbrd1zy88GLknrPwbWdPcJhX/EE4GzgRVK3I+IiDQCdfsRkYpz9zlm9iww0My+JhLGrYHtiAG0AJ+l173N7CFipptngdvMrD8xmLYv0U++0P/9NiJ5HmVmA4hZec4FqomuKzV5nejOc4aZtQWWIVquNwOqzaxNoZ9/U3H3N81sAnAEMbg3e/PzPNDHzP4FfErMNPT7tG7ZEncxlThnJ5jZVOA74BDmJfEl1ePu35nZecBFZjadmF2pC9FlqVdaP4gYRExavy5wPvAmMd1rjUYO3rNF9uMF9TEWkYWTWv5FpKU4jpi+sj/RFeRQoJ+7F7qzPJqWX0nMwjOHaKUeQST0I4jZa3q4+ygAd59NTGH5Xqr78rT9e8wbzLoAd59BzKzTkegGczUxZel+xN/NrRrpmOsylLhhuSO3vC/xPQi3EAOcNyJmUfqSeCpQp3TzsjdxHu5MdS1LfJMypdaT6hpMTBO6L9Fdqhfwa3e/L62/ipjidQ9gNDHQdziwW1PfRImIyPzaVFfr766ILJrMbFNgXXe/P7NseaLV+xR3/3PFgpNSVbfUVuyW2sKuuMqjuMqjuMpTwdl+auy6qW4/IrIo6wDcl7qlPEx8O+2JRJeeYc0dTOpCVFdf+rmZ/vciIiKNSt1+RGSR5e5PEF/01YMYI3ArMZXldu5e6zSYTeRtYHYd/wZUIC4REWkl1PIvIos0d78duL3ScSQ9WfBLwvI+aI5ARESkdVLyLyLSTNz91UrHICIirZu6/YiIiIiItBJK/kVEREREWgkl/yIiIiIirYSSfxERERGRVkIDfkVEBDNr0xK/bbdnv/sqHUKrdfNpO1Q6BBFpAmr5FxHMbJKZXVXpOBqTmQ0ysy8aoZ4hZjaxMWIqY59dzazazH7eDPtaysyuAPbMLFvkrgcREQlq+RcRad1WB44Dnsgs6wV8WplwRESkKSn5FxGR+bj7S5WOQUREmoaSf5GFhJlVA0cCPYCdgSnAJe5+dVq/DvAOcAJwItAR2M3dx5tZL+BPwMbAJ8AQ4Ex3/66Gfa0MnA/8ClgReAY4xd0npPV9gEuA3sBgYCPgbeA0d78/U0834EJgU+C/wB+AB4DfuvsQMxsEnOTu7TLbbA68BHRz93Fp2RbARUBn4EtgGHCqu8/KbHcScCywMjCCenxTrpktDpwD9AGWBW4E2hYpd1za11rAW8BZ7n5nZv0ywNnAr4EOwL9SvE9k9jMg7WcVYGJa/2gtsdV6DsxsHPAGsDbQBbjR3Y81s18Ag4Bt0jG9A1zq7n/JXDMAw83sn+7e1cwmAaPc/ZhU9zpp312BZYB/EJ/bm2n9IGB34lo4M52XV4Hj3f2pVGY54Apgt3RO/gOc4+4jajpmERFpfEr+RRYuFxLJ897ALsBVZvatu9+QKXMGcDSwFPC8mR0B/AW4BugPbE4kaOsCv8nvwMzaAU8BSwCnATOIpP1xM9sq8y217YGbiWR5UtrvnWb2A3f/xMw2BcYADwMDgU2AuyiSTNfGzDYGHgeeBvYnkuULUvy7pzInpWXnpnJ9iRugb8vZF3A5cChxo/RmOu7tiKS6EM9A4PS0vyeIm7E7zGyuuw9Pxe4EtifOyWvE5zHGzDZ397eAG9KxDAD+TXwOY8ysayFZLvccJH2Jz3kw8KmZrQU8Rlwz+xF/848CrjOzp9Jx7U3cLP0JWGB0rZmtCTwHvA/8HmiT4h5vZj9198JN1obAWcRnPYO4Voeb2drpJvMKYAeii9F04LC0fhN3/09+v1J5VVXtF4o6G4PiKo/iKk9Li0vJv8jC5XV3PzD9PNbMfgj8kUgmC24vtEKbWVsiOR/m7ken9Q+Z2QwiAbzI3f+V20df4EfApu7+WqrnQSIZHgTsk8otCZzs7nelMlOAV4BuwD3EjcNkoFdK/saY2VziiUE5zgA+Anq4+7dpX28SNyNdgPHAqcAN7j4wE+/L6ThKYmYrEk9W+rv75WnZo8D/MmU6pOO60N3PSIsfMrP2RDI+3Mw2A3oCB7v739J2jxNPM36ZWv37AIe7+42pjrFmtjrxWRWbYqXWc+Duj6dyM4ET3X1uKvMr4obhQHefnZY9Qzz92d7dXzWzQhefNwufd86JRGv/zu4+PdUxjniS0y/9g7gZ3Mndn0tl2hI3E5sBLwDbAg8XbpDM7Eni6ZX+H2qhpk2b2aj1VVW1b/Q6G4PiKo/iKk+l4qrthkN/dEUWLsNy7+8D9k2tswWe+bkTUAUMZ37DgOuI7iH55L8L8O9sIuju35rZCOCgXNlnMj9PTq/LpdeuwPBc16LhlJ/8dwPuBeamxBkiof0c2BGYRnT1GZOJtzrFe3IZ+9mKeCqRredrMxsNbJkWbQ0sDTyQiYW0zaFmti7RvQZgZKaeb4EfA5jZkWnx6Fwdo4HzzWzJIrHVdQ4Kyf9bhcQ/7XcMcdO1dHp6sAHwi7R6qVrORVYX4LFC4p/qnZ5ujLbPlPsOmJB5n78engAOTzc5o4huRf0QEZFmpeRfZOHyYe79tPS6IpEIAkzNrO+YXqdkN3L3GWb2DbB8kX10zJfP1JEvPyvzcyHpLEwhvHImvmwd5VoJ+F36l7c6845xem7dR2Xup5R6VkqvC3TNycSzIjDb3T+roUyhjvdrWL9yDdvUdg4Ksp99ofV9cNpuSWJcRuFGoU0N+8/rSDxFyZtCuqFJvsneeLDg9XAcMQ7jIOLJyNx0Y9U3e2MhIiJNS8m/yMJlpdz7VdLrNIq35H6SXlfNLkzdV5YCPq5hm05Flq9WQ/mavE88dcjKv69mwe8baZd7P4N4wnFtkX1MZ95xr5Jblz9XdSkc2yrMP1g4W8+M9NqLeS3bWQ78DFjCzFZw90J5zKwzMX3mDOK4tyFay/OmE/3ns+o6BzXpDxwBHAyMdvcvzWxZor99qT4hd/0kZV0P7v4VMR5goJkZsC/RnelsYiyBiIg0A33Jl8jCZffc+72IcQD5JwIFTiSH++WW906vTxbZZjzwYzPbqLAgdUXpVUP5mjwB7GZm2b8ze+bKfA4sk25GCrYrEk8n4AV3n5BmHHqP6GO/CTFo9QNi4GpWjzJihehG8022ntTFZudMmWeB2cAqhVhSPJsQg2DbMO+pwO6ZepYkBjsfnI6nDbB8ro6diP71xW4I6joHNekMTHD34e7+ZVrWPb0WWv7n1LJ9Yd/d0gxQheNZmehuVNL1YGZtzWyimZ0A4KEwOHutUuoQEZHGoZZ/kYVL9/TNq/cTUyb2ImZ/Kcrd55jZmcCVZvYJ0Xr8E2K2n+HuXuyba28hpgsdbWanE63OJxKtv+eWEesFRHeRe8zsL0Rr9tlpXaFLyBjgUuCmdFybE7PRZJ1NJNR3mdnNRJ/7M4AfAi+l/v0DgBvSoOOHiXOyBXUntt9z98/N7GLgNDP7ihig+3uihfvtVGaamf0ZGGxmHYlZcDYnzst97v458KKZjSJmYlqemAr0SKLv+1/c/X9mdg9wW5oi8z/E+IjTgYvcfW40jJd+Dmo5rOfT8RxDTL25JXGTUk1M+wnznmbsZGZvuvsruTouIwYoP2xm56RlpxMzKV1ey76/l67DZ4lW/6+B14nxE9tRvCvT90YO3rNFDuIDDTAUkYWTWv5FFi4XAesTSfyOQG93v7u2Ddz9KqKbRzdiEOoxRD/wA2soP5MY5PkscDUxOHgu0KWcL39K0zf2JKajvI/ofnJiWv1FKvM68Fuiq8wY4knGvrl6XiBmwKkiZhG6iehS1NXd309lbkr190r7KvdGpWAAMaPR0WlfM4Drc2VOIZLxw4GxwPFEEtwnU6Y38Feim8vfiXEAO7p7YeagA4mbrD+mOg4gZhH6U7GgSjkHNbgAuDXFMQr4P+Lzf5h4KkC6YbmQ6Iv/tyL7fo9I0j9Idd1ETO3a2d2LdX2qyXGp/v7Ag8SUqv3SZyciIs2kTXV1daVjEJESWHzJ18nuXu5sORVhZjsBM9392cyyXYjEb7MiU4yKFFPdUluxW2oLu+Iqj+Iqj+IqTwWn+qxxUgd1+xGRprI1cHL6Ai4nvnn2LODx5k78c1Nq1mSOu6s1REREFmlK/kWkqVxAzMRzGvADYtaYvxNdXZqNma0DvFNC0W7AuCYNRkREpMKU/IssJNy91HnZW4T05V5npH+V9AHzvqSrNl53ERERkYWbkn8RWaSlb9edUGdBERGRVkCz/YiIiIiItBJK/kVEREREWgkl/yIiIiIirYSSfxFZ5JjZQjU4WkREpLlowK+IVExdX1yWmaZzv7q+yTiVXxO4EfgNML0RQ23VzKwr8BiwpbtPMLMhwM/dfZOm3nfPfvc19S6kBDeftkOlQxCRRqLkX0Rasg+BzsAbJZbfCdi16cKR5GxguUoHISIi5VPyLyItlrt/AzxT6Thkfu7+dqVjEBGR+lHyLyKVtpKZ3QHsDnwD/A04xd1n57v9mNlywBXAbkAH4D/AOe4+wsz6ALekOqeZ2ZnuPsjM2gEDgX2B1YBXgdPd/aFCAGk/VxDf8vsFcCnQHZjs7n0y3V6OBM4ElgB+DkwGTgcOANYGZqVyx7v7e6nuScC1wPrA/sBs4Oq0j6uBvYCPgYHuPqTUk5a63rQjbo5OADoCo4HD0vujgbbAUOAEd5+btlsFuCSd7yWBf6R438nU3QM4F+gEvATcUGTf33f7MbPVU/nuQBUwDbgLONXdv8l8jnukuLoAnwLXuPu5pR6ziIg0nAb8ikilnUL0z98TuI5IXI+soewVwA7AcUAP4DVguJltBDwAnJPKdQduNLPFgLFAX+ACYG/gXWC0me0KYGbLAI8ClsqdDBwPbFtk/6cChxPJ9DvAZcCxqe5dgP7AjsDlue36E4l4LyIpHgA8B3yUjnsicL2ZrVXLeSpml3RMh6e49ya+0Gxr4GAiaT8W6J051sfSsR0LHETcED1uZh1Tma2B+4lvPO4FPELcvBSVOcc/IxL7XYkbuOOBI3LFbwGeJW48RgLnmNmvyjxmERFpALX8i0ilPezux6af/2FmexAt8FcWKbttKj8cwMyeBKYAi7v7NDMrdEd5wd2nm1lP4JdAd3d/MK0bY2ZPA+cBDxKDg9cGOrn7W6ne1yn+rcBXuvvIzPsq4CR3vzm9/6eZGXBgbrvJwGHuXp32/TvgfXc/Ke1vEvAWkUC/W/OpWkA7YB93/zDVcxCwMbCFu88EHjSz3wBbAXcQNwQGbOLur6dtHgX+R9wMnEXcjL0BHODu1cBYM+uQ1hfzA6IV/zh3/1da9g8z6w5sz/yf413uPjDtdxzxNKYHMKaMY5YKqKpq36LqaWyKqzyKqzwtLS4l/yJSaU/l3k8iuvQU8wRweOpmMgoY5e79aqm7CzAzk/gXDAMuM7P2QFdgYiHxB3D3F8zsHRbk871xL7So/4BIqjciblCWym33fEqkcfevzGwm899cfJxeO9RyLMW8V0j8kylA25T4Z+su1NsNeBN4y8wKf/9nEed1RyL5/yVweyHe5B5qSP5T96auZraYmW0AbAhsBqzKgjcyz2S2m2tmH6CBwwuFadNm1l2oDlVV7RulnsamuMqjuMpTqbhqu+FQ8i8ilTYr934uNXdJPA74gOiu0hOYa2ajgb7uXmxqz45EQpw3BWgDtAdWJvqoFyuTNzX7xsy2IbrE/ASYQfSP/yrVnVXsL3/+uOuj3HpXIvrxzy6y7s302pEFp0n9qLYgzOwwos//qsQMTc9S/DyU81mLiEgT0B9dEVlouPtX7j7Q3dcjktgBwM7E1JPFfEIkpHmrZda/T3TfySu27HtmtgLx9OF/wAbu3sHdu7Hgk4yWZAbwCrBlkX/7pDIfA6vktluppgrNbHtibMFfgFXcfQ1370XxGyoREakwJf8islAws7ZmNtHMTgDwcC7wNFAYKDsnt9l4oH1hcG9Gb2JcwNdEl5dNzGzdzL42AdarI6RORCv55ZmxAosRNyMt9RuGxwPrApPcfYK7TwBeAE4kBuFCDAjumekWBNEvvyZbA9XErEvTAMxsDWBTWu55EBFptdTtR0QWCu4+x8yeBQaa2dfA60TiuR0xgBbgs/S6t5k9RMwA9Cxwm5n1J/qg9yUGwO6Ryt5GzMYzyswGELPynEsktHNrCel1otvNGWbWFliGmO1mM6DazNrk+s23BDcTXaceNrPziScfRxCt/j1TmXOJ8Qj3mtnVRJemY2qp83miIelyMxtO3Ij1J8Y9LNsUByEiIvWn5F9EFibHAV8SyeUqRJebfu5+U1r/KDGDz5XADe5+TJp15kIiqV0OeBno4e5jAdL3CexKzLn/N+IG4gLgD8Sc/0W5+wwz2we4mJgaczrwOLAfcDdxg9GivqDM3T83sy5EzNcRCfpEYE93H53K/NvMdia+C2AEMcj5aOImqVid/zCzPxBTex5KzGx0FzGu4AQzyw9+LsvIwXu2yEF8oAGGIrJwalNd3dIapkREmo+ZbQqs6+73Z5YtTwzuPcXd/1yx4ASguqUmsi01yVZc5VFc5VFc5angbD81drtUy7+ItHYdgPvM7DzgYWB5og/8TGJK0GaVuhDV1Vd+buEbe0VERMqhAb8i0qq5+xPEF331IMYI3Ap8Dmzn7lNr27aJvE10mant34AKxCUiIosAtfyLSKvn7rcDt1c6jqQnC35JWN4HzRGIiIgsepT8i4i0IO7+aqVjEBGRRZe6/YiIiIiItBJK/kVEREREWgkl/yIiIiIirYSSfxERERGRVqLVJP9mVte82YusRfXYF9Xjksal66RydO5FRFqeVjHbj5kdDqwNnF7pWOrLzDoAQ4FuxBzkP3T3b+vYZingIuAx4N60bBIwyt2PacJwG52ZjQO+cPfd0/sBwMfA1cXWN3Esk1gIz2FdzGwI8HN336SMbaqBk939kiYLrJ7MbE3gRmIO/+kVDqdZmVlX4vd+S3efUOI246jjdyh77ZtZH+AWoMrdp9f1O1pfPfvd15DNpQW6+bQdKh2CSKvWKpJ/oD8wqtJBNNBBwK+APsCbdSX+yerAccATmWW9gE8bPbqmdxQwJ/P+TODkCsUiC4edgF0rHUSFvAh0Bv7TyPXW9vdDv6MiIguB1pL8LwpWBGa5+60NqcTdX2qkeJqVu79W6RhEFhbu/jnwTBPUW+PfD/2OiogsHOpM/tNj/SOBHsDOwBTgEne/OlNmceLr5vsAqwATgVPd/dG0vivxCPpIojVoCaJ7wTtmdgRwPLAe8D9gsLvfkKn7AOBPwIbAZOByd78yF19foDuwG/ANcBtwkrt/lx5Trw0cbWZHu3ubtN2BwAnAj1NVLwOnufvjmbr3AwYCPwJeBc4lus90c/dxqcwWRNeazsCXwLB07LPqOrdZZtYrHefGwCfAEODMdAzjgO0zx3umuw+qo751gHfS2+Fm9k9371rksf0lRLeIi4ENgH8BhwCdgAuAHwDjgEPdfWqqu9bPu4RjXRGYBhzm7kPSsr2Av6dlN6dl+xDnc2XgPlKXgnQOAC42s2PcfZ30fjEzOxs4HGgPPAr8zt0/LCWutM8liO5hBxDXzSzi2j3e3d/LFF3GzG4B9gNmAjcBg9z9u1RPrddlKtOOuL72BVYjrrHT3f2htL4rRX5v0rJrgfWB/YHZRNeKS9PrXkR3i4GF85vqq/Oab6h03V0EdAWWAf6RjvnNtH4QsDswOB3TWum4j3f3pzL11Pm7V0ccfYguKQDTzOxMdx9U379V1POc1xHj9sTv1i9zx3408fu4irt/UdfnlrprdSCu1T2BR4DLyHT7SX3vjwN+S/yezyZuDv6Q+1Kxxczs4lRuLvH7d4q7f5n2NYkaurxlu/3kf0eBY4H7yXVDMrM/pbjWLPxeiIhI0yp1wO+FwBfA3kSCdlXqR19wA9APuIL4T/B1YIyZbZOr51QiMTshJf5/AK4DxhJfaT8cuN7Mfg1gZocQ/dz/mdbfClxmZvlHyZcTyeRexH/Ex6f9QDym/gi4m0jQMbN9gb8BDxA3NX2J/zzvNLMlU5nuwJ3A86neR4A7sjs1s42Bx4FqIiE4FegN3FXDeSwq3QCNAJ5L8V4JnETcAEA8Tr8J+Codw40lVPsh8XlB3FQcVUO59kRSc36K/YfEeTk3s91OwNmZbUr9vIty90+I85rt+Nk1vW6bWbYr8JS7z8hV0Tm9Xkmcr2z5zkRidxwxPuKqUmLKuIxIVC4AdiG6jO1IXGNZhwCrEp/7lcRnf0GuzOXUcF2a2WLEdd83bbc38C4w2szyXVXm+71Jy/oDbYnjv4tIaJ8jrvU9iaT2ejNbK+2vzmu+oVIf++eI5PL3aR/rAuPNbI1M0Q2Bs4BBwD7ETcLwlJiX9LtXggeAc9LP3Zn3O1Ovv1VpWVnnvARPAO8TN39ZvYGRKfEv9XPbjfh7vgcLXqukY76QOA+7Etf4xsz7G1OwC9CFuL7PAA4m/u6WK/87Oob4XTggV+5A4A4l/iIizafUbj+vu/uB6eexZvZD4I/ADWbWiUi2Dnf3GzNlVif+880meFe6+0j4Pvn5E3CLu/dL6x8xs/WA7czsLuA84PZMK9NDqUXpDDO7ptAaRSSIx6afHzWznsR/lNe6+0tm9g0wxd0Lj8HXB67Otp6b2bdEAr4h8Z/4GcDj7t43FXnQzNoD2RavM4j/+HsU+uCb2ZvA42bWpZQWVTNrm87TMHc/OnOcM4DrzOwid/+XmU0G5maOoVbu/o2ZFR7Rv1nLI/kliVbEO1M8WwOnAdtnWhZ3BrZKP5fzeddmDNG6WNAVeAnYLrNsZ+LmMH9sz5gZwLu5bgifAXsUnrqY2WbEU41yVBEt1Ten9/+02NmBuXLvpn19RyTsKwDHphbmmalMjdclkaz9Euju7g+mMmPM7Gniun+Qeb7/vcmYTDwlqU7b/A54391PSsc+CXgL+FmKtZRrvqFOJBL5nd19etrHOOC/RPJZ+D1vD+zk7s+lMm2JJzubAS9Q2u9erdx9mpm9nd6+kAak1utvVUa557yuGOea2TBgfzPrl+pdg7gu9knFSv3cFgd+7+6fpjJdc7v7IXC2u1+R3v/TzDoCl5pZO3f/Ii3/krgmC/VUA9eY2fru/lZdx5Q5tgV+R81sKNDbzE5Ox745cQNyUKn1yqKhqqp9pUNYQEuMCRRXuRRXaUpN/ofl3t8H7Jta+rqmZaMLLXeF98D5udYpz/xswErAfP/BFm4y0n/UawAP5OodQ7Qa/oJ4rA0L9m2dDLSr6WDc/YK0jw5E9xYjWswAljKzpYGtmZesFAxn/gSkG9EVYW4mxqeJ2Xh2JJ4K1KUTkXAOzy0fRiS+XYiuOE3puczPU9JrdoaQj4nWRijh8y5xMPJYYJCZbZDq/wmRYA81s1WBFYB1Ur2leiXX3WpSJu6SuHtvADP7AXFdbEQ8jVgqV/T+XGvlSOAUYAuiKwfUfl12AWZmEv+CYcTTrexfCmdBz7t7dYr5KzObyYKfGaTjr+uaL1J/fXQBHisk/mm/083sUVK3teS7XKyT0+tyZfzu1UfX9Fru36qCss55iYYSx7oVcb3sS/z9GJP2U+rnNq2QsBfj7seneqpSPZ2Ip6mFegrJ/2O5ekYC1xC/AyUn/zX4K/H0qwvxO/Ib4DV3f7GB9cpCZtq0mXUXakZVVe1bXEyguMqluBbcb01KTf7zfaanpdcViQQe4vF1MStnfp6a+XnFIsuyCvUOTf/yVs/8nO9fP5daujSZ2WpEN5pfAd8C/yYSRYA2QMe0/bTcplNy71ciWv9+V0d8telYrG53n5GeWCxfYj0NscBVWcuYhVI+7w9K2OfzRLLUjbgGPiQSvBuI1v/VgMm5/sh1KXYdlDXPeOr+cS1xMzKDeBrxVZF68tdC4VpZoY54CtdlxyJ1FOptQ7SOFxT7HSn2l6TGcSYlXPONoSPRHz1vCvP6qwN84+5zM+8LPy9G6b979VHfv1UFZZ3zUrj7i2bmxNiRZ4huZCPc/Rso63Or6e8oqZ5OxO/WtinmV4ibjHw9pVzX9ZKOdSJwgJk9Dvya8rvliYhIA5Wa/K+Ue79Kep1GJEjVwDZEi17edOLxdF6hH3dVdqGZbUj8J1xYfzTzt0wXvFNkWamGAmsS/VJfSINqezCvj/xUYkBcVW67/PsZxFOQa4vso9R5xT9Jr6tmF6aWvqWY15rYUpTyedcpPfZ/iEj+PwSeSJ/DU0Tyvw6p9bO5pK47o4DxwD6Fbg5mdhGwea54x9z7wueXT1pr8gm5zzxZLbO+MdV1zTeG2o6p1Ou41N+9+qjv36qmdgfQ18wuJWIblFnX4M8tdbEcSXwGmxKt7XPN7CgWnAq1odd1Xf5KTP/5N+LJ7u2NVK+IiJSo1AG/+S992YsYB/AhkSi1AZZ39wmFf8Qg0RMp/p8sxEC7T4rUfTYxA83rxH9Wa+bqXSmVKaclak7ufWeij/2zma4b3dNrG3efQ3Tf2SO33Z659+OJx+cvZOJ7jxjAWeoXJTmRdOyXW947vT5ZYj3F5I+7MdT38y5mDNEVYzvmdZF6nOgy1ZXau/zMrWVdfXUikp/LM4n/YsTYg3zr+C42/7eX7kO0pJY6lep4oH2Rwb29ievp63KDr0Ot13wj7WM80M3Mvm9BTz/vSInXcRm/e6XIX/+Nee02pqHEzFL9iZb3xzLrGuNzqyLGDlzv7hMzT12K1bN96npVsA/xu5b9rpBSFfsdvZ146nseMM7nn0FLRESaQakt/93N7CpiqrbdiNkb9gdw95fN7B7gNotp/P5DJG6nAxelFqYFKkwtWOcBF5nZdGJaxi5EEtwrrR9EDEgjrV+XmJXmTcpr+f8M2MJiar3HiS4nfczsX8QX1vQiZicBWDa9ngU8bGY3EN1RtiZmyIB5/6mdDTwF3GVmNwNLE4MVf0iJSaC7zzGzM4ErzewT4knCT4hpBoe7e0MGYhaenuxkZm+6+ysNqAso7fMuo7oHiZlEViNmMYH4fM4mujg8Usu2nwHbmtkT7v5sOcdQi9eJrh1npEGoyxBPnjYDqs2sTaHPN9H3+laLaRa3J/qj/9HdvypxXw8AzxLnsT8xQLQv0fc7n/g2hlKu+Ya6jBhQ+7CZFWbaOZ34LC8vo55SfvdK8Vl63dvMHqrv36qm5u5vmtkE4AhicG/2pqUxPrepxPV1gplNJW5yDmFew0u2nuWBv6enEJsQs379pZ5J+mfkfkfd/QMze4R44nBoKZWMHLxni+zHC+pjXK6WGpdIa1Nqy/9FRMvRfUQrXm93vzuz/kBiTu0/EgM5DyBmjPlTbZW6+2DiP/V9ie4WvYBfu/t9af1VxHzbexCtwGcRycBumSSsFOel+McQ89b3Jf7jv4UYYLkR0f3kS9IUdR7zfh9E3JCMJGZpOS3V90Uq8wIxQ0gVcA/RN/d9oKu719SvuNh5uAo4LMUwkkgkB7PgDDNl8fiinwvTcfytIXXl1OvzLhLfVOKbSD8h+jJDJMRfE92AvqhpW6JrRDdihpxG+bI6jylF9yFa/+8npucsPJVZjDTjUXIpMVPSKCKJOcndLypjX3OIltcRRII1grhp7OHuTfFt1HVe8w2VEsTtiDEftxK/D5OAzu4+uZZN8/XU+btXokeJG8zC1LnQSNduExhKTCOan9K0wZ9b+lu5N3Hu7kx1LUs88SBXz+3E7EwjiHN0OfNuvMo1iOK/o2OIcTT31LNeERFpgDbV1bXn0Gmqt5Pd/ZLmCallsPjSqbeyLe8W8/FfC6zk7p9VKDSRRZp+9xZtZjYa+MTdS52Gt7qltha31JZsxVUexVUexVWeCs72U2PX0EZpMV1E7Q7samanEf34NyKeINxWavJRYov0nDKfYhT6odf11KY6132gWbTU2Jrqs1gUpbEMbesq5033xUy1/u6lLll19XefW2YXtEa1MMTY3MzsRKILXXfmf4omIiLNqNRuP63RCUQ3pwuAh4muAdcy75uDSzG7hH+H1CO2m0uo99F61NsYBpQQ29s1bt0EzGydEmKazfxz0bdm21PC+UrntSmcQO2/e2+XEN+AJoqtVAtDjM1tR6Jr56nu/nylgxERaa3qbA1198aaCWShkvqbH0PDvlhoyxLK1GfK0kHUPT92pZ59XU/0g6/NN80RSMYHlPZZFPtip9boBUo7X6V8n0PZSvjd60ndX0zWJLGVYWGIsVm5e35mNxERqYA6+/yLiIhUkPr8l0lxlUdxlUdxlacl9vlXtx8RERERkVZCyb+IiIiISCuh5F9EREREpJVQ8i8iIiIi0kponn8RafXMrI2+46Fl6tnvvkqHIBV082k7VDoEkUWOWv5FZKFhZtVmdlIt69dJZfYtsb41zWwssFKjBSkiItKCqeVfRBYlHwKdgTdKLL8TsGvThSMiItKyKPkXkUWGu38DPFPpOERERFoqJf8isrBZyczuAHYnvi36b8Ap7j7bzNYhvjV7P3e/28yWA64AdgM6AP8BznH3EWbWB7gl1TnNzM5090Fm1g4YCOwLrAa8Cpzu7g8VAkj7uQLoBnwBXAp0Bya7ex8z6wo8BhwJnAksAfwcmAycDhwArA3MSuWOd/f3Ut2TgGuB9YH9gdnA1WkfVwN7AR8DA919SKknzcyGAO2Im6MTgI7AaOCw9P5ooC0wFDjB3eem7VYBLknne0ngHynedzJ17wr8CfhZOtbXgbPcfURaPyhtPzidj7XSeT3e3Z8q9RhERKTh1OdfRBY2pwDTgT2B64jE9cgayl4B7AAcB/QAXgOGm9lGwAPAOalcd+BGM1sMGAv0BS4A9gbeBUanBBczWwZ4FLBU7mTgeGDbIvs/FTicSKbfAS4Djk117wL0B3YELs9t159IxHsBdwEDgOeAj9JxTwSuN7O1ajlPxeySjunwFPfewARga+Bg4IYUX+/MsT6Wju1Y4CDihuhxM+uYyvyCuImYmGLrTdzUDDWzqsy+NwTOAgYB+wDLEJ+FGqFERJqR/uiKyMLmYXc/Nv38DzPbg2iBv7JI2W1T+eEAZvYkMAVY3N2nmdnbqdwL7j7dzHoCvwS6u/uDad0YM3saOA94EPgN0Wrfyd3fSvW+TiTReVe6+8jM+yrgJHe/Ob3/p5kZcGBuu8nAYe5enfb9O+B9dz8p7W8S8BbR0v5uzadqAe2Afdz9w1TPQcDGwBbuPhN40Mx+A2wF3EHcEBiwibu/nrZ5FPgfcTNwFvBjYIS7H13YiZm9C7yY6hmVFrcHdnL351KZtsB9wGbAC2Ucg7QiVVXtF6n9lEtxlUdxlUbJv4gsbPLdRCYRXXqKeQI43MxWJ5LQUe7er5a6uwAzM4l/wTDgMjNrD3QFJhYSfwB3f8HM3mFBPt8b90KL+g+IpHoj4gZlqdx2zxemHnX3r8xsJvPfXHycXjvUcizFvFdI/JMpQNuU+GfrLtTbDXgTeCvTQj+LOK87El17bgFuSV2sNiJa+AvzM2aP67vcMUxOr8uVeQzSikybNrPuQg1UVdW+WfZTLsVVHsW14H5rouRfRBY2s3Lv51JzF8bjgA+I7io9gblmNhro6+7Ti5TvSCTEeVOANkTr9crAtBrK5E3NvjGzbYj+/D8BZgAvAV+lurOK/U+RP+76KLfelYBOxLiDvDcBUtL/F1JXIeKG5+X0c/a4vimMI0gKP6v7qYhIM9IfXRFZZLn7V+4+0N3XI5LYAcDOwNk1bPIJsGqR5atl1r9PdN/JK7bse2a2AvH04X/ABu7ewd27seCTjJZkBvAKsGWRf/ukMlcSYwl6AO3cfROii5SIiLRASv5FZJFkZm3NbKKZnQDg4VzgaWK2GYA5uc3GA+0Lg3szehPjAr4murxsYmbrZva1CbBeHSF1Ip4sXJ4ZK7AYcTOSb/lvKcYD6wKT3H2Cu08g+uefSMzeA/G9CmPd/eE01SrEAGpoucclItJqqduPiCyS3H2OmT0LDDSzr4npJ7cGtiMG0AJ8ll73NrOHiBmAngVuM7P+xGDavsTA1T1S2duI2XhGmdkAYlaec4Fq5nVlKeZ1otvNGWmw6zLE9JqbAdVm1qbQz78FuZnoOvWwmZ1PPPk4gmj175nKPA/sYWaHEOdrB2ImIYBlGxrAyMF7tsh+vKA+xuVqqXGJtDZq+ReRRdlxxPcA9Cdm6jkU6OfuN6X1j6blVxKz8MwhWq1HEAn9COCHQA93HwXg7rOJbwV+L9V9edr+PWLO/6LcfQaRNHcE7ifm7J8O7Ef8Ld6qkY650bj758Qg6NeJaVXvI2Y62tPdR6di/YCHifMwghgIvDfxLcudmzlkERGpQ5vq6pbW0CQi0nKZ2abAuu5+f2bZ8sTg3lPc/c8VC27RVN1SW4tbaku24iqP4iqP4ipPBWf7qbHbpbr9iIiUpwNwn5mdR7R4L0/0gZ9JTAnarFIXorr61s/NzbQjIiKtlLr9iIiUwd2fIL7oqwcxRuBW4HNgO3efWtu2TeRtYirO2v4NqEBcIiLSAqnlX0SkTO5+O3B7peNIerLgl4TlfdAcgYiISMun5F9EZCHm7q9WOgYREVl4qNuPiIiIiEgroeRfRERERKSVUPIvIiIiItJKKPkXEREREWklNOBXRBYKZtbG3cv+VsL6bicN1xjnvme/+xorHJFmcfNpO1Q6BJFaqeVfpAmYWbWZnVTpOIoxsyXM7DYz+9LMPjWztRuhzq7pmH/eGDEWqf9w4Owyt1nTzMYCKzVFTC1ZfT4PMxtnZqPqKDPJzK5KP/dJ+1i52PZmNgA4qp6HICIiTUQt/yKtT3fgQOA04GngvcqGU5L+QK2JaRE7Abs2QSwLgxeBzsB/GrneXsCnNaw7CpiTeX8mcHIj719ERBpIyb9I67Nier3Z3adVNBJpEu7+OfBME9T7Ui3rXmvs/YmISONT8i+NysyqgcOA3YgW5hlEd437geuBrsD7wPHuPiaz3c7AOcBPgI+Bm4Ez3X1OWr8EcDpwALA2MAt4LNXzXiozCbgGWBfoTVzffweOcfeZJcZ/C7CNu1tu+fPA6+5+kJktn45pL2D1dIyjUyyfFamzD3ALUOXu09OyDkQLal93H5KWrQ9cAuxItKCOBE4sbFMqM+tCnMvNga+A4cBp7v6FmQ0BDklFp5rZre7ep8R6DwFOAX4ETE/1/tHdv84U29zMLgd+Tnyr7HnufmOmjnWAi4jrYBngH8BJ7v5mWj8I2B14griO3gY6Ep/50WZ2tLu3KSHWPsQ5B5hmZme6+yAzWxwYAPQBVgEmAqe6+6Npu67EdXUk0XK9RDqWx4BrgfWB/YHZwNXApel1L+K6HVj4PEuIcXtgHPBLd38qs/xo4GJglfSZHQicAPw4FXmZ+DwfT+WHAB2I34k9gUeAy1LMW7r7BDNrAxwH/BbYIMX/DPCH3JeELWZmF6dyc4FhwCnu/mXa1yRglLsfU+R4xgFfuPvu6e8AwMVmdgxwLPE3YEt3n5DZ5k8prjXd/btSzpuIiDSMkn9pCpcRidLVwNHAVcR/8H9L684DbjezNd19lpntCIwB7gYGApbKrJS2L9T5f8BJREL4Y+B84HJgn8y+/wSMBX4NdCKS6Y+AU0uM/Q6gj5n9xN3/BWBm6xEJ4IBUZiiwCdFt5kNgKyLZng70K3E/8zGzVYHxqb6DgaVSnQ+Z2dbu/m2J9fyK6B5zN3F+1ibO5aZm1o24aXmPuJHqDrxVYr1diBuyASnOjYnE92vgj5mil6W6BwHHANeb2XPu/i8zWxN4jrj5+z3QplCfmf3U3T9IdWwGfE50MVmauIkYnfY7uJR4gQeI81c4zn+n5TcQyfuAtOw3wBgz65pNwInr5XCgg7u/Y2YQXY/uTnHtm+r4NXGTtifwh3S8/3D3d0uI8Yl0LvYFsvvuDYxMif++xO/NWUQXmlWJ35E7zWztzHWxG3APsAeRtOf1S+fjVOLmYV3gXGAIsEWm3C7E790hwJrAhWmf+5ZwPFmdiS5lVxI3Ya8C04ib9wmZcgcCdyjxl0VJVVX7FllXY1Jc5WlpcSn5l6bwlLufBmBm7wN7A0+7+3lpWTXROrkhkYicAzzj7r9O2481s0+AIWZ2sbtPAqqIFuKbU5l/WmRkB+b2PRk4IM0w8lBqye1B6cn/o8AUYD/gX2nZ/kRi/7CZLQ0sCRzp7mPT+nFmtg2wfYn7KOYEItHdOfN04FngTSLB/GuJ9ZwDPOfuvQsLzOwd4oZoN3cfaWZvp1UvlPFUYRvgS+ASd/+GOP/fEi3IWee6+xVpvy8SreFdiXN5ItHanz3GccB/ieS0cOO0ONAv28XEzL4Bprh7SV1Z3H1a/jjNrBPR4n945mnEWDNbnThv2Sk6rnT3kblqJwOHuXu1mT0N/A54391PSjFOIm6mfgbUmfy7+1wzGwbsb2b9Ur1rAL9k3g3t+sDV7j4ocy6+BUYQvz8T0+LFgd+7+6epTNfc7n4InF34bIjPryNwqZm1c/cv0vIvge6ZeqqBa8xsfXcv6UYxHdsz6Ybp3cLnaGZDgd5mdnI69s2Jm8iDSq1XZGEwbVpJD5rrVFXVvtHqakyKqzyViqu2Gw4l/9IUnsv8PCW9Zlv7Pk6vHcxsWeAXQP/UJaNgLDEbVTfglkIya2Y/IJ4MbARsS7SQz7fv3NSCk4nuLyVx9zlmdheR/J+RFu8P3J1aJ78jWkcLXVg2JJ4CbEy0gtdXN6Kl9LPMeXgPeI3oBlRn8m9m7YCfEk9Hssf0oJl9Styc5BPaUo0H2gGvpPMzihgzkJ/G8fsWbHf/xMxmEl1SALoAj2VvOFJS/igL3ji9Uc84a9M1vY7OXWujgfPNbMnMMi+y/fOF43X3r9KxFb2uy4hpKHHTsxXRDWdf4qnHmLSfC+D7bmKdiGt/j7Rt9tqfVkjYi3H341M9VameTkDPTD2F5P+xXD0jia5021LiU6Ja/BU4nrgOxhFPXV5z9xcbWK+IiJRBU31KUyh2izurhrIdievwfKIVufBvalq/OoCZbWNmrxDJ/AgiSfqK6DpS237mUv51PjR2aZumfvg/TctIseyRWpXfAW4Hdk77rbMvei1WIrqnzM7925R0DkrQIcUwpci6qcDy9Q3O3ccTXVs+JLr5PAu8bWb52XRqO/8da4htSi62Lwt9zBtZYcrP95n/HF9C9O1fOVN2Kgsq57ouSUp8nbjZhLjRHJGermBmq5nZA8AnRMJ8XGbz7PVWLN7vmVknM3silRsL9AUKXYay9eQ/n8KA8BVKOZ7apGOdCBxgZosRT7T+1tB6RUSkPEr+pdI+T6/nAFsW+TfEzFYgWpr/B2zg7h3cvRvz95NuNKlryTtE14v9iBb48QBmtgEx0PVR4IfuXuXuv6J4S3FBoXU8+/vWLldmBtHaW+wcHFti6J+lfa1aZN1qzGuZrhd3H5nO+8pE3+1ZRN/z/NOXmnzSVLGVaAZxfjpT/DyXNbC6Ed0B7JOeam2T3hcMJQbodgbaufsWwE3lVJ4S7ZHEDc6mQHt334YYgJvXMfe+8Hk11qxQfyXGTGwDrEHcPIuISDNS8i8VlWbheQX4kbtPKPwjWiXPJ/oqdyKSkssL/Y5TQrMzDWttr80dxCDKvYE7M91bfkb0+b/A3SenWJYjukXUFEvhBmeNzLLtcmXGE8f5auYcTCQGzm5bSsCp3/bLzGtFJsW3K9Fy+2Qp9RRjZoPM7Jm0nxnuPoyYkWYFSn+iMB7oZulLoVK9KxPdmuqKbU4d60vZZjzxGS2fu9Z2IsYjVGrQ6VBiYHZ/ouX9scy6zsAwd382Myi2e3ot9dqvIsYOXO/uE929MCC4WD3bp3EtBfsQT2+eKHFfWcUGHt9OTDV7HjCuMFOXiIg0H/X5l5ZgAHCvmc0gpuZcmXgSMJeYJWQJosvFGWbWlhg0ejQxK0y1mbUp0ve8oYYybxabIzLLXyKSygvN7NoU60lE6/U3NdT1GDEe4AozOwdYixhPkC1/KTHLzxgzu4LojtKPSP5OLyPugcB9ZnYnMcvKWkSi9TSpH3k9PQYMMLMbiOkfOxIzK41Pg2tLqeMyYsDtw+k8QBzbt8SsTbX5DNgiTY/5eImf92fpdW8ze8jdXzaze4Db0pSi/yHGAZwOXJQGoZZyHI3K3d80swnEdXa1p+ltk+eJ2af+RUwN24uYKQlg2RJ3MZUYgHyCmU0lbnIOIaZUzdezPPB3M7uUGMtyLvCXeibpnwHbmtkT7v4sgLt/YGaPEF++dmgplYwcvGeLHMQHGmBYLsUl0jKo5V8qzt3vJ/qT/5zoinA5kax2c/dZ7j6DaIHsmNZfTXTR2I+4hrdqgpj+Tdx4vJGddcbd3yCS9J8QA0UvIhK0o4C10mwt+bo+I/pyVxHdl44mZjj5IlPmXaKFfxZwG5FgLwbs5O4vlxH3SGLO+fWB+4i56u8Ads0llWVx938SXX0Kn9FfiIHd+9S2Xa6O94gnHh8AtxLdVyYBnQtPUWpxHnFMY4AflLjLR4EHiekmC4OgDyRuiv5I9H0/gJiy9U+lHkcTGQq0Zf4uPxB98/9DxDyMGOjejZiVp3MpFacbpb2J661wU7gs8cSDXD23E7MvjSDO0eWU3u0sb1CKdUxugPUYYrzOPfWsV0REGqBNdXVjN5iKiIgUZ2ajgU/c/TclblLdUltlW2qLseIqj+Iqj+IqTwWn+qyxa6i6/UirYPENp23rKtdSv2wodXeqq4/33Ex/7lLrXYy6nwBWN+SpQWNrqnPRmBaGGJubmZ1IdNXrThM8rRMRkdKo24+0Ftuz4DSaC/xLc/e3RG9Td/wDaty6ZgNKqPftGreujKY6F41pYYixue1IjFk41d2fr3QwIiKtlVr+pbV4gZjOsS4fNHUg9dSTBb/QLK8+sV9PjEOoTU0DmSulqc5FY1oYYmxW7r573aVERKSpKfmXViFNKTqhzoItlLu/2kT1fsBCloQ21bloTAtDjCIi0jqp24+IiIiISCuh5F9EREREpJVQ8i8iIiIi0koo+RcRkcJ0uCIisojTgF8RkRYmTTn7DrCfu9/dDPsbAHxMfHs2ZjYO+KIlzNDTs999lQ5BZJFx82k7VDoEaQGU/IuIyJnAyZn3RwEt5ovdRESk8Sj5FxGR+bj7a5WOQUREmoaSfxGRHDNbHjgb2AtYHZgBjAaOd/fPzKwtcCpwWFr/BjDI3e9N29e6PpU5DjgWWAt4CzjL3e+sJab1gUuIb8qdA4wETnT36Wn9EKADMAvYE3jE3fc0MwPOAboCKxDf63ATcI67V5tZddrFxWZ2jLuvk+/2Y2YrA+cDvwJWBJ4BTnH3CWl9nxRbb2AwsBHxLcenufv9mXNyPvBrYNW0/s/ufl0tH4WIiDQyDfgVEVnQUCKBPg3YhUhs/w84I62/DBgI3EJ8m++zwN1mtm0p681sIJEkD0vrHwbuMLP9igVjZqsC44G1gYOBI4HOwENmtmSm6G7E3/U9gMvNrB0wDlgJOCSt/wdwFlDoz985vV4J9Cqy73bAU8BO6Xz0BtoAj5vZppmi7YGbiXEDuwPTgTvNbMW0/o/EzdDpwK7AWOBaM9u12DGLiEjTUMu/iEiGmS0NLAkc6e5j0+JxZrYNsH1KZo8CznT3c9L6R1MLexcze62O9ROJJPpCdy/cTDxkZu2BC4DhRcI6AVga2DnT0v8s8CbRkv7XVG5x4Pfu/mkqswXxVKG3u09Lyx4lkvztgZHu/kyExrvu/lKRffcFfgRsWugOZGYPpn0PAvZJ5ZYETnb3u1KZKcArQDfgHmBbYIK7F2IdZ2aziCcVItIMqqrat6h6GpviKo2SfxGRDHf/mmjtL8y6syGwCbAx8DWwFdCW6HaT3a5r2uZXdazvTiTyD5hZ9m/wGOBQM1sXqGZ+3YCngc8y27wHvEZ0Ayok1NMKiX/a5wvAdma2hJltnI7lp8ASwFIlnpIuwL+z4wDc/VszGwEclCv7TObnyel1ufT6BHCOmT0G3EvceJxeYgwi0gimTZvZ4Dqqqto3Sj2NTXEtuN+aKPkXEckxsz2IrjvrEd1XJhAt1G2JPu8AU2vYvK71K6XXp2pYvzrRLz+/zVbA7CLlP8r8vMA+zaw/MZPPCsD/0n5nE113StERmFJk+RRg+dyybCv+3PRa6F56QVp/GHA50S1pPNDH3d8uMRYREWkgJf8iIhlmtgHR9eZWYHt3n5yW30W0/s9IRavIJOlmtjmRUJe6vhfzWseznHk3CAUziCcDA4qUr7FJycwOJgYuHwXc4e4z0vKabkyK+QToVGT5asR3A5TE3ecQN1SXmdlaxGDqM4GriIHEIiLSDJT8i4jM72dE//ULMon/ckSf9U+B54DviEGtr2S2+wsxg80Jdaw/nmh5XyU3+08fYG/gN0ViGk8M4n3V3b9J5ZcG7gb+TtwwFNMZmJydUcfMfkbcmGRb/ufmN8ztex8z28jd/5PqWJK4eXmylu3mY2YPARPd/Q/u/i7wZzP7KfCLUusQEZGGU/IvIjK/l4ipNC80s2uBlYGTiJbub9x9qpldB5xuZrOBF4D9gM2Bo0tYP83M/gwMNrOOxM3E5sC5wH3u/nlmhpyCS4lZfsaY2RXEzUM/Irmvrd/888CR6Rt8/0lMwTmQGFOwbKbcZ8C2ZvaEuz+bq+MW4oZmtJmdTjyFOJGYrvPcWvad9wRxTj5McW1EnJfLyqhDREQaSMm/iEiGu7+RussMJOb2/wh4gDSNpZmtQSTD04FjiJuDicCvCvPel7D+FKJ//hHEtJsfEv3gz6whpnfTNKEXAbcRyfsLwE7u/nIthzOEGOR7JPG9A5OAi4nEe9tMuUHEdwF0MbNVcvueaWZdiOlOryb+33ga6FLD7EA1OY8YM/H7tK+PiMS/6DEXjBy8Z4scxAcaYFguxVWelhqXLPzaVFfnJ5UQERFpMapbagLUUpMzxVUexVUexVWeCs72U+OkDvqSLxERERGRVkLJv4iIiIhIK6HkX0RERESklVDyLyIiIiLSSij5FxERERFpJZT8i4iIiIi0Ekr+RURERERaCSX/IiIiIiKthL7hV0QWCmbWxt3L/lbC+m4nDdcY575nv/saKxwRkYXGzaft0GR1q+VfpBZmVm1mJ1U6jsZmZuPMbFQj1rdOOlf7Nladufr3BK4rc5sOZnY78LOmiKklq8/nYWZDzGxiHWW+v27MrGvax8+LbW9mhwNn1/MQRESkiajlX0QWBicCX5S5zebA/wGXNXo0Ld+HQGfgjUau9yhgTg3rzgaWy7zvDzTaDaaIiDQOJf8iIosYd/8GeKYJ6n2tlnVvN/b+RESk8Sn5l1bNzJYnWiz3AlYHZgCjgePd/bNUbGUzuxfYFfgIuMLdL0/brwO8A+wBHA10AT4FrnH3czP7WRk4H/gVsCKRmJ3i7hPS+j7AJcCFwGnALKAT0dp9GLAb0D3FdzZwP3A90BV4P8U7JtXVBjgO+C2wATA77e8P7v5qA85VD+AsYOMU1yjgJHf/JFNsHTMbneL6DLi6Ec7DR0Cha0k1sK67T6oj1q7AY+nt82Z2q7v3SeuOA44F1gLeAs5y9zvTunWIz/ME4mlDR+LcnwO8DHwDHAosBdwOnJSO52DgK+DP7n5+bbFlYizs60B3H5pZvhtxbtd397fNbFfgT0T3pSWA11PMI1L5QcDuwBPEtfI20CvVvZ+7353KHZiO68dpVy8Dp7n747m4Tk7HtQwwEjjR3aemdeOAL9x99yLHMwT4ubtvYmaTgLWBo83saOAnwL+y8aRt/g+4BVjD3T8u5byJiEjDqM+/tHZDgT2JRHMXIvH8P+CMTJmTgM+JG4R7gMvM7JhcPbcAzxJJ2EjgHDP7FYCZtQOeAnZK++kNtAEeN7NNM3V0AA5M+z/R3b9Myy8D3gR6Ak8DVwGPAE+m2D8DbjezZVP5fkTyfCNxw3IskbAPKevMZJjZ+sCItM8eaR89gatzRc8FnkvnYVQ6Dz0bch6AQ4CX0r47E11a6vIicTMG0JfU99zMBgKDgWEp/oeBO8xsv9z2ZwCnEufu+bTsUOI8HpDqODLFtQKwLzAWOM/MOpcQH+kG5um0bVZv4LmU+P+CuBmdSHzWvYkboqFmVpXZZrP0rxdwen5fqe//34AHiM+vL3Ge7zSzJTNFOwFHEOfuaGBHYJSZlft/RS/ipu1uoHO66XyZOHdZBwKjlfiLiDQftfxLq2VmSwNLAke6+9i0eJyZbQNsnyn6lLsfnH5+0Mx+SCSGV2XK3OXuA1O944iErgcwhki0fgRsWug2YWYPEgn9IGCfVEdb4Ex3fzAX6lPuflra7n1gb+Bpdz8vLasmbgY2JBKsHwJnu/sVaft/mllH4FIza+fu5fadh2h5Xwq4wN0/TPv9gmjdzbrF3Qel9Y+n89CNuCGq93kws8+JFueSurK4++dmVuiiMjEl0h2Im44L3b1wc/eQmbUHLgCGZ6q4vfA0IO0f4Dtgf3f/CnjEzH5HNKD0dfe5ZvYEcBCwFZHUl2IocHHhczGzpYinSIPS+h8DI9y9cCODmb1L3Nxsxbw+9YsD/dz9pVRmndx+1ieewhTqxcy+JW7oNiRuLgp2c/c3UpnpxM3HDsQ1VhJ3f8nMvgGmZD6zW4HzzWz59PlUETfcvUutV0Sktaiqat9kdSv5l1bL3b8mko9CsrQhsAnRuvt1puiI3KYjgf1zCdb3SWlKBD9g3uDHLsC/s/2l3f1bMxtBJIvzhVUk1OcyP09JrxMyywqtph1S3cenY6oiWnI7Ea3cEAl8fZL/54guL8+Z2TCiBfl+d88P/nyq8IO7f2dmkwtx0fDz0FBbA0sDD5hZ9m/fGOBQM1sXKExLWWz/r6bEv2Aq8Ia7z4XoZ59uiDqUEdNdxJOd3YmnEbsC7YE7U523ALeY2XLARsQ1Wpj/balcXTUO7nX3CyBmQCKuByNuMvL1TCwk/slYotvYtpSR/NdgKHAxcfM6hEj6ZxLXkoiIZEybNrNB29d286BuP9KqmdkeZvY20T/6dmBnoltFm0yxKbnNpqXXFTLLZuXKzGXe71fHInUU6l0+t2xqkXLF/gLk9/c9M+uUWqGnEslbX+DbtLpNTdvVxt3/S3QBeYXoCvMY8L6ZHZwr2pTnoaFWSq9PEQlt4V+hxX/1OvZf1udQitSX/lGg0O1of2Bc5unKcmZ2G9G162mi7//SqWz2s/wy001sAWa2mpk9AHwCjCPGhBTUeK2nOfo/Zv5rvV7SsY5lXtefA4knZt80tG4RESmdkn9ptcxsAyLxexT4obtXufuvWLDVt2Pu/arpdRql+SSzTdZqzGu1bxSpb/ZIYmDopkB7d9+GGCDcIO7+ZBrouSLRavwmcLOZ/aDEKprtPNRgRnrtBWxZ5F+9B0M30B3Ar1LXrJ7pfcGVxNOpHkA7d98EOK8e+xhKDP7unOrZAripSLn5rvV0Pa1M6dd6Xf4K7GBmmxBPYv7WSPWKiEiJlPxLa/Yzos//Be4+GaKllejikG0N7Z7bbh/gTXf/oMT9jAd+bGYbFRakQZa9iEGsjamK6N99vbtPLHRJYd4x1Kvl38x+a2bvmNkS7j7L3UcSA0vbAmuUWE1DzkNNc8uXs82zREv/Ku4+ofCP6Oo1gHqem0YwIu37XKJV/57Mus7AWHd/ONNCXp/PsjMwzN2fdffvaqlnczPL3qDtSXQPHVfGvgqKfWb3E93OrgH+6+6Nff2LiEgd1OdfWrOXiATlQjO7lmjhPIloic52RdjJzC4h+ob3Ilq9f13Gfm4hplgcbWanEy3QJxKt4OfWsl19TAXeBU4ws6nEINVDiD7lAMvWtGEdHidaoYeb2TXETdPpRHepl4FSWv8bch4+IxLTrsCzub73tW0DsJuZfeHur5vZn4HBqZX9OeKLwM4F7kuDUFcsod5G5e4z07fmHgE84O6fZlY/D+xhZocQn+sOwMlpXTmf5fNAHzP7FzEVbS/g90XqmQOMTNOHrkb00R/j7k9Rvs+ALcxse+Bxd69O4yLuBH5HTBtbp5GD92xw39emUlXVvkXGprjKo7jKo7jK0xLjUsu/tFppYOPBxBzko4GLiCTpKGAtMyu0aPcnutA8QAzIPDA7E0wJ+5lJDHZ9lpgacxjRF75LYXaWxpL6aO9NtK7eSSTcyxLTa0K0ANen3jeILimrENM3DiX6h+/s7rNLrKMh5+FSYmDqWOCnJYb9b6JbyR+JJBbgFGLaz8NTXccDlwN9SqyzqQwlnqLckVvej5iO9HLiCcGOxOf7BuV9ln2B/xDXwzBi8HA34MtcPf8krvO/Eed8OPPGI5TrPOIp1Bjmvzkck15vq2e9IiLSAG2qq6vrLiUiItII0pOjn7j7tiVuUt3SWs0KWmKLHiiucimu8iiu8lQqrqqq9jV2DVW3H5FWLH0bcNu6ymX6iVecmbWl7v7uczPjHZrdwhBjczOzQ4mnNkdQXrc5ERFpROr2I9K6HcL8017W9K8leZS64725YtGFhSHG5vZzovvRle5+d6WDERFprdTyL9K6jSSmuVyY/I74IqzaTG+OQGqxMMTYrNz9KGI8jYiIVJD6/IuIiIiItBLq9iMiIiIi0koo+RcRERERaSWU/IuIiIiItBJK/kVEREREWgkl/yIiIiIirYSSfxERERGRVkLJv4iIiIhIK6Ev+RIRkWZjZocDpwBrAi8Df3D3p2spvwlwBbAV8AlwNXCRu1dnymwHXAJsCrwPnO/uZX2DchPFtTtwBrAx8DFwP9Df3WdWMq5c+VuAbu6+TqkxNVVcZrYecCmwI/A1MBbo5+5TKxxXT2AQYMBk4Crg6prOaWPEldmuPTCROA9359Y1+3VfYlzNft2XEleuXLNc96XE1RjXfbnU8i8iIs3CzA4BrgNuA/YBPgMeNLN1ayi/CvAIUA3sD1wPnAv0y5TZiPjP8h1gb2AUcJOZ7VvhuHYgkp5/pzrPAX4NDKtkXLnyuwB9So2nKeMys47AE8CqxHk6AegK3FnhuLYB7gVeA/ZKdV8BHN1UcWW2aw/cB6xVZF2zX/clxtXs130pceXKNct1X0pcjXHd14da/kVEpMmZWRvgTOB6dz8zLXsYcOBE4Lgimx1N/D+1h7vPAkab2VLAH83sCnefDZwGTAIOSC2xY81sZWAAUGPLXzPE1Q940t0PzexrBnCXmW3s7q9VKK5C/e2IZPf9Ok5Rc8X1B6JBcpdCC7GZfQ5cbWaruftHFYrrIOA94BB3nws8YmYbA0cSTwCa4nxhZtsTieaqNVRdieu+lLgqcd2XElehXHNe96XE1aDrvr7U8i8iIs1hfWBtolUQgJRcPQB0r2GbnYBHU2JWcC+wIrBlpsyoXBeMe4FNzWyNCsb1DNGFJMvTa60thU0cV8EFwH8pIVFsprh6AXdku4a4+0h3X6vEBKip4loK+CIl/gUfpzKlqE9chTherSP25r7uS4mrEtd9KXEVNOd1X0pcDb3u60XJv4iINIcN0+tbueX/BX5kZm1r2KZYeYANzWw5YI3aylQiLgB3P9vd810deqbX1ysVF3zfV7wvcEQJcTR5XGa2JNAJeMfM/mxmn5rZLDMbmrpFVCSu9HojsL6ZHWdmK5jZTsAhwB1NGBfAdu6+P7BAv+8KXve1xgUVu+7rjAsqct3XGlcjXff1ouRfRESaw/LpNT/obybxf9FyNWxTrHxhXW11ZvfZ3HEtwMw2A/4IjHD3tysVl5ktTSS0Z7l7PpEpRVPE1RFoC/yJaB3uDRwD7AwMrWBcuPtTwPlEP//PgIeB8cCpTRgX7j6xnnVm1zd3XAtopuu+zrgqdN3XFVdjXPf1oj7/IiLSHNqk15pmSJlbZFmbOsrXp87miGs+ZvYT4CGin3GprY5NFdcg4EtgcIlxNEdcS6SfPwd6uft38H3f5+Fm9gt3f64CcWFm5xDJ6wXAg8SMP+cQydn+dcRU37iao86miGs+zXjdl2IQzX/d16Uxrvt6Ucu/iIg0hxnptX1ueXtgjrt/UcM2xcoX1n1eS53ZfTZ3XN8zs67A40Sr8U7u/nEJMTVJXGa2BTGbyJEptsVJSY2ZLZ4GNTZ7XEBhm0cLCVDycHrdtBJxmdkSxADW69z9j+4+zt3/QgwC3i/NbNMUcdWlUtd9yZr5uq8rlkpd93VpjOu+XpT8i4hIc3gzva6XW74e8EYt2xQrD+DpP9wPaytTibgKC8xsD+ZNx7idu79XQjxNGVdPYgDrs8Ds9O84YiDjbKIve7PH5e6fEYNol8yVKbSMljKfflOcr5WBpYlBrFnj0+vGTRRXrSp43ZekAtd9XSp13deqka77elHyLyIizeFNYsrEvQoLUsvqbsCjNWzzKLBTGuBYsBfxH+bLmTI9cwPu9gImlvglOU0Sl5n9ArgLeB7Yvh5f2NMUcV1PzGKT/XcHkUhuCYysUFwQ3UN6mNmymTK7pdenKhTXNKLl+pe57bZKr+80UVylqMR1X6cKXfd1qdR1X4qGXvf1oj7/IiLS5Ny92swuAK4ys0+BJ4nBbSsDlwGY2Y+AKncvtLReAxxLzL9+MVAYPHiau3+bylxCJBrDzewGYrDcb4D9KhzXDUSr4nnAxmaW3e0b7v5JBeL6IP37nplNBb519wkVPl9nA3ukMhcSX4h0ITDM3eucJaap4jKzc4EL01z1Y4ENgLOA59L7poirFJW47ktRieu+Vu5eqeu+FA267utLLf8iItIs3P0a4GSiz/TdQAdgV3cvTFF4BvB0pvyHxHzmi6fyRwD93f2STJlXiMf66wF/B3YH+rp7yfN4N3ZcZrYO8BOgHTA6bZv9V0pf8SY5X42hiT7H/wDbA3OAe4gE+2bK+CbWJorrEuAoIkEbS8zyczvRj31OU8RVYp3Nft3XpVLXfXNpos+xwdd9fbSprm6yLkUiIiIiItKCqOVfRERERKSVUPIvIiIiItJKKPkXEREREWkllPyLiIiIiLQSSv5FRERERFoJJf8iIiIiIq2Ekn8RERERkVZCyb+IiIiISCvx/6Z2kelZgA2UAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pd.Series(grid_tree.best_estimator_.feature_importances_, index=X.columns).nlargest(10).plot(kind='barh')\n",
    "plt.title('Feature Importance', fontsize=20)\n",
    "plt.yticks(fontsize=16)\n",
    "plt.xticks(fontsize=16)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**findings**\n",
    "- In the Above graphe you can see the top ten importanct features for our grid search random foret model.  Abnormal_short_term_variatability, acceleration and mean_value_of_short_term_varitability seem to be key factors in health classification.  We can infer from this that extended rapid extreme changes in fetal heart rate are detrimental to fetal health."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## XGboost"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- We wanted to try an XGboost classifer to attempt a better model than the gridsearch random forest.  XGboost uses gradiant descent and boosting principles to regression trees.  We first tried an XGboost model with some parametrs we thought would be ideal for the model.  Next we ran a grid search on the XGboost parameters to produce our best possible model.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.820612Z",
     "start_time": "2021-03-17T03:13:54.811936Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "        if (window._pyforest_update_imports_cell) { window._pyforest_update_imports_cell('import xgboost as xgb'); }\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "XGBClassifier(base_score=None, booster=None, colsample_bylevel=None,\n",
       "              colsample_bynode=None, colsample_bytree=None, gamma=None,\n",
       "              gpu_id=None, importance_type='gain', interaction_constraints=None,\n",
       "              learning_rate=None, max_delta_step=None, max_depth=None,\n",
       "              min_child_weight=None, missing=nan, monotone_constraints=None,\n",
       "              n_estimators=100, n_jobs=None, num_parallel_tree=None,\n",
       "              objective='binary:logistic', random_state=None, reg_alpha=None,\n",
       "              reg_lambda=None, scale_pos_weight=None, subsample=None,\n",
       "              tree_method=None, use_label_encoder=True,\n",
       "              validate_parameters=None, verbosity=None)"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb.XGBClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:15:18.390060Z",
     "start_time": "2021-03-17T03:15:18.383580Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "        if (window._pyforest_update_imports_cell) { window._pyforest_update_imports_cell('import xgboost as xgb'); }\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "xg_clf = xgb.XGBClassifier(objective ='binary:logistic', \n",
    "                           colsample_bytree = 0.75, \n",
    "                           subsample = 0.85,\n",
    "                           learning_rate = 0.1,\n",
    "                           max_depth = 11, \n",
    "                           alpha = 1, \n",
    "                           n_estimators = 1000,\n",
    "                          verbose=1, n_jobs=-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T04:14:30.303751Z",
     "start_time": "2021-03-17T04:14:30.297595Z"
    }
   },
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "\n",
       "        if (window._pyforest_update_imports_cell) { window._pyforest_update_imports_cell('import xgboost as xgb'); }\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "'1.3.0'"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Checking if XGBoost is in the correct version\n",
    "xgb.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's known that the 1.3.0 version has problems. I need to update it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:15:58.592078Z",
     "start_time": "2021-03-17T03:15:58.566908Z"
    }
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "dlsym(0x7ff7747d59b0, XGDMatrixSetDenseInfo): symbol not found",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-73-35c107e5e418>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mxg_clf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_train\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_train\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/core.py\u001b[0m in \u001b[0;36minner_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    419\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    420\u001b[0m             \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0marg\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 421\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    422\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    423\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0minner_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/sklearn.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, sample_weight, base_margin, eval_set, eval_metric, early_stopping_rounds, verbose, xgb_model, sample_weight_eval_set, feature_weights, callbacks)\u001b[0m\n\u001b[1;32m    905\u001b[0m             \u001b[0mfeature_weights\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfeature_weights\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    906\u001b[0m             \u001b[0meval_set\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0meval_set\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msample_weight_eval_set\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msample_weight_eval_set\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 907\u001b[0;31m             eval_group=None, label_transform=label_transform)\n\u001b[0m\u001b[1;32m    908\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    909\u001b[0m         self._Booster = train(xgb_options, train_dmatrix,\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/sklearn.py\u001b[0m in \u001b[0;36m_wrap_evaluation_matrices\u001b[0;34m(self, X, y, group, sample_weight, base_margin, feature_weights, eval_set, sample_weight_eval_set, eval_group, label_transform)\u001b[0m\n\u001b[1;32m    264\u001b[0m         train_dmatrix = DMatrix(data=X, label=y, weight=sample_weight,\n\u001b[1;32m    265\u001b[0m                                 \u001b[0mbase_margin\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbase_margin\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 266\u001b[0;31m                                 missing=self.missing, nthread=self.n_jobs)\n\u001b[0m\u001b[1;32m    267\u001b[0m         \u001b[0mtrain_dmatrix\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfeature_weights\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mfeature_weights\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgroup\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mgroup\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    268\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/core.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, data, label, weight, base_margin, missing, silent, feature_names, feature_types, nthread, enable_categorical)\u001b[0m\n\u001b[1;32m    506\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhandle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    507\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 508\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbase_margin\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbase_margin\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    509\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    510\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_names\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeature_names\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/core.py\u001b[0m in \u001b[0;36minner_f\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m    419\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mk\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marg\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mzip\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msig\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparameters\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    420\u001b[0m             \u001b[0mkwargs\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0marg\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 421\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    422\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    423\u001b[0m     \u001b[0;32mreturn\u001b[0m \u001b[0minner_f\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/core.py\u001b[0m in \u001b[0;36mset_info\u001b[0;34m(self, label, weight, base_margin, group, label_lower_bound, label_upper_bound, feature_names, feature_types, feature_weights)\u001b[0m\n\u001b[1;32m    527\u001b[0m         \u001b[0;34m'''Set meta info for DMatrix.'''\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    528\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mlabel\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 529\u001b[0;31m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_label\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    530\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mweight\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    531\u001b[0m             \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/core.py\u001b[0m in \u001b[0;36mset_label\u001b[0;34m(self, label)\u001b[0m\n\u001b[1;32m    656\u001b[0m         \"\"\"\n\u001b[1;32m    657\u001b[0m         \u001b[0;32mfrom\u001b[0m \u001b[0;34m.\u001b[0m\u001b[0mdata\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mdispatch_meta_backend\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 658\u001b[0;31m         \u001b[0mdispatch_meta_backend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlabel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'label'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'float'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    659\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    660\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mset_weight\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/data.py\u001b[0m in \u001b[0;36mdispatch_meta_backend\u001b[0;34m(matrix, data, name, dtype)\u001b[0m\n\u001b[1;32m    664\u001b[0m         \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    665\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0m_is_numpy_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 666\u001b[0;31m         \u001b[0m_meta_from_numpy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhandle\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    667\u001b[0m         \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    668\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0m_is_pandas_df\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdata\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/data.py\u001b[0m in \u001b[0;36m_meta_from_numpy\u001b[0;34m(data, field, dtype, handle)\u001b[0m\n\u001b[1;32m    598\u001b[0m     \u001b[0mptr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0minterface\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'data'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    599\u001b[0m     \u001b[0mptr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mctypes\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mc_void_p\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mptr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 600\u001b[0;31m     _check_call(_LIB.XGDMatrixSetDenseInfo(\n\u001b[0m\u001b[1;32m    601\u001b[0m         \u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    602\u001b[0m         \u001b[0mc_str\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfield\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/ctypes/__init__.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m    359\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'__'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mendswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'__'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    360\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 361\u001b[0;31m         \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    362\u001b[0m         \u001b[0msetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    363\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/ctypes/__init__.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, name_or_ordinal)\u001b[0m\n\u001b[1;32m    364\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    365\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname_or_ordinal\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 366\u001b[0;31m         \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_FuncPtr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_ordinal\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    367\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_ordinal\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    368\u001b[0m             \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname_or_ordinal\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: dlsym(0x7ff7747d59b0, XGDMatrixSetDenseInfo): symbol not found"
     ]
    }
   ],
   "source": [
    "\n",
    "xg_clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:15:45.313909Z",
     "start_time": "2021-03-17T03:15:45.293987Z"
    }
   },
   "outputs": [
    {
     "ename": "AttributeError",
     "evalue": "dlsym(0x7ff7747d59b0, XGDMatrixSetStrFeatureInfo): symbol not found",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mAttributeError\u001b[0m                            Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-72-0dc2f5f15ac6>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0my_pred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mxg_clf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpredict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_test\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/sklearn.py\u001b[0m in \u001b[0;36mpredict\u001b[0;34m(self, data, output_margin, ntree_limit, validate_features, base_margin)\u001b[0m\n\u001b[1;32m    970\u001b[0m         \"\"\"\n\u001b[1;32m    971\u001b[0m         test_dmatrix = DMatrix(data, base_margin=base_margin,\n\u001b[0;32m--> 972\u001b[0;31m                                missing=self.missing, nthread=self.n_jobs)\n\u001b[0m\u001b[1;32m    973\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mntree_limit\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    974\u001b[0m             \u001b[0mntree_limit\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"best_ntree_limit\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/core.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, data, label, weight, base_margin, missing, silent, feature_names, feature_types, nthread, enable_categorical)\u001b[0m\n\u001b[1;32m    508\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mset_info\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mlabel\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mweight\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mweight\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbase_margin\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbase_margin\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    509\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 510\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_names\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeature_names\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    511\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfeature_types\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfeature_types\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    512\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/site-packages/xgboost/core.py\u001b[0m in \u001b[0;36mfeature_names\u001b[0;34m(self, feature_names)\u001b[0m\n\u001b[1;32m    836\u001b[0m             c_feature_names = (ctypes.c_char_p *\n\u001b[1;32m    837\u001b[0m                                len(c_feature_names))(*c_feature_names)\n\u001b[0;32m--> 838\u001b[0;31m             _check_call(_LIB.XGDMatrixSetStrFeatureInfo(\n\u001b[0m\u001b[1;32m    839\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhandle\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mc_str\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'feature_name'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    840\u001b[0m                 \u001b[0mc_feature_names\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/ctypes/__init__.py\u001b[0m in \u001b[0;36m__getattr__\u001b[0;34m(self, name)\u001b[0m\n\u001b[1;32m    359\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstartswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'__'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mendswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'__'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    360\u001b[0m             \u001b[0;32mraise\u001b[0m \u001b[0mAttributeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 361\u001b[0;31m         \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    362\u001b[0m         \u001b[0msetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    363\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mfunc\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/anaconda3/envs/learn-env/lib/python3.6/ctypes/__init__.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, name_or_ordinal)\u001b[0m\n\u001b[1;32m    364\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    365\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname_or_ordinal\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 366\u001b[0;31m         \u001b[0mfunc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_FuncPtr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_ordinal\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    367\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0misinstance\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_ordinal\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mint\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    368\u001b[0m             \u001b[0mfunc\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname_or_ordinal\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mAttributeError\u001b[0m: dlsym(0x7ff7747d59b0, XGDMatrixSetStrFeatureInfo): symbol not found"
     ]
    }
   ],
   "source": [
    "y_pred = xg_clf.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.882232Z",
     "start_time": "2021-03-17T03:13:36.848Z"
    }
   },
   "outputs": [],
   "source": [
    "evaluation(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.883874Z",
     "start_time": "2021-03-17T03:13:38.458Z"
    }
   },
   "outputs": [],
   "source": [
    "coef_table = pd.DataFrame(list(X_train.columns)).copy()\n",
    "coef_table.insert(len(coef_table.columns),\"Coefs\",xg_clf.feature_importances_.transpose())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.885853Z",
     "start_time": "2021-03-17T03:13:39.427Z"
    }
   },
   "outputs": [],
   "source": [
    "coef_table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-03-17T03:13:54.887668Z",
     "start_time": "2021-03-17T03:13:44.233Z"
    }
   },
   "outputs": [],
   "source": [
    "clf_xgb = xgb.XGBClassifier(objective = 'binary:logistic')\n",
    "param_dist = {'n_estimators': [500,1000,1500],\n",
    "              'learning_rate': [0.1,0.07,0.05,0.03,0.01],\n",
    "              'max_depth': [9,10,11,12,13],\n",
    "              'colsample_bytree': [0.5,0.45,0.4],\n",
    "              'min_child_weight': [1, 2, 3]\n",
    "             }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-22T00:36:33.407537Z",
     "start_time": "2021-02-22T00:36:28.357Z"
    }
   },
   "outputs": [],
   "source": [
    "grid_xg = GridSearchCV(estimator=clf_xgb,\n",
    "                      param_grid= param_dist,\n",
    "                      scoring='f1',\n",
    "                      n_jobs=-1,\n",
    "                      verbose=1,\n",
    "                      iid=False,\n",
    "                      cv=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-22T00:36:33.411370Z",
     "start_time": "2021-02-22T00:36:28.360Z"
    }
   },
   "outputs": [],
   "source": [
    "grid_xg.fit(smX_train,smy_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-22T00:36:33.414616Z",
     "start_time": "2021-02-22T00:36:28.363Z"
    }
   },
   "outputs": [],
   "source": [
    "grid_xg.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-22T00:36:33.417318Z",
     "start_time": "2021-02-22T00:36:28.366Z"
    }
   },
   "outputs": [],
   "source": [
    "y_pred = grid_xg.best_estimator_.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-22T00:36:33.419458Z",
     "start_time": "2021-02-22T00:36:28.368Z"
    }
   },
   "outputs": [],
   "source": [
    "evaluation(y_test,y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-22T00:36:33.421319Z",
     "start_time": "2021-02-22T00:36:28.371Z"
    }
   },
   "outputs": [],
   "source": [
    "from xgboost import plot_importance\n",
    "plot_importance(grid_xg.best_estimator_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-02-22T00:36:33.423789Z",
     "start_time": "2021-02-22T00:36:28.374Z"
    }
   },
   "outputs": [],
   "source": [
    "plot_importance(grid_xg.best_estimator_,max_num_features=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**findings**\n",
    "- The grid search xgboost produced our overall best model.  We where extremely pleased with the highest recall and accuracy score.  The precision, which was our target metric was also very high.  When graphing the featrue importance, the most important features where abnormal_short_term_variatability and the histogram mean, min, width  and mode.  Our engineered features where not as important as some of the unchanged features.  Our best preforming engineered feature was sqrt_total_change, but no engineered feature was in the top ten.  Just like our random forest model short term varitability of the FHR is a key feature in classifying fetal health."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}