Chapter 05 - SupportVectorMachines.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Support Vector Machines"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.pipeline import Pipeline\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "sns.set()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Iris Dataset\n",
    "The following code, loads the iris dataset, and detects iris-virginica flowers using svm classifier. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.svm import LinearSVC\n",
    "from sklearn.datasets import load_iris\n",
    "\n",
    "iris = load_iris()\n",
    "X = iris['data'][:, (2, 3)]\n",
    "y = (iris['target'] == 2).astype(np.float64)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('std_scale',\n",
       "                 StandardScaler(copy=True, with_mean=True, with_std=True)),\n",
       "                ('linear_svc',\n",
       "                 LinearSVC(C=1, class_weight=None, dual=True,\n",
       "                           fit_intercept=True, intercept_scaling=1,\n",
       "                           loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "                           penalty='l2', random_state=None, tol=0.0001,\n",
       "                           verbose=0))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_clf = Pipeline([\n",
    "    ('std_scale', StandardScaler()),\n",
    "    ('linear_svc', LinearSVC(C=1, loss='hinge')),\n",
    "])\n",
    "svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1.])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svm_clf.predict([[5.5, 1.7]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM Kinds\n",
    "There are two kinds of svm models.\n",
    "1. Hard margin classifier\n",
    "2. Soft margin classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polynomial Regression in SVMs\n",
    "\n",
    "We can use polynomial features to linear a non-linear train set. For example bellow code represent it on moon dataset.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/usr/local/lib/python3.7/site-packages/sklearn/svm/base.py:929: ConvergenceWarning: Liblinear failed to converge, increase the number of iterations.\n",
      "  \"the number of iterations.\", ConvergenceWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('poly_feature',\n",
       "                 PolynomialFeatures(degree=3, include_bias=True,\n",
       "                                    interaction_only=False, order='C')),\n",
       "                ('std_scale',\n",
       "                 StandardScaler(copy=True, with_mean=True, with_std=True)),\n",
       "                ('svm_clf',\n",
       "                 LinearSVC(C=10, class_weight=None, dual=True,\n",
       "                           fit_intercept=True, intercept_scaling=1,\n",
       "                           loss='hinge', max_iter=1000, multi_class='ovr',\n",
       "                           penalty='l2', random_state=None, tol=0.0001,\n",
       "                           verbose=0))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.datasets import make_moons\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "m = make_moons()\n",
    "X, y = pd.DataFrame(m[0]), pd.DataFrame(m[1])\n",
    "Xy = X.copy()\n",
    "Xy['y'] = y\n",
    "poly_svm_clf = Pipeline([\n",
    "    ('poly_feature', PolynomialFeatures(degree=3)),\n",
    "    ('std_scale', StandardScaler()),\n",
    "    ('svm_clf', LinearSVC(C=10, loss='hinge')),\n",
    "])\n",
    "poly_svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sns.scatterplot(x=0, y=1, hue='y', data=Xy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Polynomial Kernel \n",
    "low polynomial degree can not deal with complex datasets, and high degree creates a huge set of features which makes model very slow. \n",
    "\n",
    "We can use `poly` kernel in svc instead of polynomial features. It is very faster and acctualy doesn't use all polynomial features but consider them. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.7/site-packages/sklearn/utils/validation.py:724: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('std_scaler',\n",
       "                 StandardScaler(copy=True, with_mean=True, with_std=True)),\n",
       "                ('svm_clf',\n",
       "                 SVC(C=5, cache_size=200, class_weight=None, coef0=1,\n",
       "                     decision_function_shape='ovr', degree=3,\n",
       "                     gamma='auto_deprecated', kernel='poly', max_iter=-1,\n",
       "                     probability=False, random_state=None, shrinking=True,\n",
       "                     tol=0.001, verbose=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import SVC\n",
    "poly_kernel_svm_clf = Pipeline([\n",
    "    ('std_scaler', StandardScaler()),\n",
    "    ('svm_clf', SVC(kernel='poly', degree=3, coef0=1, C=5))\n",
    "])\n",
    "poly_kernel_svm_clf.fit(X, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Adding Similarity Features \n",
    "We can change dimension of our data, so it is very easier to classify using linear methods.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVM Regression "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "X, y = pd.DataFrame(np.random.randn(100, 1)), np.random.randn(1, 100)[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LinearSVR(C=1.0, dual=True, epsilon=1.5, fit_intercept=True,\n",
       "          intercept_scaling=1.0, loss='epsilon_insensitive', max_iter=1000,\n",
       "          random_state=None, tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.svm import LinearSVR\n",
    "svm_reg = LinearSVR(epsilon=1.5)\n",
    "svm_reg.fit(X, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}