Lecture1-Assignment-Part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Movie Review Classification\n",
    "\n",
    "![This is getting exciting](https://i.kinja-img.com/gawker-media/image/upload/s--hIgTSFEs--/c_fit,fl_progressive,q_80,w_320/17j2zn73qxdlfgif.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using all that we have learned, we will now combine our techniques to perform some basic classifcation! We'll be using the nltk movie reviews data set, we will classify positive and negative reviews. Here's some code to get you started:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.corpus import movie_reviews as reviews\n",
    "\n",
    "X = [reviews.raw(fileid) for fileid in reviews.fileids()]\n",
    "y = [reviews.categories(fileid)[0] for fileid in reviews.fileids()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1 - Print a positive and negative review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A positive review:\n",
      "every now and then a movie comes along from a suspect studio , with every indication that it will be a stinker , and to everybody's surprise ( perhaps even the studio ) the film becomes a critical darling . \n",
      "mtv films' _election , a high school comedy starring matthew broderick and reese witherspoon , is a current example . \n",
      "did anybody know this film existed a week before it opened ? \n",
      "the plot is deceptively simple . \n",
      "george washington carver high school is having student elections . \n",
      "tracy flick ( reese witherspoon ) is an over-achiever with her hand raised at nearly every question , way , way , high . \n",
      "mr . \" m \" ( matthew broderick ) , sick of the megalomaniac student , encourages paul , a popular-but-slow jock to run . \n",
      "and paul's nihilistic sister jumps in the race as well , for personal reasons . \n",
      "the dark side of such sleeper success is that , because expectations were so low going in , the fact that this was quality stuff made the reviews even more enthusiastic than they have any right to be . \n",
      "you can't help going in with the baggage of glowing reviews , which is in contrast to the negative baggage that the reviewers were likely to have . \n",
      "_election , a good film , does not live up to its hype . \n",
      "what makes _election_ so disappointing is that it contains significant plot details lifted directly from _rushmore_ , released a few months earlier . \n",
      "the similarities are staggering : \n",
      "tracy flick ( _election_ ) is the president of an extraordinary number of clubs , and is involved with the school play . \n",
      "max fischer ( _rushmore_ ) is the president of an extraordinary number of clubs , and is involved with the school play . \n",
      "the most significant tension of _election_ is the potential relationship between a teacher and his student . \n",
      "the most significant tension of _rushmore_ is the potential relationship between a teacher and his student . \n",
      "tracy flick is from a single parent home , which has contributed to her drive . \n",
      "max fischer is from a single parent home , which has contributed to his drive . \n",
      "the male bumbling adult in _election_ ( matthew broderick ) pursues an extramarital affair , gets caught , and his whole life is ruined . \n",
      "he even gets a bee sting . \n",
      "the male bumbling adult in _rushmore_ ( bill murray ) pursues an extramarital affair , gets caught , and his whole life is ruined . \n",
      "he gets several bee stings . \n",
      "and so on . \n",
      "what happened ? \n",
      "how is it that an individual screenplay ( _rushmore_ ) and a novel ( _election_ ) contain so many significant plot points , and yet both films were probably not even aware of each other , made from two different studios , from a genre ( the high school geeks revenge movie ) that hadn't been fully formed yet ? \n",
      "even so , the strengths of _election_ rely upon its fantastic performances from broderick , witherspoon , and newcomer jessica campbell , as paul's anti-social sister , tammy . \n",
      "broderick here is playing the mr . rooney role from _ferris bueller_ , and he seems to be having the most fun he's had since then . \n",
      "witherspoon is a revelation . \n",
      "it's early in the year , it's a comedy , and teenagers have little clout , but for my money , witherspoon deserves an oscar nomination . \n",
      "and once campbell's character gets going , like in her fantastic speech in the gymnasium , then you're won over . \n",
      "one thing that's been bothering me since i've seen it . \n",
      "there is an extraordinary amount of sexuality in this film . \n",
      "i suppose that , coming from mtv films , i should expect no less . . . \n",
      "but the film starts off light and airy , like a sitcom . \n",
      "as the screws tighten , and the tensions mount , alexander payne decides to add elements that , frankly , distract from the story . \n",
      "it is bad enough that mr . m doesn't like tracy's determination to win at all costs , but did they have to throw in the student/teacher relationship ? \n",
      "even so , there's no logical reason why mr . m has an affair when he does . \n",
      "there's a lot to like in _election_ , but the plot similarities to _rushmore_ , and the tonal nosedive it takes as it gets explicitly sex-driven , mark this as a disappointment . \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('A positive review:')\n",
    "print(reviews.raw('pos/cv001_18431.txt'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A negative review:\n",
      "plot : two teen couples go to a church party , drink and then drive . \n",
      "they get into an accident . \n",
      "one of the guys dies , but his girlfriend continues to see him in her life , and has nightmares . \n",
      "what's the deal ? \n",
      "watch the movie and \" sorta \" find out . . . \n",
      "critique : a mind-fuck movie for the teen generation that touches on a very cool idea , but presents it in a very bad package . \n",
      "which is what makes this review an even harder one to write , since i generally applaud films which attempt to break the mold , mess with your head and such ( lost highway & memento ) , but there are good and bad ways of making all types of films , and these folks just didn't snag this one correctly . \n",
      "they seem to have taken this pretty neat concept , but executed it terribly . \n",
      "so what are the problems with the movie ? \n",
      "well , its main problem is that it's simply too jumbled . \n",
      "it starts off \" normal \" but then downshifts into this \" fantasy \" world in which you , as an audience member , have no idea what's going on . \n",
      "there are dreams , there are characters coming back from the dead , there are others who look like the dead , there are strange apparitions , there are disappearances , there are a looooot of chase scenes , there are tons of weird things that happen , and most of it is simply not explained . \n",
      "now i personally don't mind trying to unravel a film every now and then , but when all it does is give me the same clue over and over again , i get kind of fed up after a while , which is this film's biggest problem . \n",
      "it's obviously got this big secret to hide , but it seems to want to hide it completely until its final five minutes . \n",
      "and do they make things entertaining , thrilling or even engaging , in the meantime ? \n",
      "not really . \n",
      "the sad part is that the arrow and i both dig on flicks like this , so we actually figured most of it out by the half-way point , so all of the strangeness after that did start to make a little bit of sense , but it still didn't the make the film all that more entertaining . \n",
      "i guess the bottom line with movies like this is that you should always make sure that the audience is \" into it \" even before they are given the secret password to enter your world of understanding . \n",
      "i mean , showing melissa sagemiller running away from visions for about 20 minutes throughout the movie is just plain lazy ! ! \n",
      "okay , we get it . . . there \n",
      "are people chasing her and we don't know who they are . \n",
      "do we really need to see it over and over again ? \n",
      "how about giving us different scenes offering further insight into all of the strangeness going down in the movie ? \n",
      "apparently , the studio took this film away from its director and chopped it up themselves , and it shows . \n",
      "there might've been a pretty decent teen mind-fuck movie in here somewhere , but i guess \" the suits \" decided that turning it into a music video with little edge , would make more sense . \n",
      "the actors are pretty good for the most part , although wes bentley just seemed to be playing the exact same character that he did in american beauty , only in a new neighborhood . \n",
      "but my biggest kudos go out to sagemiller , who holds her own throughout the entire film , and actually has you feeling her character's unraveling . \n",
      "overall , the film doesn't stick because it doesn't entertain , it's confusing , it rarely excites and it feels pretty redundant for most of its runtime , despite a pretty cool ending and explanation to all of the craziness that came before it . \n",
      "oh , and by the way , this is not a horror or teen slasher flick . . . it's \n",
      "just packaged to look that way because someone is apparently assuming that the genre is still hot with the kids . \n",
      "it also wrapped production two years ago and has been sitting on the shelves ever since . \n",
      "whatever . . . skip \n",
      "it ! \n",
      "where's joblo coming from ? \n",
      "a nightmare of elm street 3 ( 7/10 ) - blair witch 2 ( 7/10 ) - the crow ( 9/10 ) - the crow : salvation ( 4/10 ) - lost highway ( 10/10 ) - memento ( 10/10 ) - the others ( 9/10 ) - stir of echoes ( 8/10 ) \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print('A negative review:')\n",
    "print(reviews.raw('neg/cv000_29416.txt'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "2 - Using the scikit train_test_split function (http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html), split the data into a training set and a test set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train_raw, X_test_raw, y_train_raw, y_test_raw = train_test_split(X, y, test_size = 0.33, random_state = 1992)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1340"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X_train_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "660"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(X_test_raw)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3 - Then lemmatize or stem the reviews, and transform the documents to tf-idf."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk import word_tokenize\n",
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.preprocessing import MultiLabelBinarizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def tokenize(text):\n",
    "    words = word_tokenize(text)\n",
    "    words = [PorterStemmer().stem(word) for word in words]\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vectorizer = TfidfVectorizer(tokenizer=tokenize)\n",
    "X_train = vectorizer.fit_transform(X_train_raw)\n",
    "X_test = vectorizer.transform(X_test_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1340, 27510)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(660, 27510)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 1,\n",
       " 1,\n",
       " 0,\n",
       " 0,\n",
       " 0,\n",
       " 1,\n",
       " 1]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "label_encoder = {'pos':1, 'neg':0}\n",
    "y_train = [label_encoder[label] for label in y_train_raw]\n",
    "y_test = [label_encoder[label] for label in y_test_raw]\n",
    "y_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4 - Finally, build a model. To start, use a logistic regression (which we will review in detail in the coming lectures) (http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#build logistic regression model\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "logreg = LogisticRegression()\n",
    "\n",
    "logreg.fit(X_train, y_train)\n",
    "y_pred = logreg.predict(X_test)\n",
    "y_pred_proba = logreg.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "5 - Measure the efficacy of your model using the Reciever Operator Characteristic (ROC) Area Under the Curve (AUC). Report this metric on the test set of your data.\n",
    "\n",
    "For more info on this, see: http://scikit-learn.org/stable/auto_examples/model_selection/plot_roc_crossval.html#sphx-glr-auto-examples-model-selection-plot-roc-crossval-py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#Load packages and calculate auc\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.metrics import roc_curve\n",
    "\n",
    "import matplotlib.pyplot as plt \n",
    "plt.rc(\"font\", size=14)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.78533040578405344"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logit_roc_auc = roc_auc_score(y_test, y_pred)\n",
    "logit_roc_auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/anaconda/lib/python3.6/site-packages/matplotlib/axes/_axes.py:545: UserWarning: No labelled objects found. Use label='...' kwarg on individual plots.\n",
      "  warnings.warn(\"No labelled objects found. \"\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZcAAAEiCAYAAAA1YZ/LAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XeYVPX1x/H3R0QsiIoYxAp2iYpKs6HYYo892FBiQUWN\nxlgTCxrLz967UWyIXVGxYAUbCooNuxQLoAjSpZ7fH+eODOvs7uwyfc/reebZnVvmnrkMc/bbZWaE\nEEIIubRYsQMIIYRQeSK5hBBCyLlILiGEEHIukksIIYSci+QSQggh5yK5hBBCyLlILiGnJPWQZJJa\nZ3m8Seqd16BKWF3vV6WS1FtSjIuoIJFcKljaF1fqMVfS95LulrRqAeM4RNIphbpeNiS9ltyTr6vZ\n3yXtvvUocHgFJ6lrlc/KPEk/SXpU0obFji+UH8UgysqVfCneDfQGvgGWBLYGDgdGAxuZ2YwcX7MR\n0BiYZcmHS9IzybVaZzh+SWCumc3NZRxZxPka0Bm/J1uY2ZAq+28Fjkj2/93M+uQpjj/cr2KQ1BV4\nFbgJeCeJaRPgOGA6/u83Lo/XXxxY3Mx+y9c1QmEtXuwAQkG8YGbvJL/fKWkicCqwN/BgLi9kZvOA\neXU4vphfJqPxWA8Bfk8ukhoDBwDPAvvnM4C63q8CeMPM+qWeSPoCuAX/g+TyfF00+eOioH9ghPyK\narGG6ZXkZ5vUBkmtJT0k6RdJMyW9J2mfqidK6iXpY0nTJU2WNFzSsWn7F2pDSEoIewBrple7pB3/\ne5uLpA7J86MyXHebqlVUklpJulPSOEmzJH0m6fg63ou+QLekBJGyK9A82fcHktpJGiBpSnIfXpPU\nJW1/1u+jujYXSR2Ta0xO/j3ekLR92v6Nk/P2T9u2frLtqyqvdZ+k0XW4J+kGJz/XzvBeaovxgCSe\nHTOce1iyr2vyPGObi6S/SHpd0rTk8bykTdP275W8Tvu0bbsk2wZWea3Bkl5Pe755Ev/Pkn6TNDq5\nV0vV7RaFTCK5NEypL4pfACT9CXgL2B24FTgbEPC4pINTJyVfljcBn+Mln/8AbwNb1XCti4HhwASg\ne9rjD8xsKPA10C3D7m7ALOCJtJjfwRPBzcDJwCfAzZLOqenNV9EXaAmkfwEeArwJjKl6cNL+MBjY\nDLgCr3JcBXhJ0rZ1fR+ZSNouuUZz4ELgTKAJ8GLqyxh/r5OAbdNO3RaYD6wjqVXa9i7AoOquV4vW\nyc9J9YjxWWAq1d+HH2uKS9IhwPPAb/hnsjewFjBY0gbJYW8CRub7sGVS3YakJkDH1PUkrQQMxP8v\nXA6cCPQBNgKWqS6mUAdmFo8KfQA98P94uwAtgNXw/9QTgBnAqslxVyfHdU07dylgBDAWaJxsewL4\nJMtrtk7b9gwwqprjDeid9vwivHpkpbRtiyVxPJm27XZgXPpxyfY7kve2fC1xvgZ8nvz+NnBP8ntT\nvI3hOKBDEl+PtPMeB2YD66Zta5Hc06H1eB8L3S88qX8OvETSJppsXwL4FHgrbdvTwAdpz+9N7vU0\noFuybfXk9Y+p5X50TR2XvJ9WyefmK/yLumPasXWJ8f7k3iyetm15PMFem7atN2Bpz5cBJgJ3VYlz\nBeAnoG/ato+BJ9KeDwIeSt5P52Rbl+T5zsnzvZPnHYr9/7RSH1FyaRieB34GvgP6AeOBvczsh2T/\nHsD7ZvZa6gQzm4mXCFYGNk82TwZWk9Qxj7H2AxqxcFvHdkkc/QAkiQVtIiapReoBvIgnxs51uGZf\nYN+kOmQfvDH7kaoHJVVnuwBPm9nvVU9mNgH/q7e9pJbZvo9qtAPWT2JaMe19NcP/0u4saenk2MHA\nJpKWS55vi1d5vsOCv+S7pB2bjdvxz8qP+OemGdDdzN6rZ4z9gBWBndLO3xdPRDXdh53xRNK3yr9v\no+S9bJ927GBgG7kmQCc8uXzDwvdhLv6HBPhnGWBPeRtbyLFILg3DP/D/rF2ANfGePy+n7V8T+CLD\neZ8lP1snPy/DqznelfSNpFvT69hzwcw+wf/6PSht80F4aeTp5PlK+BfPkfgXYfrj4eSYP9Xhsg8B\nSwN7AYfiHSB+yXDcSslxtd6rLN9HJuslP//HH9/byfj/2RWTYwYnz7eRtDr+7zgoeaR/qf5kZp/X\ncM10F+OflX3whLkcMGcRYnwBr1Kreh9G2YJOJpmkrjEwwzX2Y+F/38F4aastnliaJNuq3ocPzGxa\n8vx14FHgfOAXSU9LOkZSVInlSPQWaxjeq+U/clbM7DNJ6+NtM7sAewLHSrrZzE5Y1NdP0w+4IGk3\nSH2ZPG1m05P9qT+KHgTuquY1Ps32Ymb2k6SXgZOALaimTageansfmaTe21nAsGqO+Tn5ORSYiX+B\nLo8n/g+AZYHekprjX6pv1CHmT8zspeT3p+Rdxf8n6W0z+66uMZrZHEmPAwdIWgIv3ewAXFVLHKlr\n9AB+qOE4WNBusy3eBvSZmf0saTBwVdLushVwZ+oE87qxAyV1wj/HO+OltrMlbWFmP9VyzVCLSC4B\nvEvu+hm2pxpNR6U2mI+LeRR4NPlP2wfoJemStGq2quo6fqMf8F/gQLyU0IKFq1B+xr9IF0/7IlxU\nffH3Mg3oX80xP+Mlj6zuFbW/j0y+SX5Ore29JV/cqSqw5fC2jnnJtrl4u0JbvB2qvs7GqyD/g7dD\n1SnGRD/gKLzzRSv8eyfb+/BzFvfhB0kj8fuwAguSzaDk+WF4UvtD5wEzexd4FzhP0m7AALzd6eLa\n31aoSVSLBfBG4M21cHfaJYHj8UbzYcm2FdNPMh+b8HHydPkaXn86sHzSVlIrM/s6uWa35DEZeC5t\n/zw8we0jqV3V85OeQHX1OHAB0MuqGViaXPd5YC9Jv3fNTUoIR+AN+uOzfR/VGIb3NDtV0rJVd2Z4\nb4OB9vhf3oOS687ESzVn4o3v9e0phpmNwtufeqT1QKtrjK/i7Xyp+/C5mQ2v5dIvAL8C/05KPLVd\nYzDeprUVC+7DN3jb0Vn4HziD085fIcPn8f3kZ02f5ZClKLkE8LaUg4FnJV2P/4V+GP5X76G2YPT8\ni5J+wqtZxgHr4FVJH7GgzSGTofiXyrWShgDzLW2gXjX64V1ENwEeM7NZVfafhfdwelvSHXg12ArA\npniD8ZK1vel0ZjYV77FUm3OAvwBvSLoJ7yZ7DP6FdEA93kfVOOYnXb6fB0ZIugv4Hu/uvB2eLKo2\nZp+Hd9FNTyKD8OQyBfgwi/dVkyvxz8c/gTPqGmNSmnoU+Dv+7/Lf2i5oZlMkHQc8AHwg6UE8Qa2B\nl4A+xavMUgbjAz1h4fswGP/sfWpmE9O2HwGcIOkJvJS0VBJf6g+XsKiK3V0tHvl7sKCb6xZZHNsG\nbwyfiH9hDgX2rXJMT7wL7894V9KRwPXAnzJcs3XatqWBe/BxNfNZuMvpQl2R07avnjoW2LWamFdK\nrj8a7x48Dv8r+YQs3u9rJF2RazjmD12Rk+3t8OqTqXip7HVg22peo8b3kel+Jds3wUsMqXs9Gv/S\n27XKccvgDe4zgSXStu+RvO6ALD8rXZPjD6pm/yt4olq+rjEmx26TvL4BG2TY3zv9c5G2vQte2puU\nvMdv8C7XW1Y5bv3ktb+tsv2EZPvNVbZvhieuUcnn/Se8tJTx3zEedX/E3GIhhBByLtpcQggh5Fwk\nlxBCCDkXySWEEELORXIJIYSQcxXbFblFixbWunXrYocRQghlZdiwYRPMrD5jxRZSscmldevWDB06\ntNhhhBBCWVH91/5ZSFSLhRBCyLlILiGEEHIukksIIYSci+QSQggh5yK5hBBCyLmCJhdJ20rqL+kH\nSSapRxbnbCzpdUkzk/POy3bq9hBCCMVR6JJLU+ATfCnUmbUdLCm1Jvd4oGNy3unAqXmMMYQQwiIq\n6DgXMxuAT1WOpD5ZnHIoPl37EeYLIH0iaQN8kaKrLaZ0DiGEklTqgyi3BAYniSXlBXyxodb4eiIh\nhFDR+g4Zw1PDq1tFPDdWmvAjhz9yfc5er9STy8r4CnfpxqftWyi5SOqJL2jFGmuskffgQgghV2pK\nIENG+iKands0z/l1G82dw14vPch+z97N/MUa5ex1Sz251ImZ3Q7cDtChQ4eoMgshFEV9Sho1JZDO\nbZqz96arckjnPPzRfMMN8OStsN9+cN11sPrqOXnZUk8u44CWVba1TNsXQggl56nhPzBi7BTatmqW\n9Tl5TSBV/fILjBoF7dtDz56w3nqwyy45vUSpJ5e3gcskLWlmvyXbdgZ+xNe+DiGEhRSifaI2qcTy\n0LFbFjWOPzCDe+6B006D5ZaDL7+EJk1ynligwMlFUlNgneTpYsAakjYFJprZGEmXAp3MbMfkmL7A\n+UAfSRcB6wFnARdET7EQKlt9k0Q+2yey1bZVM/bedNWiXT+jESPg+ONh0CDYemu45RZolLs2lqoK\nXXLpALya9vyC5HEP0ANoBayd2mlmkyXtDNwEDAUmAVcBVxco3hBCDuS6DaImBa1eKhfDh0PHjtCs\nGdx5J/z977BYfoc5qlILAB06dLBYzyU0FKVQFVST+iaKSBKL6PvvYbXVvDrs8svhyCNhpZrXAZM0\nzMw6LOqlS73NJYSQQdVkUgpVQTWJ0kSB/fADnHIKPP88fP45rLoqnHlmQUOI5BJCmek7ZAz/fuJj\nYEEyiS/vAMDcuXDzzXDOOTBnjv9s0aIooURyCaHMpEosl+y7cSSTsMDMmdClCwwb5r2/broJ1l67\n9vPyJJJLCGUiVRU2YuwUOrdpHokluDlzoHFjWGop2HFHOOMMOPBAKPLk8ZFcQihh6W0r6e0qJdfN\nNRSeGTzyCPzrX/DUU7D55nDZZcWO6neRXEIoMdUllGhXCb/75hs48URvsN9887yOV6mvSC4hFFlN\nPb8ioYQ/uPJKOPdcrwq77jo44YRILiE0NNmMP6najTgSSqjRjBmw555w7bXexbhERXIJIUcyJZJs\nxp9EMgk1+vlnOP10n7X4r3/17sV5Hl2fC5FcQshCfUogqd8jcYR6mT8f7r7be39NnQrt2vn2Mkgs\nEMklhFplGrSYSSSSkDOffgrHHQdvvOFjV269Fdq2LXZUdRLJJTR4tZVKUiWSGLQYCmbYMPjsMy+5\nHHFE0ces1Eckl9BgVJdEamsXiRJJKIhnn4WJE6F7d3/stRessEKxo6q3SC6hYmU7uWMkj1BU338P\nJ58Mjz8OW2wBhx3mJZUyTiwQySVUiGx6akUSCSVl7lxfv/6882DePLj0Ujj11LKsAsskkksoa6mk\nEj21QtkZOtSTyW67+SSTbdoUO6KciuQSylKmpBKJJJS8X3+Fl1+G/ff3KrB334UOHSqmtJIukkso\nOXUdUxJJJZQ8M+jXD/75T5g0CUaNglatfOnhChXJJZSc1LTybVs1q/aYSCqhbHz1lc//NXCgl1IG\nDPDEUuEiuYSS0nfIGIaMnEjnNs156Ngtix1OCItmyhQvnZjBjTf6wMgSnGQyHyK5hJKRPhI+1isJ\nZe3DD326lmbN4K67YMstG0RpJV0kl1B0VRvnYyR8KFvjx8Npp8H993v11267+YSTDVAkl1AUNa2w\nGIkllJ358+GOO+Css2D6dJ+5uGvXYkdVVJFcQlGkN9pHUgllb999oX9/Tyi33AIbbFDsiIoukkvI\nu0xdi1OJJRrtQ9maPh2WXNIb6A891MeudO9ekWNW6qM8FgYIZS1VSknXtlWzaLQP5at/f58C/6ab\n/Pnf/gaHHx6JJU2UXEJepJdWopQSKsaYMfCPf8BTT8FGG/m4lZBRJJeQU5mmZYlSSqgI994Lxx/v\nY1Yuu8xH2zduXOyoSlYkl1Bvtc1EHI30oSKYeXXXGmvAjjvC9ddD69bFjqrkZZ1cJDUGdgHWBu42\nsymSVgcmm9mUms8OlaK6LsQpkVRCxZg0ybsWN20KV13lPcEaePfiusgquUhqDQwEWgJLA08DU4B/\nAUsBx2Z7QUm9gNOBVsCnwClmNriG43cBegMbAbOAN4HTzezLbK8ZFl2m6q5IJKEimcEDD8C//gW/\n/OLT4qdKLyFr2ZZcrsO/1I8BJqZtfwL4X7YXk9Qtea1ewBvJz+cktTWzMRmObwM8BVwPdAeaApcB\nA4B1sr1uyF42SwFHQgkV69tv4Zhj4JVXoFMneOEF2HTTYkdVlrJNLtsAW5nZHC2cvUcDq9TheqcC\nfczsjuT5SZJ2BY4Hzs5wfHugMXC2mc0DkPR/wCuSWpjZhDpcO2QQSwGHkGbOHPj0Ux8IecwxDWaS\nyXzINrksBmS6y6sBU7N5AUlL4Mniyiq7XgS2qua094A5wNGS7sSr5HoA70ViWXTpE0XGUsChwRo4\n0OcBu+YaWH99GD0amjQpdlRlL9vkMhA4CS9hAJikZYDzgeezfI0WeIIaX2X7eGCnTCeY2WhJOwOP\nADfhSe4DYLdMx0vqCfQEWGON+HKsSXpiiYkiQ4M0bpy3pzz4IKy7Lpx7LjRvHoklR7IdoX8asIuk\nj4AlgXuBb4E2wJl5ig1JK+NtOvcBHYGueEnpYUl/iN3MbjezDmbWYaWVVspXWGWt75AxdLvt7Ugs\noeGaN2/B/F+PPQbnnw8ffeSJJeRMViUXMxsjaRO8Ub09npQeAu4xs6yqxYAJwDy8x1m6lsC4as45\nAZhuZqenNkg6DPgOr0p7I8trN3ix5nwIicmT4bzzoH17TzLrrVfsiCpStl2ROwHDzOyWKtsbSepk\nZu/W9hpmNlvSMCBVzZWyM/BYNactjSekdKnnMS9alqq2rURSCQ3O1Klw220+qr55c3j3XR8IGd2L\n8ybbNpe38XEpP1XZvnyyL9suFVcD90l6F+/afBze2+xWAEmXAp3MbMfk+GeBf0o6D3gQWBa4BC+5\nDMvymg1eqjdYVIGFBscMnnzS5wP7/nvYfHPYYQdo06bYkVW8bJOLAMuwfQVgRrYXM7OHJK0InIMn\nq0+A3c1sdHJIK3wGgNTxr0g6BDgjecwA3gF2NbPp2V63IaluevvObZpHYgkNy+jRcOKJ8MwzsMkm\n8PDDvtxwKIgak4ukh5NfDbhT0qy03Y2AdviXfdbM7Gbg5mr29ciwrR/Qry7XaKgydS2GmN4+NEBm\nvr7K55/DlVfCySfD4jGVYiHVdrdT7RsC5rNw+8dM4AHglqonheKI6q/Q4L31lpdSmjaFO+/09pUY\nllAUNSYXMzsYQNIo4KKoiipNqaqwqP4KDdYvv/gkk3fe6V2Le/eOaVuKLNuuyJmmZglFVl334hAa\nDDNfZ+W003wW49NO80courpMuX8wcDCwBrBE+j4za5vjuEINYsxKCImzzoLLL/eG+ltv9SqxUBKy\nHedyCnABcBfwF+BOYF2gMz5jccijmiaXjKQSGpyZM2HGDFhxRTjqKFh7bTj6aFgshr6VkmxLLscD\nPZOuxEcDV5vZt5IuBGKelTyKySVDSPPCC9Crl7enPPaYj66PEfYlKdvksjoLuhzPxAczgs/59Q4L\nJrQMORY9wEIAxo710fUPPeTJ5MQTix1RqEW25cjxQGrgxBigU/L7mng35ZBH0QMsNGgvv+yTTD75\nJFx4oU8yuf32xY4q1CLb5PIqsGfy+z3AtZKeAx7GV4oMIYTcmjPHf7ZrB7vsAh9/7NPix5T4ZSHb\narHjUsea2Q2SpgBbAy8DN+QpthBCQzRliieRIUPgzTehRQufuiWUlWzHucwGZqc9vwcvwYQcqW7t\n+hFjp9C2VbMiRBRCgZl5I/3JJ3sby/HHw6xZsPTSxY4s1MMiTbYjaU/gQjPbPEfxNCjpCaW6tetj\nXrDQIPz8MxxxBDz3nPcEe+IJ6NSp9vNCyao1uUjqjo9tmQPcaGbvS9oCuBbYjJhUsk6qSyjRvTg0\naM2aeYK55hrvCRaTTJa92mZFPhm4EvgCnwr/QEnnAOcDtwH7mdmPeY+yQlQdsxIJJTRogwfDxRfD\no4/6RJNDhsRAyApS258HPYETzew2STsDLwD7AeuZ2YS8R1fGMrWhpEoqMWYlNGgTJsAZZ8Ddd8Oa\na8LIkbDxxpFYKkxtyaU18DyAmQ2UNBc4KxJLZrW1oURJJTRoZtCnD5x+uq9jf+aZvpZ9NNhXpNqS\ny1L4iPyUWfiAylBFVHmFkIX77vMBkbfeChttVOxoQh5l02rWQ9K0tOMPk7RQySVZXbJBqW4yyajy\nCiHNjBlw6aXQsyesvrp3NV5uuagCawBqSy4/Af9Me/4rf5xHzKhm2eJKllqcKzUGJUoqIVQxYACc\ncAKMGgWtWvmEkyusUOyoQoHUthLlyoUKpBy1bdWMh47dsthhhFBafvgBTjnFe4FtuCG89hpst12x\nowoFFmXTEEJuXXQRPPOMdzMePjwSSwMVI5VCCItu6FBYYglfCfKii7xH2FprFTuqUERRcgkh1N/k\nyXDSST5Vy3/+49tWXDESS4jkEkKoBzNfuGuDDeCmm3zKlvvvL3ZUoYREtVgIoe4eeAC6d4fNN4en\nn4YOHYodUSgxWScXSY2BXfA5xu42symSVgcmm9mUfAUYQigRs2b5VC0bbAAHHgizZ/tMxo0aFTuy\nUIKyqhaT1BoYAfQFrgJaJLv+BVyRj8BKWd8hY34fNBlCg/D66z4V/l/+Ar/95qtBHnlkJJZQrWzb\nXK4D3gRWZOHpYJ4Adsx1UKUsfZqXWGclVLyff4YePaBrVy+53HYbLLlksaMKZSDbarFtgK3MbI6k\n9O2jgVVyHlUJS035EtO8hIr3zTfQsSNMmwb//rf3BotJJkOWsk0uiwGZyr+rAVNzF05pS1WHdW7T\nPBJLqFxTpvjiXWutBUcdBX//O7RtW+yoQpnJtlpsIHBS2nOTtAy+aNjzOY+qRKVKLVEdFirS9Ok+\nDX7r1j6FiwRXXBGJJdRLtiWX04DXJH0ELAncC6yHl1q61+WCknoBpwOtgE+BU8xscA3HCzgZOA5o\nA0wE7jGzs+py3fpKn/14xNgpUWoJlemZZ3ysyujR3lAf7SphEWWVXMxsjKRNgMOBzfESz0P4l3zW\n1WKSuuGdA3oBbyQ/n5PU1szGVHPaVcCeeEL6GFgOT0x5V3WNlratmkWpJVSWuXOhWzd4/HEvoQwa\nBF26FDuqUAFkZrUfJC1nZpMX+WLSEOAjMzsmbdtXwKNmdnaG49cHPgE2MbPP6nKtDh062NChQ+sV\nZ6q0Emu0hIpl5tVe4CWW1VaDU0/1+cFCgyZpmJkt8qjYbNtcxkl6VNLeyWDKOpO0BNAeeLHKrheB\nrao5bW/gW2BXSd9KGiXpHkl/qk8M2Uqt1dK5TfNILKHyDBnivcDef9+f33gjnHVWJJaQU9kml27A\nXHwQ5ThJt0rauo7XaoH3OKu6TPJ4oLp1Y9YC1gQOAnrg7TsbAE9L+kPsknpKGipp6M8//1zH8Fyq\nR1hqrZZILKFi/PqrL9i15ZYwbhxMmlTsiEIFyyq5mFl/MzsIaAmcijesv5aUJv6b5/iaAN3NbFDS\n8N8d6AR0zBDn7WbWwcw6rLTSSnW+WAyQDBXr4Yd92pbbboOTT4bPPoMdG9T451BgdZoV2cymmdk9\nZrYL0A6YDPw7y9MnAPPwBJWuJTCumnPGAnPN7Mu0bV8lr5PzIkUMkAwV6/PPfQ37996Da66BZZct\ndkShwtUpuUhqIukASU8A7+PTwVyZzblmNhsYBuxcZdfOwFvVnPYmsLiktdO2rYVXr42uS+zZiq7G\noSLMmgUXXugzFgOcfTa8847PYhxCAWQ7ceWOku7C20fuxMea7AasaWZn1uF6VwM9JB0taUNJ1+HT\nx9yaXOdSSS+nHf8SnsTukrSZpM2Au4AhQP26goVQ6V55xVeEPP98ePVV39a4cUwyGQoq20GUA/CR\n+McA/c1sVn0uZmYPSVoROAcfq/IJsLuZpUohrfAp/VPHz5e0J3A9MAifNHMgcKqZza9PDCFUrPHj\n4bTTfNGutdeG55+HXXYpdlShgco2ubQys5zMMW9mNwM3V7OvR4ZtY4EDc3HtECraiy/66pDnnuvV\nYEstVeyIQgNWbXKRtLSZzUie/iap2ulQ044LIRTShx/CV1/BAQfAYYfBNttAmzbFjiqEGttcpqYN\nVpyGzyNW3SOEUEjTpnkVWPv2Ptnk3Lk+4j4SSygRNVWL7Y433Kd+r32emBBC/j31FJx0Enz3HRxz\nDPzf/8HiWa9YHkJBVPuJNLMX0n5vMNPqh1DSPv4Y9tkHNt4YHnwQtq7rRBkhFEa2XZFnSPrDkHdJ\nzSVVRHtLatqXEErOnDkLuhRvvDE8+ywMGxaJJZS0bAdRLgmomu11GohZqmIhsFCS3n4bOnSAnXby\nhnuA3Xf3cSshlLAaK2qThb3A21t6SJqWtrsRsB3w5R9OLDOxfHEoOZMm+UzFt9/u0+E/9hiss06x\nowoha7W1Ap6b/BTwLyB94OJsYBS+4FdZi1JLKCmzZkG7dvDjj77GygUXQNOmxY4qhDqpMbmYWSsA\nSW/jI+krdo7uKLWEovv+ey+lNGni84Jtuqk/QihD2U65v2UlJpa+Q8bQ7ba3GTF2SrFDCQ3Zb7/5\nPGBrrw39+/u2Hj0isYSyVtMI/cuBC8xsevJ7tczsjJxHVgCpFSfbtmoWVWKhOAYO9AW8vv4aDjkE\nOnUqdkQh5ERN1WJdgMZpv1enrAdXplacDKHg/vEPuOEGWHddTzI77VTsiELImZoGUW6Z6fdKkd5D\nLISCmTfPfzZqBFtsAc2be6+wJZcsblwh5Fi9x6hIWk1S2c45ET3EQsF98AFstRXcdJM/P+QQ6N07\nEkuoSNmO0O8t6bC0588AY4BxkjrkK7h8ix5ioSCmTvUuxR06wKhR0KpVsSMKIe+yLbn0AL4BkLQL\nsCXQFXgE+L98BBZCRRg4EDbcEK69Fnr29LXsD4zliULly7Zaa2Xg++T33YFHzGyQpLHAu3mJLE/6\nDhmzUC+xEPJqiSWgRQt49FFvYwmhgcg2uUwEVgO+A3Zhwcj9xfBpYMpGdD8OeTVnDlx9NUyZAhdf\nDNttB++/D4tVxBR8IWQt2+TyJHC/pM+APwGpKfjbkVSXlZPofhzy4o034Ljj4NNPYf/9Yf58TyqR\nWEIDlO3gNZ3qAAAaaUlEQVSn/hTgLuAHYFczS60+uSZwez4CC6FsTJwIRx8NXbp44/1TT3k1WCSV\n0IBlVXIxs9nAxRm2X5HziEIoNxMmQL9+cPrpPo3LMssUO6IQii7rcSqSmgPHAW3xUfmfArebWayw\nFRqezz6DRx6B886D9daDMWN8QGQIAch+nEtnvG3lOKAJvkhYL+BrSR3zF15uxWqTYZHNnAnnnONT\n4l97rc9kDJFYQqgi20rhq/BG/bXM7EAzOxBYC+gPXJOv4HItRuWHRfL887DRRt4L7KCDfMzKaqsV\nO6oQSlK21WLtgaPNbG5qg5nNTWZLHpqXyPIkRuWHepk2Dbp3hxVXhFdege23L3ZEIZS0bEsuU4HV\nM2xfLdkXQuWZNw8eeMB/Nm0KL70EH34YiSWELGSbXB4G/idpf0mtkscBwB3JvhAqy/vv+4j6ww6D\nJ57wbe3a+SqRIYRaZZtcTgOeA/rh08B8DzwIDABOz09ouRWN+SErU6bAySdDx47w3Xfw4IM+IDKE\nUCfZjnP5DThW0pnAusnmr8zs17xFlmPRmB+yst9+3qZy/PHecL/88sWOKISyVGtykbQKsAPeBfl1\nM3sv71HlSTTmh4xGjvTJJZdd1hOKFMsNh7CIaqwWk7QVMAK4F29f+SRpa6k3Sb0kjZT0m6Rhkmpa\nQjn9vHUlTZU0bVGuH8LvZs+GSy+Ftm3hv//1bZ07R2IJIQdqa3O5CHgHWBvvGdYXuLK+F5PUDbgO\nuATYDHgLeE5SjcUJSUvg7T2D6nvtEBYyaBBsuin8+9+wxx6+nn0IIWdqSy7tgDPMbKSZ/QicDKwu\nqb4V0acCfczsDjP7zMxOAsYCx9dy3mXAR/jiZCEsmmuv9anwZ86EZ57xSSZjMGQIOVVbm8sKwLjU\nEzObKmlGsr1OjflJ6aM9fyz5vAhsVcN5ewB74iWdRaqSCw3Y/Pkwfbq3q+y+O/z0k0/jsvTSxY4s\nhIqUTW+x9SS1SHsuYF1JS6U2mNmILF6nBb6w2Pgq28cDO2U6IelMcAewr5lNk1TjBST1BHoCrLFG\nNNyHxKefeu+vFi3g8cd9oslLLil2VCFUtGySy+tVngtfLMyS3438rUZ5H3CLmQ3J5mAzu51kfZkO\nHTpYnmIK5WLGDG+ov/JKaNYMrrgCzLw3WAghr2pLLhvm8FoTgHlAyyrbW5JW9VbFDsB2ks5PngtY\nTNJcoFeSTEL4ow8+8DEro0ZBjx6eWFq0qO2sEEKO1JhczOyLXF3IzGZLGgbszMIN8zsDj1Vz2sZV\nnu8N/AfohK+KGcLCUiWT1Vf3R58+3ngfQiiorBcLy5GrgfskvQu8ia8PswpwK4CkS4FOZrYjgJl9\nkn6ypA7A/KrbQ2DuXLjpJl9ieOBAL6UMip7rIRRLQRf5NrOHgFOAc4DhwDbA7mY2OjmkFT6mJoTs\nvfeeD3485RSfWHLy5GJHFEKDV9DkAmBmN5tZazNrYmbtzWxQ2r4eZta6hnP7mFnTulyv75AxdLvt\nbUaMnbIIUYeSNH06nHiiJ5axY+Hhh2HAgFgVMoQSUPDkUmhPDf+BEWOn0LZVs5i0stIsvji8+qon\nmM8/hwMPjJ5gIZSIOrW5SGqKV1uNMLM5+Qkp99q2asZDx25Z7DBCLnzzDVx4obevNG0Kw4bBkksW\nO6oQQhVZlVwkLSPpXmAKMIxkVUpJN0r6Tx7jC8HNmgUXXQR//rMv3vXBB749EksIJSnbarFLgfXx\naVp+S9v+InBgroMKYSGvvuqrQJ57Lvz1r14F1iWrybRDCEWSbbXY3sDfzGyIpPSR7yOAtXIfVggJ\nM19jZfZsb6zfbbdiRxRCyEK2yWUl4KcM25fJYSwhuPnz4a67YNddfbbi++6D5ZaLSSZDKCPZVosN\nA3ZPe54qvRwJvJ3TiELD9vHHXuV1zDFwxx2+rVWrSCwhlJlsSy7/AQZI2iA55wRJfwa6AjG3Rlh0\n06d7L7Crr/ZSSp8+cPjhxY4qhFBPWZVckoGO2wF/wuf02g+YDmxtZu/mL7zQYPTuDZdf7gnliy/g\niCNizEoIZSzrcS5mNgzolsdYQkPz3XdeYtlgAzjzTO8JFr3AQqgI2Y5zWbqmR76DDBVm7lyv/tpw\nQzj2WN/WokUklhAqSLYll2ksaMTPJF+LhYVKM2SIJ5QPP4Q99oAbbyx2RCGEPMg2uVQdXNAYX9P+\naODcnEYUKteAAbDnnrDKKvDYY7DvvtGuEkKFyiq5mNkLGTY/I+lL4DDg3pxGFSqHGfz4I6y6Kuy4\no/cIO/lkWHbZYkcWQsijRZ0VeSi+FHEIf/TVV/CXv8AWW8C0ab7WyjnnRGIJoQGod3KRtARwArHc\ncKhq1iy44ALYeGN49104+2xYaqliRxVCKKCsqsUk/czCDfoClgdmAzHSLSwwbpyvWf/ll3DQQd4r\nrFWrYkcVQiiwbBv0z6nyfD7wM/CWmWWac6wk9B0yhiEjJ9K5TaxMmHdz5kDjxtCyJWyzDdxwg1eJ\nhRAapFqTi6TFgTnAADMbl/+Qcuep4V5jFytQ5tH8+T4H2MUXw1tv+UST//tfsaMKIRRZrW0uZjYX\nuBFokv9wcq9zm+Yc0nmNYodRmT78ELbeGo47DtZZx0svIYRA9g367wLt8hlIKCNmcPrp0L69Lzt8\n333w8svQpk2xIwshlIhs21xuBK6StAo+/f709J1mNiLXgYUSJsHEiXDUUXDppdA82rRCCAvLNrk8\nnPy8OfmZ6jmm5PeY/qXSjR4Np5ziSw1vvrm3syy2qMOkQgiVKtvksmFeowila84cuPZanxIf4MAD\nPblEYgkh1KDG5CLpLuBkM/uiQPGEUvLWW95Y//HHsNde3r14zTWLHVUIoQzU9ufnEUAMrW6oXnoJ\nfv0VnnwS+vePxBJCyFptySWmrG1IzOD+++G55/z5mWfCiBGw997FjSuEUHayqTivaR2XkjVx+myG\njJxY7DDKxxdf+KzF3bvDXXf5tiZNoGnT4sYVQihL2SSXcZLm1fTIe5T18OsMH9AXo/Nr8dtvcP75\nsMkm8MEHcOut8NBDxY4qhFDmsukt1hP4Nd+B5EOMzs9C//6+xsqhh8JVV/ncYCGEsIiySS5Pl/Lk\nlKEexo2D4cNh1129a/Gaa0LnzsWOKoRQQWqrFst5e4ukXpJGSvpN0jBJXWo4tqukpySNlTRD0keS\njsx1TA3GvHlw882w/vretjJzpo+2j8QSQsixgvYWk9QNuA64BNgMeAt4TlJ1dVdbAR8DBwAbAbcA\nt0s6JJdxNQgffABbbQUnnAAdO8Kbb8YCXiGEvKmxWszMcj0M+1Sgj5ndkTw/SdKuwPHA2Rmuf0mV\nTbdI2h7YH+ib49gq16hR0KmTzwH2wANw8MFeYgkhhDwp2BweybLI7YEXq+x6ES+hZKsZMKmaa/SU\nNFTS0DkNffp3Mx9ZD9C6tXcv/vxzOOSQSCwhhLwr5ARRLfAJLsdX2T4eWDmbF5C0J7AjcHum/WZ2\nu5l1MLMOjRs3XpRYy9uoUT5dy6abwkcf+bbu3WGFFYoaVgih4Sib2QclbY1Xhf3DzN4tdjwlac4c\nuOwyaNsWXnsNrrzSfw8hhALLdlbkXJgAzAOqDqRoCdS4fLKkbYABwHlmdkt+witz8+bBllvCsGGw\nzz5w/fWw+urFjiqE0EAVrORiZrPxhcZ2rrJrZ7zXWEaStgWeA3qb2bX5i7BMTZniPxs1giOPhKee\ngieeiMQSQiiqQleLXQ30kHS0pA0lXQesAtwKIOlSSS+nDpbUFU8stwJ9Ja2cPFYqcNylxwz69IG1\n1vJR9gC9esFf/1rUsEIIAQpbLYaZPSRpReAcoBXwCbC7mY1ODmkFrJ12Sg9gaeC05JEyGmid73hL\n1mef+Torgwb52JW11ip2RCGEsJCCN+ib2c1m1trMmphZezMblLavh5m1rvJcGR6tM712g3DlldCu\nnXczvuMOGDwYNtqo2FGFEMJCClpyCYvAzMentGzpgyCvvBJWitrBEEJpKpuuyA3Wjz9Ct27e+wt8\nvMo990RiCSGUtEgupWrePF+zfoMNvAfY3LnFjiiEELIW1WKlaPhwOPpoH7Oy884+k/E66xQ7qhBC\nyFokl1I0ebJXh/XrB3/7W8wFFkIoO5FcSoEZPPaYr2P/n//AdtvBt9/CkksWO7IQQqiXaHMptm+/\nhT328BUhn3wSZs/27ZFYQghlLJJLscyeDZdcAn/+s49VufZaePttWGKJYkcWQgiLrGKrxabPLvHe\nVWPGwAUX+NT4110Hq65a7IhCCCFnKrrksvemJfaFPWGCdy8G7/01YgQ8+mgklhBCxanY5LLMEotz\nSOc1ih2Gmz/fV4Jcf3049VRfERJg7bVrPi+EEMpUxSaXkvHpp97766ijfOGu4cN9YGQIIVSwim1z\nKQmzZ/sgyNmzveRyxBGwWOTzEELli+SSD6++Cttu6z2/Hn7YSyotWhQ7qhBCKJj4MzqXvv8eDjgA\ndtjBJ5cE2GabSCwhhAYnkksuzJ3r3Yk33BCefdbHrxx2WLGjCiGEoolqsVw4/HB48EHYdVe46aZY\nGTKE0OBFcqmvyZOhUSNo2tTXrt93X68Si0kmQwghqsXqzMxnK95gAzjvPN+2zTY+N1gklhBCACK5\n1M3XX3vV18EHw2qrwaGHFjuiEEIoSZFcstWvH2y0kU8uecMN8M470L59saMKIYSSFG0utZkzBxo3\nhs03h/33hyuugFVWKXZUIYRQ0qLkUp2ffvJeYN26+fP11oMHHojEEkIIWYjkUtX8+XDHHd5g36+f\nr7cyb16xowohhLIS1WLpRo70wY9vveWTTd5yiw+MDCGEUCeRXNI1awaTJkGfPl4lFl2LQwihXqJa\n7OmnvaF+3jxYcUX45BOfvTgSSwgh1FvDTS7ffeej6v/6V/jiCxg71rfHlPghhLDIGt436dy5cPXV\n3pbywgtw2WXwwQc+KDKEEEJONLw2l3nzvDdY165w443QunWxIwohhIpT8JKLpF6SRkr6TdIwSV1q\nOX5jSa9LminpB0nnSXVsEJk0Cc48E6ZOhSZN4I03vK0lEksIIeRFQZOLpG7AdcAlwGbAW8Bzktao\n5vhmwEBgPNAROBk4HTg1qwuaQd++PmblyivhlVd8+4orRoN9CCHkkcyscBeThgAfmdkxadu+Ah41\ns7MzHH88cBnQ0sxmJtvOAY4HVrMagm+56to2fsM28PLL0LEj3HYbbLZZrt9SCCFUFEnDzKzDor5O\nwUoukpYA2gMvVtn1IrBVNadtCQxOJZbEC8AqQOuarrfSrz/B0KFw880+2WQklhBCKJhCNui3ABrh\nVVzpxgM7VXPOysD3GY5P7RuZvkNST6Bn8nSW4BN69fLFvBq2FsCEYgdRIuJeLBD3YoG4Fwusn4sX\nqajeYmZ2O3A7gKShuSjaVYK4FwvEvVgg7sUCcS8WkDQ0F69TyAb9CcA8oGWV7S2BcdWcM66a41P7\nQgghlKCCJRczmw0MA3ausmtnvNdYJm8DXSQtWeX4H4FRuY4xhBBCbhR6nMvVQA9JR0vaUNJ1eOP8\nrQCSLpX0ctrxfYEZQB9JG0naDzgLuLqmnmKJ2/MQf7mKe7FA3IsF4l4sEPdigZzci4J2RQYfRAmc\nAbQCPgH+aWaDkn19gK5m1jrt+I2Bm4BOwCQ8EV2YRXIJIYRQJAVPLiGEECpfw5u4MoQQQt5Fcgkh\nhJBzZZtcijIBZomqy72Q1FXSU5LGSpoh6SNJRxYy3nyq6+ci7bx1JU2VNC3fMRZKPf6PSNIpkj6X\nNCv5jPxfoeLNp3rci10kvZ18JiYk/2fWK1S8+SJpW0n9k+9Ak9Qji3Pq9d1Zlsml4BNglrC63gt8\nqp2PgQOAjYBbgNslHVKAcPOqHvcidd4SQD9gUN6DLJB63ourgF7AmcCGwO5UwD2px/dFG+ApYHBy\n/E7AksCAggScX03xjlQnAzNrOXbRvjvNrOwewBDgjirbvgIureb444EpwFJp284BfiDp1FCuj7re\ni2pe42HgsWK/l2LdC+Aa4G6gBzCt2O+jGPcCn/JjDrBhsWMvgXtxAD7gu1Hatu0BA1oU+/3k8L5M\nA3rUcky9vzvLruRS6AkwS1k970UmzfBu3mWrvvdC0h7AnsBJ+YuusOp5L/YGvgV2lfStpFGS7pH0\npzyGmnf1vBfv4Yn2aEmNJC2L/+Hxnpk1tPnH6v3dWXbJhZonwFy5mnNWrub41L5yVZ97sRBJewI7\nUv6DyOp8LyStAtwBHGZmFdPWQv0+F2sBawIH4V+k3YENgKclleP3REqd74WZjcZnArkAmAVMxquQ\n98xfmCWr3t+d5fyhCYtI0tb4LAj/MLN3ix1PEdwH3GJmQ4odSAlYDGgCdDezQWY2GE8wnfC69gZD\n0srA//DPR0egKzAVeLjME21BleONigkwF6jPvQBA0jbAc8B5ZnZLfsIrqPrcix2A8yXNlTQX/0JZ\nJnnes5pzykF97sVYYK6ZfZm27avkdWrsEFHi6nMvTgCmm9npZvaB+QwihwHbUbfq5kpQ7+/Osksu\nFhNg/q6e9wJJ2+KJpbeZXZu/CAunnvdiY2DTtMd5eA+aTYFH8hNp/tXzXrwJLC5p7bRta+FVSqNz\nHmSB1PNeLI0npHSp52X3nbmI6v/dWeweC/Xs5dANmA0cjXeZvA7v+bBmsv9S4OW045fDs2w/vO50\nP7wHxL+K/V6KcC+6AtOBK/A609RjpWK/l0Lfiwzn96ByeovV9XOxGP4l/Dre/Xaz5Pd3gMWK/X4K\nfC92AObjf2ysC2wOPA+MAZYp9vtZxHvRlAV/TM1I3uOmwBrV3It6f3cW/c0uwk3qhWfOWcl/im3T\n9vUBRlU5fmO8z/5veBXA+ZR5N+T63IvkuWV4jCp03MW+FxnOrZjkUp97gU8m+wjevvAT8ADQstjv\no0j34iDg/SQJ/QT0B9oW+33k4D50reb/f58a7kW9vjtj4soQQgg519DqD0MIIRRAJJcQQgg5F8kl\nhBBCzkVyCSGEkHORXEIIIeRcJJcQQgg5F8kllBVJiyeLHO1T7FjqS9I6yXvYtJbj7pf0ZKHiCiGX\nIrmEgpLUJ/lirfqo8Yu2kCRdlBbXPEljJN0uacUcXWIkPmDxk+R6OyXXWr7KcSfgAzvzJu3aqccv\nkl6WtEUdX6fsk37IrUguoRhewr9c0x+fFDWiP/oUj2sN4ERgX3xBsUVmZvPMbJyZza3luMlm9msu\nrpmF9fH3uz2+ts8ASS0KdO1QgSK5hGKYlXy5pj/mAkjaXdIbkn6VNFHSc5LWr+6FknXfe0sanbbu\n+91p+xeTdHayANZMSR9LOjiLGOcmcf1gZv2BG4DdksWnkNRO0ivJa/4i6a5kSViq7J+SrMM+XNJ2\nyb7fq8UkrYMvIwswKdl+Z3Lc79Vi8jXgf6w65bukhyU9nvZ8b0nvy9eKHynpv6mYa/FT8n4/Ai4G\nViBtqn1JnSUNlK8nP0XSYEmd0s4flfx8InkPX+cgplDGIrmEUrMMvpZ7R/yv6BlAf0mNqzn+b8Ap\nwHH4JIN/xVcSTLkUOBxfrrUtcBnwP0m71DGu3/AZghtLaoqvxjcJX+9kf2BbfOGxlH7Ad8n+zYAL\nk9eoamTyHmBB6SHT+uQP4Qtf7ZDakCSzvYD7k+e7A/cC1wN/Bo7C58i6MNs3KWkZFlTFzUnbtSxw\nD9AF6Ax8jK9Dv0KyP5WI/p68hy1yFVMoU8WeSC0eDeuBT4w3F58QMPV4robjm+Ez1G6RPF8cn2hv\nn+T5GcAIYPEM5y6Lf6FvWWX7jUD/Gq55ETA87fmG+BLAbybPjwcmkjZDLrBTEleb5Pl04NBqXn+d\n5NhNq5y7fJXj7geeTHveH7g77XkPPME1SZ6/BZxd5TUOACbX8F5T1079W6QmMhyS6Z6mnSfgZ+Cg\nTP8uacfVOaZ4VMYjSi6hGAax8DoqR6d2SFpX0oNJNdYUfN0IUf2CVQ/hSWSkpDslHZBW5bIRvrri\nQEnTUg/gGGDtal4vZePk+Jl4+8tIfGVG8GTzoZlNTzv+zbR9AFcDfSS9JOnfktar5XrZuB/YL21t\njUOBR8xsVvK8PXBelfd6L9BM0kq1vHYXfGr5g/H3eriltQlJapl0avhS0mR85uQVqX0hsUWJKZSx\nxYsdQGiQZpjZ19Xsexb/cjsGTyzz8ZJJxjp6MxudfHHvBOwIXAOcK2lLFlT77gH8UOXU2bXE+AVe\nxTYP+DHtC7w2lsR1rqT7gN2BvwC9JR1jZvdk+TqZ9Mer3vaS9AZeRbZ92n7h06E/nuHcibW89kjz\nzgNfJlVjT0hqZ2apqrH7geXxKsjR+NT1r1HNv0uOYgplLJJLKBmSWuLtJkeZr+FO0mhcYwnbzGYC\nTwNPS7oC+B6v8x+GJ5E1zOz1OoYzu4YE+BlwmKRl0kovW6ftS8X1JfAlcK2kO/D2hkzJJZXoGtUU\nkJn9JukxvMSyGt6mMzjtkA+A9WuIO1t9gHPx6r/rk23bAD3NbACApFb4InMp85JH1feQq5hCmYnk\nEkrJBPyv2Z6SxuJfoFfgpZeMJB2Z/Pou3s5xCN4Q/bWZTZZ0DXCNpEb4F3EzYEs8edxZzzjvw/8a\nv0dSb7yh/VbgYTMblTT4Xwo8iveiWgVPPoOqeb3UMsJ7SHoOmGlm06o59n58ier1gb5mlr4g0wXA\nU5K+wxf9mocv9NTezM7K9s2Z2TxJ1wFnS7rTzGbgSbK7pKF4NeQVeOkldY5JGgPsKOlNvEfgpFzF\nFMpPtLmEkmFm8/AlaTfHx73cAJzNwr2WqvoV6Am8gfdg2htvVB6T7D8bb6A/Ey9VvAjsg1e91TfO\nacAuQHO8Z9rjeOI4JjlkLp5w7sW/lB/DE9tp1bzeaPxL+HJgPHBtDZd/DV8ZcQOSXmJprzMA7z22\ncxLXu3iHhzHU3Z3AUvgYH/DOA8vjJZG+wG14ySndqcm1v0uun+uYQhmJlShDCCHkXJRcQggh5Fwk\nlxBCCDkXySWEEELORXIJIYSQc5FcQggh5FwklxBCCDkXySWEEELORXIJIYSQc/8PJ48cP0avu04A\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10c29db00>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#draw chart to show AUC and ROC\n",
    "\n",
    "fpr, tpr, thresholds = roc_curve(y_test, y_pred_proba[:,1])\n",
    "\n",
    "plt.figure()\n",
    "plt.plot(fpr, tpr)\n",
    "plt.plot([0, 1], [0, 1],'r--')\n",
    "plt.xlim([0.0, 1.0])\n",
    "plt.ylim([0.0, 1.05])\n",
    "plt.xlabel('False Positive Rate')\n",
    "plt.ylabel('True Positive Rate')\n",
    "plt.title('Positive Moview Reviews')\n",
    "plt.legend(loc=\"lower right\")\n",
    "plt.savefig('Log_ROC')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "6 - Change a parameter in your model (introduce regularization) or change a parameter in your word vector transformation (http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html). Try introducing the use of stop words, or employing a cutoff on terms with min or max df."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#change parameter in vector transformation\n",
    "vectorizer = TfidfVectorizer(tokenizer=tokenize,max_df=0.9, min_df=0.1)\n",
    "X_train_1 = vectorizer.fit_transform(X_train_raw)\n",
    "X_test_1 = vectorizer.transform(X_test_raw)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.80635915150067528"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#calculate the auc again\n",
    "\n",
    "logreg.fit(X_train_1, y_train)\n",
    "y_pred = logreg.predict(X_test_1)\n",
    "y_pred_proba = logreg.predict_proba(X_test_1)\n",
    "\n",
    "logit_roc_auc = roc_auc_score(y_test, y_pred)\n",
    "logit_roc_auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.79726415007671037"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#use stop words\n",
    "vectorizer = TfidfVectorizer(tokenizer=tokenize, stop_words='english')\n",
    "X_train_2 = vectorizer.fit_transform(X_train_raw)\n",
    "X_test_2 = vectorizer.transform(X_test_raw)\n",
    "\n",
    "logreg.fit(X_train_2, y_train)\n",
    "y_pred = logreg.predict(X_test_2)\n",
    "y_pred_proba = logreg.predict_proba(X_test_2)\n",
    "\n",
    "logit_roc_auc = roc_auc_score(y_test, y_pred)\n",
    "logit_roc_auc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "7 - Make four models in total, changing parameters and comparing the AUC results. Report your findings in a tabular form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#make four models in total\n",
    "\n",
    "def total(y_train, y_test, max_df, min_df):\n",
    "\n",
    "    vectorizer = TfidfVectorizer(tokenizer=tokenize,\n",
    "                                     stop_words='english',\n",
    "                                     max_df=max_df,\n",
    "                                     min_df=min_df)\n",
    "    X_train = vectorizer.fit_transform(X_train_raw)\n",
    "    X_test = vectorizer.transform(X_test_raw)\n",
    "        \n",
    "    logreg.fit(X_train, y_train)\n",
    "    y_pred = logreg.predict(X_test)\n",
    "    y_pred_proba = logreg.predict_proba(X_test)\n",
    "    logit_roc_auc = roc_auc_score(y_test, y_pred)\n",
    "    \n",
    "    return logit_roc_auc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "Modle: LogisticRegression\n",
      "TfidfVectorizer-max_df:0.9\n",
      "TfidfVectorizer-min_df:0.1\n",
      "AUC score: 0.7941681748445122\n",
      "\n",
      "2\n",
      "Modle: LogisticRegression\n",
      "TfidfVectorizer-max_df:0.8\n",
      "TfidfVectorizer-min_df:0.2\n",
      "AUC score: 0.7383717191390065\n",
      "\n",
      "3\n",
      "Modle: LogisticRegression\n",
      "TfidfVectorizer-max_df:0.7\n",
      "TfidfVectorizer-min_df:0.3\n",
      "AUC score: 0.7126990105740876\n",
      "\n",
      "4\n",
      "Modle: LogisticRegression\n",
      "TfidfVectorizer-max_df:0.6\n",
      "TfidfVectorizer-min_df:0.4\n",
      "AUC score: 0.6323919853745028\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#call the funtion and print 4-pair-parameters results\n",
    "\n",
    "AUC_list = []\n",
    "max_df_list=[0.9, 0.8, 0.7, 0.6]\n",
    "min_df_list=[0.1, 0.2, 0.3, 0.4]\n",
    "AUC_list = []\n",
    "\n",
    "for i in [0,1,2,3]:\n",
    "    AUC = total(y_train, y_test, max_df_list[i], min_df_list[i])\n",
    "    AUC_list.append(AUC)\n",
    "    print (i+1)\n",
    "    print (\"Modle: LogisticRegression\")\n",
    "    print (\"TfidfVectorizer-max_df:{}\".format(max_df_list[i]))\n",
    "    print (\"TfidfVectorizer-min_df:{}\".format(min_df_list[i]))\n",
    "    print ('AUC score: {}'.format(AUC))\n",
    "    print ()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+--------+--------+----------------+\n",
      "| max_df | min_df |      AUC       |\n",
      "+--------+--------+----------------+\n",
      "|  0.9   |  0.1   | 0.794168174845 |\n",
      "|  0.8   |  0.2   | 0.738371719139 |\n",
      "|  0.7   |  0.3   | 0.712699010574 |\n",
      "+--------+--------+----------------+\n"
     ]
    }
   ],
   "source": [
    "#print the table form\n",
    "\n",
    "from prettytable import PrettyTable\n",
    "t = PrettyTable(['max_df', 'min_df','AUC'])\n",
    "t.add_row(a)\n",
    "t.add_row(b)\n",
    "t.add_row(c)\n",
    "print (t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}