Training_day_Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#TreeTagger\" data-toc-modified-id=\"TreeTagger-0.0.0.1\"><span class=\"toc-item-num\">0.0.0.1&nbsp;&nbsp;</span>TreeTagger</a></span></li></ul></li></ul></li></ul></li><li><span><a href=\"#Regular-expressions\" data-toc-modified-id=\"Regular-expressions-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Regular expressions</a></span><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><ul class=\"toc-item\"><li><span><a href=\"#Examples-of-regular-expressions:\" data-toc-modified-id=\"Examples-of-regular-expressions:-1.0.0.1\"><span class=\"toc-item-num\">1.0.0.1&nbsp;&nbsp;</span>Examples of regular expressions:</a></span></li></ul></li><li><span><a href=\"#Modifiers\" data-toc-modified-id=\"Modifiers-1.0.1\"><span class=\"toc-item-num\">1.0.1&nbsp;&nbsp;</span>Modifiers</a></span></li><li><span><a href=\"#Escaping-special-characters\" data-toc-modified-id=\"Escaping-special-characters-1.0.2\"><span class=\"toc-item-num\">1.0.2&nbsp;&nbsp;</span>Escaping special characters</a></span></li><li><span><a href=\"#Defining-sets-of-characters\" data-toc-modified-id=\"Defining-sets-of-characters-1.0.3\"><span class=\"toc-item-num\">1.0.3&nbsp;&nbsp;</span>Defining sets of characters</a></span></li><li><span><a href=\"#Defining-groups\" data-toc-modified-id=\"Defining-groups-1.0.4\"><span class=\"toc-item-num\">1.0.4&nbsp;&nbsp;</span>Defining groups</a></span></li><li><span><a href=\"#Exercises-in-online-tester\" data-toc-modified-id=\"Exercises-in-online-tester-1.0.5\"><span class=\"toc-item-num\">1.0.5&nbsp;&nbsp;</span>Exercises in online tester</a></span></li></ul></li></ul></li><li><span><a href=\"#Introduction-to-Python\" data-toc-modified-id=\"Introduction-to-Python-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Introduction to Python</a></span><ul class=\"toc-item\"><li><span><a href=\"#Rubbish-in,-Rubbish-out\" data-toc-modified-id=\"Rubbish-in,-Rubbish-out-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>Rubbish in, Rubbish out</a></span></li><li><span><a href=\"#Basic-syntax\" data-toc-modified-id=\"Basic-syntax-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Basic syntax</a></span><ul class=\"toc-item\"><li><span><a href=\"#Variables-and-data-types\" data-toc-modified-id=\"Variables-and-data-types-2.2.1\"><span class=\"toc-item-num\">2.2.1&nbsp;&nbsp;</span>Variables and data types</a></span></li></ul></li><li><span><a href=\"#Basic-string-manipulation\" data-toc-modified-id=\"Basic-string-manipulation-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Basic string manipulation</a></span><ul class=\"toc-item\"><li><span><a href=\"#String-functions\" data-toc-modified-id=\"String-functions-2.3.1\"><span class=\"toc-item-num\">2.3.1&nbsp;&nbsp;</span>String functions</a></span><ul class=\"toc-item\"><li><span><a href=\"#str.upper()\" data-toc-modified-id=\"str.upper()-2.3.1.1\"><span class=\"toc-item-num\">2.3.1.1&nbsp;&nbsp;</span><a href=\"https://docs.python.org/3/library/stdtypes.html#str.upper\" target=\"_blank\"><code>str.upper()</code></a></a></span></li><li><span><a href=\"#str.lower()\" data-toc-modified-id=\"str.lower()-2.3.1.2\"><span class=\"toc-item-num\">2.3.1.2&nbsp;&nbsp;</span><a href=\"https://docs.python.org/3/library/stdtypes.html#str.lower\" target=\"_blank\"><code>str.lower()</code></a></a></span></li><li><span><a href=\"#str.rstrip()\" data-toc-modified-id=\"str.rstrip()-2.3.1.3\"><span class=\"toc-item-num\">2.3.1.3&nbsp;&nbsp;</span><a href=\"https://docs.python.org/3/library/stdtypes.html#str.rstrip\" target=\"_blank\"><code>str.rstrip()</code></a></a></span></li><li><span><a href=\"#str.capitalize()\" data-toc-modified-id=\"str.capitalize()-2.3.1.4\"><span class=\"toc-item-num\">2.3.1.4&nbsp;&nbsp;</span><a href=\"https://docs.python.org/3/library/stdtypes.html#str.capitalize\" target=\"_blank\"><code>str.capitalize()</code></a></a></span></li><li><span><a href=\"#str.count(sub[,-start[,-end]])\" data-toc-modified-id=\"str.count(sub[,-start[,-end]])-2.3.1.5\"><span class=\"toc-item-num\">2.3.1.5&nbsp;&nbsp;</span><a href=\"https://docs.python.org/3/library/stdtypes.html#str.count\" target=\"_blank\"><code>str.count(sub[, start[, end]])</code></a></a></span></li><li><span><a href=\"#str.replace(old,-new[,-count])\" data-toc-modified-id=\"str.replace(old,-new[,-count])-2.3.1.6\"><span class=\"toc-item-num\">2.3.1.6&nbsp;&nbsp;</span><a href=\"https://docs.python.org/3/library/stdtypes.html#str.replace\" target=\"_blank\"><code>str.replace(old, new[, count])</code></a></a></span></li><li><span><a href=\"#str.split(sep=None,-maxsplit=-1)\" data-toc-modified-id=\"str.split(sep=None,-maxsplit=-1)-2.3.1.7\"><span class=\"toc-item-num\">2.3.1.7&nbsp;&nbsp;</span><a href=\"https://docs.python.org/3/library/stdtypes.html#str.split\" target=\"_blank\"><code>str.split(sep=None, maxsplit=-1)</code></a></a></span></li><li><span><a href=\"#str.join(iterable)\" data-toc-modified-id=\"str.join(iterable)-2.3.1.8\"><span class=\"toc-item-num\">2.3.1.8&nbsp;&nbsp;</span><a href=\"https://docs.python.org/3/library/stdtypes.html#str.join\" target=\"_blank\"><code>str.join(iterable)</code></a></a></span></li><li><span><a href=\"#str.title()\" data-toc-modified-id=\"str.title()-2.3.1.9\"><span class=\"toc-item-num\">2.3.1.9&nbsp;&nbsp;</span><a href=\"https://docs.python.org/3/library/stdtypes.html#str.title\" target=\"_blank\"><code>str.title()</code></a></a></span></li></ul></li></ul></li><li><span><a href=\"#Regex-in-Python\" data-toc-modified-id=\"Regex-in-Python-2.4\"><span class=\"toc-item-num\">2.4&nbsp;&nbsp;</span>Regex in Python</a></span><ul class=\"toc-item\"><li><span><a href=\"#Syntax\" data-toc-modified-id=\"Syntax-2.4.1\"><span class=\"toc-item-num\">2.4.1&nbsp;&nbsp;</span>Syntax</a></span></li><li><span><a href=\"#Basic-matches\" data-toc-modified-id=\"Basic-matches-2.4.2\"><span class=\"toc-item-num\">2.4.2&nbsp;&nbsp;</span>Basic matches</a></span></li><li><span><a href=\"#Using-regex-in-Python\" data-toc-modified-id=\"Using-regex-in-Python-2.4.3\"><span class=\"toc-item-num\">2.4.3&nbsp;&nbsp;</span>Using regex in Python</a></span></li></ul></li><li><span><a href=\"#Lists-and-dictionaries\" data-toc-modified-id=\"Lists-and-dictionaries-2.5\"><span class=\"toc-item-num\">2.5&nbsp;&nbsp;</span>Lists and dictionaries</a></span></li><li><span><a href=\"#Input-and-Output\" data-toc-modified-id=\"Input-and-Output-2.6\"><span class=\"toc-item-num\">2.6&nbsp;&nbsp;</span>Input and Output</a></span><ul class=\"toc-item\"><li><span><a href=\"#Working-with-local-files\" data-toc-modified-id=\"Working-with-local-files-2.6.1\"><span class=\"toc-item-num\">2.6.1&nbsp;&nbsp;</span>Working with local files</a></span></li><li><span><a href=\"#Working-with-web-pages\" data-toc-modified-id=\"Working-with-web-pages-2.6.2\"><span class=\"toc-item-num\">2.6.2&nbsp;&nbsp;</span>Working with web pages</a></span></li><li><span><a href=\"#Exercises\" data-toc-modified-id=\"Exercises-2.6.3\"><span class=\"toc-item-num\">2.6.3&nbsp;&nbsp;</span>Exercises</a></span></li></ul></li><li><span><a href=\"#Natural-Language-Processing-demo\" data-toc-modified-id=\"Natural-Language-Processing-demo-2.7\"><span class=\"toc-item-num\">2.7&nbsp;&nbsp;</span>Natural Language Processing demo</a></span><ul class=\"toc-item\"><li><span><a href=\"#Named-Entity-Recognition\" data-toc-modified-id=\"Named-Entity-Recognition-2.7.1\"><span class=\"toc-item-num\">2.7.1&nbsp;&nbsp;</span>Named Entity Recognition</a></span></li><li><span><a href=\"#Part-of-Speech-tagging\" data-toc-modified-id=\"Part-of-Speech-tagging-2.7.2\"><span class=\"toc-item-num\">2.7.2&nbsp;&nbsp;</span>Part-of-Speech tagging</a></span></li></ul></li></ul></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Programming\n",
    "\n",
    "A programme (or script) is a series of linked commands that can be run in sequence, in a loop, or multiple times across a group of objects.\n",
    "\n",
    "For example, you might have a simple task to open a text file, decide if the content is written in English or Greek, run a different set of find and replace functions on it depending on the language, and then save the file again. If you have one file, or even ten, doing this by hand may not be too onerous. Once you have a hundred, a thousand, a million files…\n",
    "\n",
    "A simpler programme follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# defining variables:\n",
    "\n",
    "greet = \"hello\"\n",
    "myname = \"Slartyblartfast\" # replace this with your name\n",
    "\n",
    "print(greet.title() + ', ' + myname + ', ' + greet.lower() + '!')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Add your name to the script above. Run it.\n",
    "\n",
    "Change the value of the variable _greet_ to a greeting in another language (ignore the case). Run it again.\n",
    "\n",
    "Pretty basic, but you've run a programme!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Progamming languages used in DH\n",
    "\n",
    " * [XSLT](https://www.w3schools.com/xml/xsl_intro.asp) – transforming XML\n",
    " * [R](https://www.r-project.org/) – statistics and visualisation\n",
    " * [PHP](https://www.w3schools.com/pHP/default.asp) – server-side web page control\n",
    " * [Javascript](https://www.w3schools.com/js/default.asp) – web page behaviour (client-side)\n",
    " * [Java](https://docs.oracle.com/javase/tutorial/) – platform independent applications (compiled)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Regular expressions\n",
    "\n",
    "\n",
    "A regular expression (regex for short) is a special text string for describing a search pattern. \n",
    "\n",
    "Python has a special regular expression module (re) for advanced string manipulation. \n",
    "This module needs to be imported before we can use it.\n",
    "\n",
    "Regular expressions specify a pattern and allow you to match anything that fits that pattern\n",
    "[A-z] – matches all the letters from A to z\n",
    "[0-9] – matches any numbers from 0 – 9\n",
    "\n",
    "More information:\n",
    "http://www.regular-expressions.info/\n",
    "<br>Regex tester:\n",
    "http://pythex.org/, https://regex101.com/\n",
    "\n",
    "#### Examples of regular expressions:\n",
    "\n",
    "```r```: single letter r\n",
    "\n",
    "```r.```: single letter r followed by (almost) anything\n",
    "\n",
    "```r[ae]```: single letter r followed by \"character class\" (either a or e)\n",
    "\n",
    "```[Rr]```: character class: a selection of characters\n",
    "\n",
    "```[0123456789]``` or ```[0-9]```: selection of numbers in the text\n",
    "\n",
    "```[aeiou]```: all the vowels\n",
    "\n",
    "```[aeiou][aeiou]```: two vowels in a row\n",
    "\n",
    "```[^aeiou]```: not a vowel\n",
    "\n",
    "```to see```: spaces are OK in a pattern"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Modifiers\n",
    "\n",
    "<img src=\"modifiers.png\" style=\"width: 60%;\" align=\"left\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Escaping special characters\n",
    "\n",
    "If you want to find one of the special characters (e.g. a '.') then you have to escape it.\n",
    "<br>The \\ character either escapes special characters or signals a special sequence.\n",
    "\n",
    "\n",
    "| Code        | Meaning   |\n",
    "| -------------|-----------------|\n",
    "| `.` | match any character \n",
    "| `\\.` | match a full stop \n",
    "| `?` | match 0 or 1 or preceding character \n",
    "| `\\?` | match ? \n",
    "| `\\` | escape next character \n",
    "| `\\\\` | match a \\ character \n",
    "| `\\b` | a word boundary \n",
    "| `\\d` | a digit (number)\n",
    "| `\\D` | a non-digit (anything not a number)\n",
    "| `\\w` | an alphanumeric character (a letter or number)\n",
    "| `\\W` | a non-alphanumeric character\n",
    "| `\\s` | matches a whitespace character; includes space, `\\t` (tab), `\\n` (new line)\n",
    "| `\\S` | matches a non-whitespace character\n",
    "| `^` | (caret) match the start of the string\n",
    "| `$` | match the end of the string or just before the newline at the end of the string\n",
    "| `*` | match 0 or more repetitions of the preceding regular expression\n",
    "| `+` | match 1 or more repetitions of the preceding regular expression\n",
    "| `?` | match 0 or 1 repetitions of the preceding regular expression\n",
    "| `[ ]` | indicate a set of characters\n",
    "| &#124; | A&#124;B will match either A or B\n",
    "| `( )` | match whatever regular expression is inside the parentheses, and indicate the start and end of a group\n",
    "| `{m}` | match exactly m copies of the previous regular expression \n",
    "| `{m,n} | match from m to n repetitions of the preceding regular expression, e.g. `a{3,5}` will match from 3 to 5 \"a\" characters\n",
    "\n",
    "<br>`\\b[Cc]at[Ss]?`\n",
    "<br>Looking for the word \"cat\" the first letter might be \"c\" or \"C\". \n",
    "<br>\"c\" or \"C\" must be the beginning of the \n",
    "word, cannot be in the middle of a word.\n",
    "<br>Followed by the letters \"at\" may be followed by \"s\" or \"S\".\n",
    "<br>Which must be the end of the word."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining sets of characters\n",
    "\n",
    "[...] are used to indicate a set of characters.\n",
    "<br>Characters can be listed individually e.g `[abcd]`.\n",
    "<br>Or can denote a range using the hyphen e.g. `[a-d]`.\n",
    "<br>\"Normal\" special characters are not active inside sets: \n",
    "- `-` denoting ranges\n",
    "- `^` at beginning of set indicates a negative set, e.g. `[^5]` will match any character except \"5\"\n",
    "- `\\` escapes characters and allows 'shortcut' codes\n",
    "- `-` and `]` need escaping unless they are the first character"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Defining groups\n",
    "\n",
    "(…) matches whatever regular expression is inside the parentheses, and indicates the start and end of a group.\n",
    "Defining groups allows you to reference the group."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises in online tester\n",
    "\n",
    "1. Go to https://regex101.com/.\n",
    "2. From the left-hand menu select 'python'.\n",
    "3. Put your text in the blank field and experiment with simple pattern matches following the instructions above.\n",
    "\n",
    "Example texts here..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Python\n",
    "\n",
    "\n",
    "<br>Python is a programming language created at the beginning of the 1990s by Guido van Rossum when he worked at the Centrum voor Wiskunde en Informatica (CWI) in Amsterdam. Work on Python began in 1989, and it was finally released in 1991.\n",
    "\n",
    "Python is not named after the snake, but after Monty Python.\n",
    "\n",
    "<img align=\"left\" src=\"3-Python-IDLE-name.gif\" style=\"width: 250px;\"/> <img align=\"center\" src=\"python-300x214.gif\" style=\"width: 200px;\"/> \n",
    "\n",
    "Since 2001 the non-commercial Python Software Foundation holds the rights on Python. Python is continually developed further by van Rossum (who now works for Google) and a large number of volunteers.\n",
    "Python is widely used in all kinds of areas.\n",
    "\n",
    "Among the bigger firms or institutions that use Python are the NASA, Google, CERN…\n",
    "\n",
    "- Python 2 - 16 October 2000\n",
    "- Python 3 - 3 December 2008 \n",
    "- Python 3 has better Unicode support but many of the popular libraries support Python 2\n",
    "\n",
    "\n",
    "## Rubbish in, Rubbish out\n",
    "'On two occasions I have been asked, — \n",
    "\"Pray, Mr. Babbage, if you put into the machine wrong figures, will the right answers come out?\" \n",
    "... I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a question.' \n",
    "\n",
    "Passages from the Life of a Philosopher (1864), ch. 5 \"Difference Engine No. 1\"\n",
    "\n",
    "<img src=\"15401495.jpg\" title=\"The Thrilling Adventures of Lovelace and Babbage by Sydney Padua\"/>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic syntax\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Hello! I am a comment.  \n",
    "# every line of code following the sign # is ignored:\n",
    "# print(\"I'm a line of text: do you see me?\")\n",
    "# see? You don't see me...\n",
    "\n",
    "2 + 3\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Hello again! Please run this code:\n",
    "\n",
    "print(\"Hello world\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variables and data types\n",
    "\n",
    "We might want to use Python to execute an operation and process the result to print it, process it, and reuse it more than once.\n",
    "\n",
    "We store the values in variables. Think of it as a shoebox where you place your content; the next time you need that content (i.e. the result of a previous operation) you simply call the shoebox's name.\n",
    "\n",
    "```variable_name = variable_value```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = 2 + 3\n",
    "\n",
    "#now we print the result\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you think we will get when we execute the following code?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result + 5\n",
    "\n",
    "# What is the  value of result now? Why?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What types of values can we put into a variable? What goes into the shoebox? We can start with this list:\n",
    "\n",
    "- Integers (-1,0,1,2,3,4...)\n",
    "- Strings (\"Hello\", \"s\", \"Monty Python and the Flying Circus\", \"Antonio Lucio Vivaldi\", \"I'm hungry\"...)\n",
    "- Floats (3.14159; 2.71828...)\n",
    "- Booleans (True, False)\n",
    "\n",
    "If you're not sure what type of value you're dealing with, you can use the function type(). Yes, it works with variables too!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(\"'Tis but a scratch!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(3.14159)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = \"hello\"\n",
    "type(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You declare strings with single ('') or double (\"\") quote: it makes no difference. <br>However:\n",
    "\n",
    "- what happens if you forget the quotes?\n",
    "- what happens if you put quotes around a number?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hello = \"goodbye\"\n",
    "print(hello)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(hello)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Why is this important? Try to sum two strings and see what happens..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"2\" + \"3\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#you probably wanted this... \n",
    "#int(\"2\") + int(\"3\")\n",
    "\n",
    "#which you can also accomplish like this:\n",
    "2 + 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we are working with strings, then the \"+\" sign is used to concatenate the strings, like the example \"2\" + \"3\" = 23."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#supplement the code, so you can print the two sentences together \n",
    "#When Caesar says, \"Do this,\" it is performed.\n",
    "\n",
    "a = \"'Do this,' \"\n",
    "b =  \"it is performed.\"\n",
    "print(\"When Caesar says, \" + a + b)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic string manipulation\n",
    "\n",
    "### String functions\n",
    "\n",
    "#### [`str.upper()`](https://docs.python.org/3/library/stdtypes.html#str.upper)\n",
    "\n",
    "Returns a copy of the string in upper case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('I am shouting'.upper()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [`str.lower()`](https://docs.python.org/3/library/stdtypes.html#str.lower)\n",
    "\n",
    "Returns a copy of the string in lower case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('I AM WHISPERING!'.lower())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [`str.rstrip()`](https://docs.python.org/3/library/stdtypes.html#str.rstrip)\n",
    "\n",
    "Gets rid of any whitepsace at the end of a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Start-->', 'Exeunt [all but Brutus and Cassius]                 '.rstrip(), '<-- Stop!', sep = '')\n",
    "# I've demonstrated this by printing the string out, surrounded by start and stop arrows."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [`str.capitalize()`](https://docs.python.org/3/library/stdtypes.html#str.capitalize)\n",
    "\n",
    "Returns a copy of the string with its first character capitalized and the rest lowercased."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"capitol\".capitalize())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [`str.count(sub[, start[, end]])`](https://docs.python.org/3/library/stdtypes.html#str.count)\n",
    "\n",
    "Return the number of non-overlapping occurrences of substring sub in the range [start, end]. Optional arguments start and end are interpreted as in slice notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sentence = \"The fault, dear Brutus, is not in our stars,\\\n",
    "            But in ourselves that we are underlings.\"\n",
    "print(sentence.count(\"s \"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [`str.replace(old, new[, count])`](https://docs.python.org/3/library/stdtypes.html#str.replace)\n",
    "\n",
    "Return a copy of the string with all occurrences of substring old replaced by new. If the optional argument count is given, only the first count occurrences are replaced."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \"People and senators, be not affrighted.\\\n",
    "     Fly not! Stand still! Ambition's debt is paid.\"\n",
    "print(s.replace(\"affrighted\", \"afraid\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [`str.split(sep=None, maxsplit=-1)`](https://docs.python.org/3/library/stdtypes.html#str.split)\n",
    "\n",
    "Return a list of the words in the string, using sep as the delimiter string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \"Cowards die many times before their deaths;\\\n",
    "    The valiant never taste of death but once.\"\n",
    "#print(s.split())\n",
    "print(s.split(\";\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [`str.join(iterable)`](https://docs.python.org/3/library/stdtypes.html#str.join)\n",
    "\n",
    "Return a string which is the concatenation of the strings in iterable. The separator between elements is the string providing this method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = \"As Caesar loved me, I weep for him; as he was fortunate, I rejoice at it;\\\n",
    "    as he was valiant, I honor him; but as he was ambitious, I slew him.\"\n",
    "l = s.split(',')\n",
    "print(l)\n",
    "\n",
    "brian = \" —\"\n",
    "print(brian.join(l))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [`str.title()`](https://docs.python.org/3/library/stdtypes.html#str.title)\n",
    "\n",
    "Return a titlecased version of the string where words start with an uppercase character and the remaining characters are lowercase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "brian = \"the tragedy of julius caesar\"\n",
    "print(brian.title())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#the function len(x) takes a sequence, x, and returns the length. The sequence can be a string.\n",
    "\n",
    "print(len(\"how long am i?\"))\n",
    "\n",
    "magic_word = \"Supercalifragilisticexpialidocious!\"\n",
    "\n",
    "#How long is the magic_word? \n",
    "print(len(magic_word))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Regex in Python\n",
    "\n",
    "### Syntax\n",
    "```result = re.search(pattern, text)```\n",
    "<br>e.g. ```result = re.search(\"abc\", \"abcdefg\")```\n",
    "\n",
    "### Basic matches\n",
    "```re.search()``` checks for a match anywhere in the string\n",
    "<br>```re.findall(pattern, text, flag)``` returns list of strings; case sensitive unless the flag ```re.IGNORECASE``` is set\n",
    "<br>```result.group([group1, …])``` returns one or more subgroups of the match; if there is a single argument, the result is a single string; if there are multiple arguments, the result is a tuple with one item per argument\n",
    "<br><br>.  this matches any character except a newline\n",
    "<br>e.g. pattern: c.t\n",
    "<br>matches:cat, cbt, cct, cdt, c1t, c@t…"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import re"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "help(re)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "text = \"Beware the Ides of March.\"\n",
    "found = re.search('a', text)\n",
    "print(found.group())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "text = \"The ides of March are come.\\\n",
    "        Aye, Caesar, but not gone.\"\n",
    "found = re.findall('a', text)\n",
    "print(found)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "text = \"Remember March, the ides of March remember.\"\n",
    "found = re.findall('h....e', text)\n",
    "\n",
    "print(found)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "text = \"Yond Cassius has a lean and hungry look.\\\n",
    "        He thinks too much. Such men are dangerous.\"\n",
    "found = re.findall('.uch?', text)\n",
    "print(found)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "text = \"Speak, strike, redress!\"\n",
    "\n",
    "found = re.findall('\\w{5}', text)\n",
    "print(found)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "s = \"http://internetshakespeare.uvic.ca/Library/Texts/JC/\"\n",
    "r = re.search(\"(http)://(.*?)/(.*)\", s)\n",
    "g1 = r.group(1)\n",
    "g2 = r.group(2)\n",
    "g3 = r.group(3)\n",
    "print(g1)\n",
    "print(g2)\n",
    "print(g3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using regex in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# match a string of 9 characters\n",
    "\n",
    "import re\n",
    "\n",
    "text = \"Here is a mourning Rome, a dangerous Rome\"\n",
    "\n",
    "found = re.findall('\\w{9}', text)  # {} a repetition operator\n",
    "print(found)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# match any of the following characters: T, t, h, e\n",
    "\n",
    "import re\n",
    "\n",
    "text = \"There is a tide in the affairs of men,\\\n",
    "        Which taken at the flood leads on to fortune;\\\n",
    "        Omitted, all the voyage of their life\\\n",
    "        Is bound in shallows and in miseries.\"\n",
    "\n",
    "found = re.findall('[theT]+', text)\n",
    "print(found)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# match any string enclosed in parentheses\n",
    "\n",
    "import re\n",
    "\n",
    "brian = \"Stoop then, and wash. How many ages hence\\\n",
    "        Shall this our lofty scene be acted o'er,\\\n",
    "        In states unborn and accents yet unknown!\\\n",
    "        (The conspirators dip their hands in Caesar's blood.)\"\n",
    "\n",
    "r = re.findall(\"\\(.*?\\)\", brian)\n",
    "print(r)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# match all capitalised words\n",
    "\n",
    "import re\n",
    "# Create a regular expression to find all capitalised words.\n",
    "\n",
    "caius = \"Shall Rome stand under one man's awe? What Rome?\\\n",
    "         My ancestors did from the streets of Rome\\\n",
    "         The Tarquin drive, when he was called a king.\\\n",
    "         Speak, strike, redress!\"\n",
    "r = re.findall(\"[A-Z]\\w*\", caius)\n",
    "print(r)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a regular expression that will find characters ending their speeches\n",
    "# with a question.\n",
    "\n",
    "import re\n",
    "\n",
    "octavius = \"\"\"Cassius: Did Cicero say anything?\\n\n",
    "            Casca: Ay, he spoke Greek.\\n\n",
    "            Cassius: To what effect?\\n\n",
    "            Casca: Nay, and I tell you that, I'll ne'er look you\\n \n",
    "            i'th'face again. But those that understood him, smiled at one\\n \n",
    "            another and shook their heads. But for mine own part, it was\\n \n",
    "            Greek to me.\"\"\"\n",
    "\n",
    "r = re.findall(\"([A-Z].*):(.*\\?)\\n\", octavius)\n",
    "\n",
    "print(r)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lists and dictionaries\n",
    "\n",
    "Lists and dictionaries are two very useful types to store collections of data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lists"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# how do we check if this is indeed a list\n",
    "\n",
    "poets = [\"Homer\", \"Pindar\", \"Aeschylus\", \"Sophocles\", \"Aristophanes\"]\n",
    "\n",
    "type(poets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Items in a list are accessible via their index. Do remember that indexing starts from 0!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# print the first and the last items in the list\n",
    "\n",
    "print(poets[0])\n",
    "\n",
    "print(poets[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There several of methods you can apply to lists.\n",
    "\n",
    "You can append items at the end of a list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# append the name of the next major poet to the end of the list\n",
    "\n",
    "poets.append(\"Menander\")\n",
    "print(poets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can insert elements at a predefined index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# insert a poet according to chronological or received order\n",
    "\n",
    "poets.insert(1, \"Hesiod\")\n",
    "print(poets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# find the index of a given poet\n",
    "\n",
    "print(poets.index(\"Pindar\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# add Euripides somewhere? Anyone else?\n",
    "\n",
    "# ... your code here ... \n",
    "\n",
    "print(poets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# combine two lists\n",
    "\n",
    "hellenistic = [\"Callimachus\",\"Apollonius\",\"Theocritus\",\"Nicander\",\"Philo\"]\n",
    "\n",
    "poets = poets + hellenistic\n",
    "\n",
    "print(poets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Dictionaries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# a dictionary is a collection of key : value pairs\n",
    "# verify that this is a dictionary\n",
    "\n",
    "starwars = { \"IV\" : \"A New Hope\" ,\n",
    "             \"V\" : \"The Empire Strikes Back\",\n",
    "             \"VI\" : \"Return of the Jedi\",\n",
    "             \"VII\" : \"The Force Awakens\"}\n",
    "type(starwars)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# once a dictionary is defined you can add further keys and values\n",
    "\n",
    "starwars[\"VIII\"] = \"The Last Jedi\"\n",
    "\n",
    "print(starwars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dictionaries are collections of key : value pairs. You access the value using the key as an index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# print a values of your choice\n",
    "\n",
    "starwars[\"V\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# add the prequels\n",
    "\n",
    "# ... your code here ...\n",
    "\n",
    "print(starwars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With dictionaries, we can remove key:value pairs with the `del` keyword."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "book = {\"title\": \"The Thrilling Adventures of Lovelace and Babbage\",\n",
    "         \"author\": \"Sydney Padua\",\n",
    "         \"published\": 2015,\n",
    "         \"place\": \"London\",\n",
    "        \"genre\": \"Comic\",\n",
    "        \"shelf\": \"58.3\",\n",
    "        \"on_loan\": True}\n",
    "\n",
    "del book[\"place\"]\n",
    "\n",
    "print(book)\n",
    "\n",
    "# print the dictionary to check that the key:value pair was deleted"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also change the value associated with a key."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "book[\"on_loan\"] = False\n",
    "\n",
    "print(book)\n",
    "#print the dictonary to double check"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input and Output\n",
    "\n",
    "One of the most frequent tasks that programmers do is reading data from files and writing some of the output to other files.\n",
    "\n",
    "In Python we first open a file-handler with the appropriate mode in order to process it. Files can be opened in:\n",
    "\n",
    "    - read mode (\"r\")\n",
    "    - write mode (\"w\")\n",
    "    - append mode (\"a\")\n",
    "\n",
    "Let's try to read the content of one of the txt files of our **texts** directory.\n",
    "\n",
    "First, we open the file handler in read mode (note that \"r\" is optional: read is the default mode).\n",
    "Now there are a bunch of things we can do:\n",
    "\n",
    "* Read the full content in one variable with this code:\n",
    "\n",
    "content = f.read()\n",
    "\n",
    "* Read the lines in a list of lines:\n",
    "\n",
    "lines = f.readlines()\n",
    "\n",
    "* Read the content of one line at a time with a for loop; the f object is iterable.\n",
    "    \n",
    "Once you're done, don't forget to close the handle!\n",
    "\n",
    "### Working with local files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# try one or all of the above\n",
    "# edit the text file to create more new lines, see what happens\n",
    "\n",
    "f = open(\"texts/starwars.txt\")\n",
    "for l in f:\n",
    "    print(l)\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, there is a shortcut statement, which you'll often see and is very convenient, because it takes care of opening, closing and cleaning up the mess, in case there is an error:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(\"texts/starwars.txt\") as f:\n",
    "    #mind the indent!\n",
    "    for l in f:\n",
    "        #double indent, of course!\n",
    "        print(l)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try to write a simple message to a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out = open(\"texts/test.txt\", \"w\")\n",
    "\n",
    "#the file is now open; let's write something in it\n",
    "out.write(\"This is a test!\\nThis is a second line (separated with a new-line feed)\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The file has been created! Let's check it out.\n",
    "\n",
    "But before we can do anything (e.g. open it with your favorite text editor) we have to **close** the file-handler!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "out.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at its content."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#open the file and print its content\n",
    "\n",
    "with open(\"texts/test.txt\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we don't want to overwrite an existing file, we need to **append** to the file rather than **write**.\n",
    "\n",
    "When we write, the file is truncated to the start of the file and we write from there.\n",
    "When we append, the pointer goes to the end of the file and the string is added on from there.\n",
    "\n",
    "So after we opened the file in \"write mode\" for the second time, all content of the file was erased and replaced with the new content.\n",
    "\n",
    "Pay attention: when you open a file in \"w\" mode:\n",
    "\n",
    "- if it doesn't exist, a new file with that name is created\n",
    "- if it does exist, it is completely overwritten and all previous content is lost\n",
    "\n",
    "If you want to write content to an existing file without losing its pervious content, you have to open the file with the \"a\" mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#append something else to the file\n",
    "\n",
    "with open(\"texts/test.txt\", \"a\") as out:\n",
    "    out.write('''\\nAnd this is some more additional content.\n",
    "The new content is appended at the bottom of the existing file''')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Mind the gap!**\n",
    "\n",
    "Indentation matters, or else **COMPUTER SAYS NO**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open(\"texts/test.txt\") as f:\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Working with web pages\n",
    "\n",
    "We need to import the python library which works with URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from urllib.request import urlopen \n",
    "\n",
    "url = 'http://shelleygodwinarchive.org/sc/oxford/frankenstein/volume/i/#/p1/mode/rdg'\n",
    "\n",
    "response = urlopen(url)\n",
    "webContent = response.read()\n",
    "\n",
    "print(webContent[0:550])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Look at this script and try to figure out what it does. Does it help if you run it?\n",
    "\n",
    "from urllib.request import urlopen \n",
    "url = \"http://www.gutenberg.org/files/1023/1023.txt\"\n",
    "conn = urlopen(url)\n",
    "text = conn.read()\n",
    "conn.close()\n",
    "\n",
    "text = text.lower()\n",
    "text = text.decode(\"utf-8\")\n",
    "text = text.replace(\"--\", \" -- \")\n",
    "words = text.split()\n",
    "wordlist = []\n",
    "for word in words:\n",
    "    word = word.strip(\"()*.,:;?!'\\\"\")\n",
    "    if len(word) > 10:\n",
    "        if word not in wordlist:\n",
    "            wordlist.append(word)\n",
    "            \n",
    "wordlist.sort()\n",
    "print(wordlist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercises\n",
    "\n",
    "- read a file, use regex and string functions to do sth, write the file\n",
    "- read multiple files and do the same"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Natural Language Processing demo\n",
    "\n",
    "### Named Entity Recognition \n",
    "\n",
    "Named-Entity Recognition is that task of Natural Language Processing that tries to define automatic ways to recognize and label Named Entities in texts.\n",
    "\n",
    "In their manual of NLP, [Jurafsky and Martin](https://web.stanford.edu/~jurafsky/slp3/21.pdf) define Named Entities as: \"roughly speaking, anything that can be referred to with a proper name: a person, a location, an organization\".\n",
    "\n",
    "Three important things can be seen from this example:\n",
    "- NER is not just about recognizing names, it is also about classifying them (distinguishing e.g. persons from locations, organizations, time expressions and money amounts);\n",
    "- NER is already a powerful tool to extract information from texts and attempt something like an automatic summarization;\n",
    "- although TIME EXPRESSIONS and AMOUNT of MONEY are not at all prototypical entities with proper names, your goal of information extraction (e.g. understanding how prices in airline fares fluctuate) might require you to extract and classify them too! For the sake of this NER application, 6$ or Thursday become Named Entities too, like San Francisco or Tim Wagner.\n",
    "\n",
    "A NER task can be broken down into a sequence of operations in which you take all the \"units\" of your text and then, for each of them, you:\n",
    "\n",
    "- decide whether it is a name or not\n",
    "- decide what type of entity it is (e.g. Person, Location, or No-Entity)\n",
    "- decide whether it is a part of name (e.g. \"New\" in \"New York City\")\n",
    "\n",
    "The second task requires you to define a tagset (i.e. a list of tags to assign) and, consequently, a taxonomy of the entities that you want to consider. Some popular options:\n",
    "\n",
    "- 3-tag system: PER, LOC, ORG, with \"O\" being used for no entity\n",
    "- a binary classification: Entity, O\n",
    "\n",
    "\n",
    "We will need the [Natural Language ToolKit (NLTK)](http://www.nltk.org/), the [Classical Language ToolKit (CLTK)](http://cltk.org/), and some local libraries that are used in this notebook. `NLTK` is the comprehensive library that collects all the most important resources (corpora and tools) for Natural Language Processing in Python. `CLTK` is a sort of \\\"spin off\\\" that provide specialized tools and access to corpora for ancient languages, including Latin and Greek.\n",
    "\n",
    "**Remember!** Everything that is not part of Python's [Standard Library](https://docs.python.org/3/library/index.html) must be installed before it can be imported. Most likely, you will have to install both `NLTK` and `CLTK` to your machine before you can run the following lines.\n",
    "\n",
    "Let's say, we want to recognize (and possibly, classify) proper names in Caesar's Bellum Gallicum. The first thing we need is a digitized version of the text!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sunoikisis Digital Classics, semester 2016-17\n",
    "[Named Entity Extraction I](https://github.com/SunoikisisDC/SunoikisisDC-2016-2017/wiki/Named-Entity-Extraction-I), Matteo Romanello and Francesco Mambrini (Deutsches Archäologisches Institut, Berlin)\n",
    "\n",
    "*Example of three methods for extracting named entities from a Latin text:*\n",
    "https://github.com/mromanello/SunoikisisDC_NER/blob/master/Sunoikisis%20-%20Named%20Entity%20Extraction%201b.ipynb\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part-of-Speech tagging\n",
    "\n",
    "\n",
    "Sunoikisis Digital Classics Spring/Summer 2018\n",
    "[Session 2. The Classical Language Toolkit (CLTK)](https://github.com/SunoikisisDC/SunoikisisDC-2017-2018/wiki/The-Classical-Language-Toolkit-(CLTK)), Patrick J. Burns (New York University, Institute for the Study of the Ancient World)\n",
    "\n",
    "*Building a text analysis pipeline with the CLTK:*\n",
    "https://hub.mybinder.org/user/diyclassics-sunoikisisdc-cltk-65rywfq9/notebooks/cltk-pipeline.ipynb\n",
    "\n",
    "\n",
    "\n",
    "[Session 9. Python 2: Part of Speech tagging and lemmatisation](https://github.com/SunoikisisDC/SunoikisisDC-2017-2018/wiki/Python-2:-Part-of-Speech-tagging-and-lemmatisation), Barbara McGillivray (The Alan Turing Institute / University of Cambridge), Matteo Romanello (École polytechnique fédérale de Lausanne)\n",
    "\n",
    "*Part of Speech tagging and lemmatisation with Python:*\n",
    "https://github.com/SunoikisisDC/SunoikisisDC-2017-2018/blob/master/Spring%202018%20-%20Slides%20and%20Materials/Session_9_PoSTagging_Lemmatization.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  },
  "toc": {
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "873px",
    "left": "0px",
    "right": "1202px",
    "top": "110px",
    "width": "364px"
   },
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}