nlp-enrichment/NLP-Enrichment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1aaf3791",
   "metadata": {},
   "source": [
    "# Enrichment of the Corpus\n",
    "\n",
    "* now that we have homogenized, we prepare the corpus i.e. we pre-process more for in-depth analyses \n",
    "* for this we:\n",
    "  * tokenize\n",
    "  * lemmatize\n",
    "  * pos-tag\n",
    "* This can be done with multiple **libraries** (*what concepts are known, do we need the notion of this, I'd say yes!) \n",
    "  * spacy\n",
    "  * stanford-core-nlp\n",
    "  * nltk\n",
    "* Pre-Processing is language specific"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be81998d",
   "metadata": {},
   "source": [
    "## 1. Read in the txt data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edf1d9f3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5acf822f",
   "metadata": {},
   "source": [
    "#### 1.1 Set path to corpus directory \n",
    "* Replace ./data/txt/ with path to the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44fa6bd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# or point to standard here?\n",
    "corpus_dir = Path(r\"../data/txt/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fd98045",
   "metadata": {},
   "source": [
    "#### 1.2 Read in the files from the directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e041af15",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import OrderedDict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3572c70",
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_corpus_linewise(corpus_dir: Path) -> OrderedDict[str,str]:\n",
    "    corpus = OrderedDict()\n",
    "    for filepath in corpus_dir.iterdir():\n",
    "        if filepath.is_file():\n",
    "            text = filepath.read_text()\n",
    "            #text = text.replace(\"\\n\", \" \")\n",
    "            corpus[filepath.name] = text\n",
    "    return corpus"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "172c1612",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus = read_corpus_linewise(corpus_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4e804de",
   "metadata": {},
   "source": [
    "**Prüfen**: Wie viele Dateien wurden eingelesen?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ac06177",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(corpus))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "867c240c",
   "metadata": {},
   "source": [
    "#### 1.3 Read in metadata "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90551a73",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6b91a52",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_dir = Path(r\"../data/metadata/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e76f723",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_filepath =  metadata_dir / Path(\"MVP-Test-Korpus_Metadata.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9f0f0d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_df = pd.read_csv(metadata_filepath, sep=\";\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db2e5b6f",
   "metadata": {},
   "source": [
    "**Prüfen**: Wie sehen unsere Metadaten aus?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e569f6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5f6d22bb",
   "metadata": {},
   "source": [
    "## 2.Worthäufigkeit mit lazy tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d07258cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_texts = \" \".join(corpus.values())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6984c038",
   "metadata": {},
   "outputs": [],
   "source": [
    "words = all_texts.split(\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "694298bd-8e8f-4570-b053-f3e7b42f77d8",
   "metadata": {},
   "source": [
    "**Prüfen**: Wie sieht die Wortliste aus?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b37fa42-ae7a-4bca-8b01-36cd1444e31a",
   "metadata": {},
   "outputs": [],
   "source": [
    "words[50:60]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2d542d5",
   "metadata": {},
   "source": [
    "Wie viele Wörter gibt es insgesamt?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3ee91df",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(words)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b971409",
   "metadata": {},
   "source": [
    "Welche Wörter kommen wie oft vor? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b3a4530",
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dc3cbbc",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_frequencies = Counter(words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4234dfea",
   "metadata": {},
   "outputs": [],
   "source": [
    "chosen_word = input(\"Geben Sie ein Wort ein, für welches die Häufigkeit angezeigt wird: \")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a590df8",
   "metadata": {},
   "outputs": [],
   "source": [
    "word_frequencies[chosen_word]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b158ecb",
   "metadata": {},
   "source": [
    "## 3. Load NLP Library\n",
    "* Do we need to install first? Probably not – do we want to show how we would install?\n",
    "\n",
    "Overview of spacy model available [here](https://spacy.io/models) \\\n",
    "\n",
    "Load language specific model (selection):\n",
    "* German: 'de_core_news_sm'\n",
    "* English: 'en_core_news_sm'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a93e2719",
   "metadata": {},
   "source": [
    "### 3.1 Load library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33b7d428",
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11e71e97",
   "metadata": {},
   "outputs": [],
   "source": [
    "! python -m spacy download de_core_news_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c52bea2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('de_core_news_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "672fb872",
   "metadata": {},
   "source": [
    "### 3.2 Setting up the pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6fb49918",
   "metadata": {},
   "outputs": [],
   "source": [
    "disable_components = ['ner', 'morphologizer', 'attribute_ruler']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13bec70c",
   "metadata": {},
   "source": [
    "### 3.3 Annotate texts and extract token, lemma, pos"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b2e3fc7-0b60-4c1f-96d2-a77da7ae388c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from time import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8011e3f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "took_per_text = []\n",
    "\n",
    "corpus_annotated = {}\n",
    "filename_list = list(corpus.keys())\n",
    "current = time()\n",
    "for i, doc in enumerate(nlp.pipe(corpus.values(), disable=disable_components)):\n",
    "    before = current\n",
    "    current = time()\n",
    "    took_per_text.append(current - before)\n",
    "    annotated_text = {}\n",
    "    annotated_text['Token'] = [tok.text for tok in doc]\n",
    "    annotated_text['Lemma'] = [tok.lemma_ for tok in doc]\n",
    "    annotated_text['PoS'] = [tok.tag_ for tok in doc]\n",
    "    \n",
    "    sentences = []\n",
    "    sentence_idx = -1\n",
    "    for token in doc:\n",
    "        if token.is_sent_start:\n",
    "            sentence_idx += 1\n",
    "        sentences.append(sentence_idx)\n",
    "    annotated_text['Sentence_idx'] = sentences\n",
    "    \n",
    "    corpus_annotated[filename_list[i]] = pd.DataFrame(annotated_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8383b4cf",
   "metadata": {},
   "source": [
    "#### Wie lange hat das Annotieren gedauert?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "780cbeac-5e2b-451d-8d76-355043fd8073",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d10ac03-b41c-45e2-ab90-7b7ceda3936c",
   "metadata": {},
   "source": [
    "Durschnittlich pro Text in Sekunden:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aca4b148-6cec-4bb2-a440-41622466367e",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.mean(took_per_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "33a2de3d-e802-4c54-be6f-264cc08e8798",
   "metadata": {},
   "source": [
    "Alle Texte zusammen in Sekunden:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2b4f043-0057-42af-912f-491d290b64b6",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.sum(took_per_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6b7e007-85f4-4831-83ec-cff9a78e6360",
   "metadata": {},
   "source": [
    "**Prüfen**: Länge des annotierten Korpus gleich Länge des Originalkorpus?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "384c72bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(corpus_annotated)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2465ba08",
   "metadata": {},
   "source": [
    "**Prüfen**: Wie sieht die Annotation aus?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03ae155a",
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_annotated[filename_list[0]].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc8160c5",
   "metadata": {},
   "source": [
    "### 3.4 Worthäufigkeit mit echter Tokenization   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c6a224b",
   "metadata": {},
   "outputs": [],
   "source": [
    "all_words_tokenized = [word for text in corpus_annotated.values() for word in text.Token]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3783a3bf",
   "metadata": {},
   "outputs": [],
   "source": [
    "len(all_words_tokenized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f560071",
   "metadata": {},
   "outputs": [],
   "source": [
    "words_tokenized_frequencies = Counter(all_words_tokenized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65c8b2de",
   "metadata": {},
   "outputs": [],
   "source": [
    "words_tokenized_frequencies[chosen_word]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca9a3fb1",
   "metadata": {},
   "source": [
    "## 4. Metadaten ausweiten"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "25d4b5c7",
   "metadata": {},
   "source": [
    "### 4.1 Metadaten sammeln\n",
    "* Anzahl Lemmata\n",
    "* Anzahl unique Lemmata\n",
    "* Anzahl Sätze\n",
    "* Durschnittliche Satzlänge "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1f0e7c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "collected_metadata_extension = []\n",
    "for filename, annotated_text in corpus_annotated.items():\n",
    "    metadata_extension = {}\n",
    "    metadata_extension['Filename'] = filename\n",
    "    metadata_extension['Lemma_Count'] = len(annotated_text) - 1\n",
    "    metadata_extension['Lemma_Count_Unique'] = len(set(annotated_text.Lemma))\n",
    "    metadata_extension['Sentence_Count'] = annotated_text.Sentence_idx.iloc[-1]\n",
    "    metadata_extension['Sentence_Length_Avg'] = annotated_text.groupby('Sentence_idx').Lemma.count().mean()\n",
    "    collected_metadata_extension.append(metadata_extension)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7cf046f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_to_extend = pd.DataFrame(collected_metadata_extension)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46c45e81",
   "metadata": {},
   "source": [
    "### 4.2 Metadaten hinzufügen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cfb962bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b5bae9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_to_extend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f417386",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_df['Filename'] = metadata_df['Identifier'] + '-' + metadata_df['Date'].astype(str) + '-0-0-0-0.txt' "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4aef0265",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_extendend_df = pd.merge(metadata_df, metadata_to_extend, on=\"Filename\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6402923a",
   "metadata": {},
   "source": [
    "## 5. Ergebnisse speichern "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5da36fae",
   "metadata": {},
   "source": [
    "### 5.1 Annotiertes Korpus speichern"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3041d07",
   "metadata": {},
   "outputs": [],
   "source": [
    "result_dir = Path(r\"../data/conll\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3cc1fab",
   "metadata": {},
   "outputs": [],
   "source": [
    "for filepath, annotated_text in corpus_annotated.items():\n",
    "    filepath = Path(filepath)\n",
    "    output_path = result_dir / filepath.with_suffix(\".conll\")\n",
    "    annotated_text.to_csv(output_path, index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a22e66e6",
   "metadata": {},
   "source": [
    "### 5.2 Erweiterte Metadaten speichern  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bae891b",
   "metadata": {},
   "outputs": [],
   "source": [
    "metadata_extended_filename = Path(r\"MVP-Test-Korpus_Metadata-v02.csv\")\n",
    "metadata_extendend_df.to_csv(metadata_dir / metadata_extended_filename, index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}