SCI564_Assignment_4.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "SCI564_Assignment_4.ipynb",
      "provenance": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/tawfiqam/MI564/blob/main/SCI564_Assignment_4.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WomcSdHU9lU2"
      },
      "source": [
        "#Part I: Data Collection"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Qz-ch6U9rKY"
      },
      "source": [
        "# **Please forget about the Pushshift API. I will keep the code in case the API comes back to life (doubted). **\n",
        "\n",
        "**The dataset used in this analysis is [here](https://drive.google.com/file/d/1JAYNVL1mVI-a2EbDKvRl0U0-xuivQ_-R/view?usp=sharing). Download it and then use it in the analysis. Basically read it as a json file to a pandas dataframe. **\n",
        "\n",
        "[5 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zTybn7Ir-c6r"
      },
      "source": [
        "#A1."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AphMKa4--JO4"
      },
      "source": [
        "#Part II: TF-IDF"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wIBVvLlL-OZJ"
      },
      "source": [
        "Q2. Create a TFIDF vectorizer using ngrams between 1 and 3, and remove all english stop words using NLTK. [10 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KWY85r159k51"
      },
      "source": [
        "#A2."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CIn4__eY-gLR"
      },
      "source": [
        "Q3. Fit the vectorizer on the body column of the dataframe. [5 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uwVi5v97-eDT"
      },
      "source": [
        "#A3."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qkCPOOhZB7AA"
      },
      "source": [
        "Q4. Find the top features (without the weights) using the TF-IDF vector using ngrams betwen 1 and 3. In other words, show the top 20 tokens for 1-gram, 2-gram, and 3-grams [15 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gARQ30gWDn5k"
      },
      "source": [
        "#A4."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p5U5GUwaCgDQ"
      },
      "source": [
        "#Part III: LDA"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XyerVQskDd77"
      },
      "source": [
        "Q5.Now create the dataframe we will use for the LDA model. Each document should be the complete Reddit thread. The thread is grouped by the field link_id. Call this dataframe **df_p_threads** [5 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ShoLHjhyCS5-"
      },
      "source": [
        "#A5"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k3PAZKjrCjWY"
      },
      "source": [
        "Q6. In the first pass, clean the text by removing the newline character code, the aperasand character code, non-breaking space and, and zero-width space. [5 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "weaUt60TDUMT"
      },
      "source": [
        "#A6"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "95k_fYayDBUy"
      },
      "source": [
        "Q7.In the second pass, tokenize, remove stopwords and punctuation as well as urls (web links). I also want you to use the nltk lemmatizer to lemmatize the body [5 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CG3jaHYSDkBk"
      },
      "source": [
        "#A7"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Hklf8jUuEeKG"
      },
      "source": [
        "#Now that we have the cleaned text for each thread\n",
        "#documents, or docs represents each thread\n",
        "docs = df_p_threads.clean_text"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B02z_6UOEoDB"
      },
      "source": [
        "Q8.Create a gensim dictionary using the docs [5 points]\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uqSG7iQgEr0e"
      },
      "source": [
        "#A8"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oF33B0PAE1ni"
      },
      "source": [
        "Q9.Create the gensim corpus using the dictionary [5 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LHpOV_HvEyWF"
      },
      "source": [
        "#A9"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i6h4mRoXE8Ii"
      },
      "source": [
        "Q10.Create the id2word mapping between the token number and the words. Call this **id2word** [5 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-0iWQej2FOh6"
      },
      "source": [
        "#A10"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zY-dDdTSFZEX"
      },
      "source": [
        "Q11.As we did in the class example, I want you to run LDA models over the documents. Start with 10 topics, and with steps of 10 topics, the limit would be 110 topics. You should have a total of 10 models. [20 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "64YRTcCfFYVo"
      },
      "source": [
        "#A11"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DnS5fT-lF19y"
      },
      "source": [
        "Q12.Graph the coherence scores for all nine models and select the optimum topic model. [10 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o3cwpRWaGAw8"
      },
      "source": [
        "#A12"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dvjmd_37GNob"
      },
      "source": [
        "Q13.Print all the topics in the optimum LDA model [5 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TDmfLSJoGGYj"
      },
      "source": [
        "#A13"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HZOBRihhGkcM"
      },
      "source": [
        "Q14. [EXTRA points] Sort the models based on coherence scores and print them in order [10 points]"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HrWPfTAVGuEA"
      },
      "source": [
        "#A14"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}