07_Practice_Optimizing_DataFrames_and_Processing_in_Chunks.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "anaconda-cloud": {},
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.4.3"
    },
    "colab": {
      "name": "07-Practice-Optimizing-DataFrames-and-Processing-in-Chunks.ipynb",
      "provenance": [],
      "include_colab_link": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/lis-r-barreto/Data-Engineering/blob/main/07_Practice_Optimizing_DataFrames_and_Processing_in_Chunks.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9EZcEwBAS-6z"
      },
      "source": [
        "# Dataquest Guided Project: Optimizing Dataframes and Processing in Chunks\n",
        "\n",
        "We'll be working with financial lending data from Lending Club, a marketplace for personal loans that matches borrowers with investors.\n",
        "\n",
        "In this project, it is assumed that  that we only have 10 megabytes of memory available throughout this project, so you can practice the concepts of process large CSV as chunk and reduce memeory usage of chunks by analysis the data in chunk and set data types of columns accordingly.\n",
        "\n",
        "The official Dataquest reference solution is in GitHub repo [here](https://github.com/dataquestio/solutions/blob/master/Mission165Solutions.ipynb).\n",
        "\n",
        "My solution is slighthly different from the 'reference' solution.  As I try to keep the column data type (and thier size) the same across all the chunks and categorize category column using distinct values across all chunks instead of only distinct value in a chunk. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "o2v-8cNjS-65"
      },
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "import pprint as pp\n",
        "\n",
        "pd.options.display.max_columns = 99"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J9_CZRYDS-68"
      },
      "source": [
        "## Explore the first 5 rows  ... there is date columns\n",
        "\n",
        "Date columns is found, these columns should read as datetiem at later stage"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vOHzPOx3S-7A",
        "outputId": "4730b4ab-eea0-4dda-f9b9-e854e3ae64f5"
      },
      "source": [
        "loan=pd.read_csv('loans_2007.csv',nrows=5)\n",
        "\n",
        "loan"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>member_id</th>\n",
              "      <th>loan_amnt</th>\n",
              "      <th>funded_amnt</th>\n",
              "      <th>funded_amnt_inv</th>\n",
              "      <th>term</th>\n",
              "      <th>int_rate</th>\n",
              "      <th>installment</th>\n",
              "      <th>grade</th>\n",
              "      <th>sub_grade</th>\n",
              "      <th>emp_title</th>\n",
              "      <th>emp_length</th>\n",
              "      <th>home_ownership</th>\n",
              "      <th>annual_inc</th>\n",
              "      <th>verification_status</th>\n",
              "      <th>issue_d</th>\n",
              "      <th>loan_status</th>\n",
              "      <th>pymnt_plan</th>\n",
              "      <th>purpose</th>\n",
              "      <th>title</th>\n",
              "      <th>zip_code</th>\n",
              "      <th>addr_state</th>\n",
              "      <th>dti</th>\n",
              "      <th>delinq_2yrs</th>\n",
              "      <th>earliest_cr_line</th>\n",
              "      <th>inq_last_6mths</th>\n",
              "      <th>open_acc</th>\n",
              "      <th>pub_rec</th>\n",
              "      <th>revol_bal</th>\n",
              "      <th>revol_util</th>\n",
              "      <th>total_acc</th>\n",
              "      <th>initial_list_status</th>\n",
              "      <th>out_prncp</th>\n",
              "      <th>out_prncp_inv</th>\n",
              "      <th>total_pymnt</th>\n",
              "      <th>total_pymnt_inv</th>\n",
              "      <th>total_rec_prncp</th>\n",
              "      <th>total_rec_int</th>\n",
              "      <th>total_rec_late_fee</th>\n",
              "      <th>recoveries</th>\n",
              "      <th>collection_recovery_fee</th>\n",
              "      <th>last_pymnt_d</th>\n",
              "      <th>last_pymnt_amnt</th>\n",
              "      <th>last_credit_pull_d</th>\n",
              "      <th>collections_12_mths_ex_med</th>\n",
              "      <th>policy_code</th>\n",
              "      <th>application_type</th>\n",
              "      <th>acc_now_delinq</th>\n",
              "      <th>chargeoff_within_12_mths</th>\n",
              "      <th>delinq_amnt</th>\n",
              "      <th>pub_rec_bankruptcies</th>\n",
              "      <th>tax_liens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1077501</td>\n",
              "      <td>1296599.0</td>\n",
              "      <td>5000.0</td>\n",
              "      <td>5000.0</td>\n",
              "      <td>4975.0</td>\n",
              "      <td>36 months</td>\n",
              "      <td>10.65%</td>\n",
              "      <td>162.87</td>\n",
              "      <td>B</td>\n",
              "      <td>B2</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10+ years</td>\n",
              "      <td>RENT</td>\n",
              "      <td>24000.0</td>\n",
              "      <td>Verified</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>Fully Paid</td>\n",
              "      <td>n</td>\n",
              "      <td>credit_card</td>\n",
              "      <td>Computer</td>\n",
              "      <td>860xx</td>\n",
              "      <td>AZ</td>\n",
              "      <td>27.65</td>\n",
              "      <td>0.0</td>\n",
              "      <td>Jan-1985</td>\n",
              "      <td>1.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>13648.0</td>\n",
              "      <td>83.7%</td>\n",
              "      <td>9.0</td>\n",
              "      <td>f</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>5863.155187</td>\n",
              "      <td>5833.84</td>\n",
              "      <td>5000.00</td>\n",
              "      <td>863.16</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>Jan-2015</td>\n",
              "      <td>171.62</td>\n",
              "      <td>Jun-2016</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>INDIVIDUAL</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1077430</td>\n",
              "      <td>1314167.0</td>\n",
              "      <td>2500.0</td>\n",
              "      <td>2500.0</td>\n",
              "      <td>2500.0</td>\n",
              "      <td>60 months</td>\n",
              "      <td>15.27%</td>\n",
              "      <td>59.83</td>\n",
              "      <td>C</td>\n",
              "      <td>C4</td>\n",
              "      <td>Ryder</td>\n",
              "      <td>&lt; 1 year</td>\n",
              "      <td>RENT</td>\n",
              "      <td>30000.0</td>\n",
              "      <td>Source Verified</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>Charged Off</td>\n",
              "      <td>n</td>\n",
              "      <td>car</td>\n",
              "      <td>bike</td>\n",
              "      <td>309xx</td>\n",
              "      <td>GA</td>\n",
              "      <td>1.00</td>\n",
              "      <td>0.0</td>\n",
              "      <td>Apr-1999</td>\n",
              "      <td>5.0</td>\n",
              "      <td>3.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1687.0</td>\n",
              "      <td>9.4%</td>\n",
              "      <td>4.0</td>\n",
              "      <td>f</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>1008.710000</td>\n",
              "      <td>1008.71</td>\n",
              "      <td>456.46</td>\n",
              "      <td>435.17</td>\n",
              "      <td>0.00</td>\n",
              "      <td>117.08</td>\n",
              "      <td>1.11</td>\n",
              "      <td>Apr-2013</td>\n",
              "      <td>119.66</td>\n",
              "      <td>Sep-2013</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>INDIVIDUAL</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1077175</td>\n",
              "      <td>1313524.0</td>\n",
              "      <td>2400.0</td>\n",
              "      <td>2400.0</td>\n",
              "      <td>2400.0</td>\n",
              "      <td>36 months</td>\n",
              "      <td>15.96%</td>\n",
              "      <td>84.33</td>\n",
              "      <td>C</td>\n",
              "      <td>C5</td>\n",
              "      <td>NaN</td>\n",
              "      <td>10+ years</td>\n",
              "      <td>RENT</td>\n",
              "      <td>12252.0</td>\n",
              "      <td>Not Verified</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>Fully Paid</td>\n",
              "      <td>n</td>\n",
              "      <td>small_business</td>\n",
              "      <td>real estate business</td>\n",
              "      <td>606xx</td>\n",
              "      <td>IL</td>\n",
              "      <td>8.72</td>\n",
              "      <td>0.0</td>\n",
              "      <td>Nov-2001</td>\n",
              "      <td>2.0</td>\n",
              "      <td>2.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>2956.0</td>\n",
              "      <td>98.5%</td>\n",
              "      <td>10.0</td>\n",
              "      <td>f</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>3005.666844</td>\n",
              "      <td>3005.67</td>\n",
              "      <td>2400.00</td>\n",
              "      <td>605.67</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>Jun-2014</td>\n",
              "      <td>649.91</td>\n",
              "      <td>Jun-2016</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>INDIVIDUAL</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1076863</td>\n",
              "      <td>1277178.0</td>\n",
              "      <td>10000.0</td>\n",
              "      <td>10000.0</td>\n",
              "      <td>10000.0</td>\n",
              "      <td>36 months</td>\n",
              "      <td>13.49%</td>\n",
              "      <td>339.31</td>\n",
              "      <td>C</td>\n",
              "      <td>C1</td>\n",
              "      <td>AIR RESOURCES BOARD</td>\n",
              "      <td>10+ years</td>\n",
              "      <td>RENT</td>\n",
              "      <td>49200.0</td>\n",
              "      <td>Source Verified</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>Fully Paid</td>\n",
              "      <td>n</td>\n",
              "      <td>other</td>\n",
              "      <td>personel</td>\n",
              "      <td>917xx</td>\n",
              "      <td>CA</td>\n",
              "      <td>20.00</td>\n",
              "      <td>0.0</td>\n",
              "      <td>Feb-1996</td>\n",
              "      <td>1.0</td>\n",
              "      <td>10.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>5598.0</td>\n",
              "      <td>21%</td>\n",
              "      <td>37.0</td>\n",
              "      <td>f</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>12231.890000</td>\n",
              "      <td>12231.89</td>\n",
              "      <td>10000.00</td>\n",
              "      <td>2214.92</td>\n",
              "      <td>16.97</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>Jan-2015</td>\n",
              "      <td>357.48</td>\n",
              "      <td>Apr-2016</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>INDIVIDUAL</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1075358</td>\n",
              "      <td>1311748.0</td>\n",
              "      <td>3000.0</td>\n",
              "      <td>3000.0</td>\n",
              "      <td>3000.0</td>\n",
              "      <td>60 months</td>\n",
              "      <td>12.69%</td>\n",
              "      <td>67.79</td>\n",
              "      <td>B</td>\n",
              "      <td>B5</td>\n",
              "      <td>University Medical Group</td>\n",
              "      <td>1 year</td>\n",
              "      <td>RENT</td>\n",
              "      <td>80000.0</td>\n",
              "      <td>Source Verified</td>\n",
              "      <td>Dec-2011</td>\n",
              "      <td>Current</td>\n",
              "      <td>n</td>\n",
              "      <td>other</td>\n",
              "      <td>Personal</td>\n",
              "      <td>972xx</td>\n",
              "      <td>OR</td>\n",
              "      <td>17.94</td>\n",
              "      <td>0.0</td>\n",
              "      <td>Jan-1996</td>\n",
              "      <td>0.0</td>\n",
              "      <td>15.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>27783.0</td>\n",
              "      <td>53.9%</td>\n",
              "      <td>38.0</td>\n",
              "      <td>f</td>\n",
              "      <td>461.73</td>\n",
              "      <td>461.73</td>\n",
              "      <td>3581.120000</td>\n",
              "      <td>3581.12</td>\n",
              "      <td>2538.27</td>\n",
              "      <td>1042.85</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>0.00</td>\n",
              "      <td>Jun-2016</td>\n",
              "      <td>67.79</td>\n",
              "      <td>Jun-2016</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.0</td>\n",
              "      <td>INDIVIDUAL</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        id  member_id  loan_amnt  funded_amnt  funded_amnt_inv        term  \\\n",
              "0  1077501  1296599.0     5000.0       5000.0           4975.0   36 months   \n",
              "1  1077430  1314167.0     2500.0       2500.0           2500.0   60 months   \n",
              "2  1077175  1313524.0     2400.0       2400.0           2400.0   36 months   \n",
              "3  1076863  1277178.0    10000.0      10000.0          10000.0   36 months   \n",
              "4  1075358  1311748.0     3000.0       3000.0           3000.0   60 months   \n",
              "\n",
              "  int_rate  installment grade sub_grade                 emp_title emp_length  \\\n",
              "0   10.65%       162.87     B        B2                       NaN  10+ years   \n",
              "1   15.27%        59.83     C        C4                     Ryder   < 1 year   \n",
              "2   15.96%        84.33     C        C5                       NaN  10+ years   \n",
              "3   13.49%       339.31     C        C1       AIR RESOURCES BOARD  10+ years   \n",
              "4   12.69%        67.79     B        B5  University Medical Group     1 year   \n",
              "\n",
              "  home_ownership  annual_inc verification_status   issue_d  loan_status  \\\n",
              "0           RENT     24000.0            Verified  Dec-2011   Fully Paid   \n",
              "1           RENT     30000.0     Source Verified  Dec-2011  Charged Off   \n",
              "2           RENT     12252.0        Not Verified  Dec-2011   Fully Paid   \n",
              "3           RENT     49200.0     Source Verified  Dec-2011   Fully Paid   \n",
              "4           RENT     80000.0     Source Verified  Dec-2011      Current   \n",
              "\n",
              "  pymnt_plan         purpose                 title zip_code addr_state    dti  \\\n",
              "0          n     credit_card              Computer    860xx         AZ  27.65   \n",
              "1          n             car                  bike    309xx         GA   1.00   \n",
              "2          n  small_business  real estate business    606xx         IL   8.72   \n",
              "3          n           other              personel    917xx         CA  20.00   \n",
              "4          n           other              Personal    972xx         OR  17.94   \n",
              "\n",
              "   delinq_2yrs earliest_cr_line  inq_last_6mths  open_acc  pub_rec  revol_bal  \\\n",
              "0          0.0         Jan-1985             1.0       3.0      0.0    13648.0   \n",
              "1          0.0         Apr-1999             5.0       3.0      0.0     1687.0   \n",
              "2          0.0         Nov-2001             2.0       2.0      0.0     2956.0   \n",
              "3          0.0         Feb-1996             1.0      10.0      0.0     5598.0   \n",
              "4          0.0         Jan-1996             0.0      15.0      0.0    27783.0   \n",
              "\n",
              "  revol_util  total_acc initial_list_status  out_prncp  out_prncp_inv  \\\n",
              "0      83.7%        9.0                   f       0.00           0.00   \n",
              "1       9.4%        4.0                   f       0.00           0.00   \n",
              "2      98.5%       10.0                   f       0.00           0.00   \n",
              "3        21%       37.0                   f       0.00           0.00   \n",
              "4      53.9%       38.0                   f     461.73         461.73   \n",
              "\n",
              "    total_pymnt  total_pymnt_inv  total_rec_prncp  total_rec_int  \\\n",
              "0   5863.155187          5833.84          5000.00         863.16   \n",
              "1   1008.710000          1008.71           456.46         435.17   \n",
              "2   3005.666844          3005.67          2400.00         605.67   \n",
              "3  12231.890000         12231.89         10000.00        2214.92   \n",
              "4   3581.120000          3581.12          2538.27        1042.85   \n",
              "\n",
              "   total_rec_late_fee  recoveries  collection_recovery_fee last_pymnt_d  \\\n",
              "0                0.00        0.00                     0.00     Jan-2015   \n",
              "1                0.00      117.08                     1.11     Apr-2013   \n",
              "2                0.00        0.00                     0.00     Jun-2014   \n",
              "3               16.97        0.00                     0.00     Jan-2015   \n",
              "4                0.00        0.00                     0.00     Jun-2016   \n",
              "\n",
              "   last_pymnt_amnt last_credit_pull_d  collections_12_mths_ex_med  \\\n",
              "0           171.62           Jun-2016                         0.0   \n",
              "1           119.66           Sep-2013                         0.0   \n",
              "2           649.91           Jun-2016                         0.0   \n",
              "3           357.48           Apr-2016                         0.0   \n",
              "4            67.79           Jun-2016                         0.0   \n",
              "\n",
              "   policy_code application_type  acc_now_delinq  chargeoff_within_12_mths  \\\n",
              "0          1.0       INDIVIDUAL             0.0                       0.0   \n",
              "1          1.0       INDIVIDUAL             0.0                       0.0   \n",
              "2          1.0       INDIVIDUAL             0.0                       0.0   \n",
              "3          1.0       INDIVIDUAL             0.0                       0.0   \n",
              "4          1.0       INDIVIDUAL             0.0                       0.0   \n",
              "\n",
              "   delinq_amnt  pub_rec_bankruptcies  tax_liens  \n",
              "0          0.0                   0.0        0.0  \n",
              "1          0.0                   0.0        0.0  \n",
              "2          0.0                   0.0        0.0  \n",
              "3          0.0                   0.0        0.0  \n",
              "4          0.0                   0.0        0.0  "
            ]
          },
          "execution_count": 2,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0YaJqwsuS-7D"
      },
      "source": [
        "## Assume memory is limited ... control the chunk size\n",
        "It is assume only 10M of memroy is available, so we need to control the memory of each chunk below 5M.  It turn out chunksize of 3000 is within our memory limit."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JzeXVhsUS-7F",
        "outputId": "5a83ff74-e106-450e-ead2-a07bb80b1e34"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000)\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    print(loanChunk.memory_usage(deep=True).sum()/(1024*1024))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "4.649013519287109\n",
            "4.6447601318359375\n",
            "4.646517753601074\n",
            "4.647870063781738\n",
            "4.6440629959106445\n",
            "4.6459455490112305\n",
            "4.644536972045898\n",
            "4.646905899047852\n",
            "4.645031929016113\n",
            "4.645082473754883\n",
            "4.657794952392578\n",
            "4.6566619873046875\n",
            "4.663469314575195\n",
            "4.896910667419434\n",
            "0.8808088302612305\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RmQg8UwBS-7H"
      },
      "source": [
        "## Explore the no. of numeric and string (object) columns\n",
        "The initial steps is to check no. of numeric and string (object) columns in the dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Q-kr9ONJS-7J"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000)\n",
        "\n",
        "\n",
        "loanChunkNumTxtColCounts=pd.DataFrame(columns=['Num Counts','Text Counts'])\n",
        "for loanChunk in loanChunkIter:\n",
        "    \n",
        "    numColCount=len(loanChunk.select_dtypes(include='number').columns)\n",
        "    txtColCount=len(loanChunk.select_dtypes(include='object').columns)\n",
        "    loanChunkNumTxtColCounts.loc[len(loanChunkNumTxtColCounts)]=[numColCount,txtColCount]\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "9ww1TMHNS-7M",
        "outputId": "66084793-ec90-4b66-9671-014887bc3ec7"
      },
      "source": [
        "loanChunkNumTxtColCounts"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Num Counts</th>\n",
              "      <th>Text Counts</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>30</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>30</td>\n",
              "      <td>22</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Num Counts Text Counts\n",
              "0          31          21\n",
              "1          31          21\n",
              "2          31          21\n",
              "3          31          21\n",
              "4          31          21\n",
              "5          31          21\n",
              "6          31          21\n",
              "7          31          21\n",
              "8          31          21\n",
              "9          31          21\n",
              "10         31          21\n",
              "11         31          21\n",
              "12         31          21\n",
              "13         30          22\n",
              "14         30          22"
            ]
          },
          "execution_count": 5,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E97mYqOAS-7S"
      },
      "source": [
        "## Why number of numeric and text colum is not the same across all chunks ?\n",
        "\n",
        "From the above data exploration/analysis, it is found the number of numeric column and object column is not the same for the last 2 chunks is different from all other chunks, need to find out which column caused the problem."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fgk3hoGIS-7U"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000)\n",
        "\n",
        "\n",
        "loanChunkTypes=pd.DataFrame(columns=loan.columns)\n",
        "loanChunkUniqueValues=pd.DataFrame(columns=loan.columns)\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunkTypes.loc[len(loanChunkTypes)]=loanChunk.dtypes\n",
        "    \n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uQzX38mfS-7X",
        "outputId": "c71ea0ca-ab4c-46b3-8a9b-952794f6748c"
      },
      "source": [
        "loanChunkTypes"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>member_id</th>\n",
              "      <th>loan_amnt</th>\n",
              "      <th>funded_amnt</th>\n",
              "      <th>funded_amnt_inv</th>\n",
              "      <th>term</th>\n",
              "      <th>int_rate</th>\n",
              "      <th>installment</th>\n",
              "      <th>grade</th>\n",
              "      <th>sub_grade</th>\n",
              "      <th>emp_title</th>\n",
              "      <th>emp_length</th>\n",
              "      <th>home_ownership</th>\n",
              "      <th>annual_inc</th>\n",
              "      <th>verification_status</th>\n",
              "      <th>issue_d</th>\n",
              "      <th>loan_status</th>\n",
              "      <th>pymnt_plan</th>\n",
              "      <th>purpose</th>\n",
              "      <th>title</th>\n",
              "      <th>zip_code</th>\n",
              "      <th>addr_state</th>\n",
              "      <th>dti</th>\n",
              "      <th>delinq_2yrs</th>\n",
              "      <th>earliest_cr_line</th>\n",
              "      <th>inq_last_6mths</th>\n",
              "      <th>open_acc</th>\n",
              "      <th>pub_rec</th>\n",
              "      <th>revol_bal</th>\n",
              "      <th>revol_util</th>\n",
              "      <th>total_acc</th>\n",
              "      <th>initial_list_status</th>\n",
              "      <th>out_prncp</th>\n",
              "      <th>out_prncp_inv</th>\n",
              "      <th>total_pymnt</th>\n",
              "      <th>total_pymnt_inv</th>\n",
              "      <th>total_rec_prncp</th>\n",
              "      <th>total_rec_int</th>\n",
              "      <th>total_rec_late_fee</th>\n",
              "      <th>recoveries</th>\n",
              "      <th>collection_recovery_fee</th>\n",
              "      <th>last_pymnt_d</th>\n",
              "      <th>last_pymnt_amnt</th>\n",
              "      <th>last_credit_pull_d</th>\n",
              "      <th>collections_12_mths_ex_med</th>\n",
              "      <th>policy_code</th>\n",
              "      <th>application_type</th>\n",
              "      <th>acc_now_delinq</th>\n",
              "      <th>chargeoff_within_12_mths</th>\n",
              "      <th>delinq_amnt</th>\n",
              "      <th>pub_rec_bankruptcies</th>\n",
              "      <th>tax_liens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        id member_id loan_amnt funded_amnt funded_amnt_inv    term int_rate  \\\n",
              "0    int64   float64   float64     float64         float64  object   object   \n",
              "1    int64   float64   float64     float64         float64  object   object   \n",
              "2    int64   float64   float64     float64         float64  object   object   \n",
              "3    int64   float64   float64     float64         float64  object   object   \n",
              "4    int64   float64   float64     float64         float64  object   object   \n",
              "5    int64   float64   float64     float64         float64  object   object   \n",
              "6    int64   float64   float64     float64         float64  object   object   \n",
              "7    int64   float64   float64     float64         float64  object   object   \n",
              "8    int64   float64   float64     float64         float64  object   object   \n",
              "9    int64   float64   float64     float64         float64  object   object   \n",
              "10   int64   float64   float64     float64         float64  object   object   \n",
              "11   int64   float64   float64     float64         float64  object   object   \n",
              "12   int64   float64   float64     float64         float64  object   object   \n",
              "13  object   float64   float64     float64         float64  object   object   \n",
              "14  object   float64   float64     float64         float64  object   object   \n",
              "\n",
              "   installment   grade sub_grade emp_title emp_length home_ownership  \\\n",
              "0      float64  object    object    object     object         object   \n",
              "1      float64  object    object    object     object         object   \n",
              "2      float64  object    object    object     object         object   \n",
              "3      float64  object    object    object     object         object   \n",
              "4      float64  object    object    object     object         object   \n",
              "5      float64  object    object    object     object         object   \n",
              "6      float64  object    object    object     object         object   \n",
              "7      float64  object    object    object     object         object   \n",
              "8      float64  object    object    object     object         object   \n",
              "9      float64  object    object    object     object         object   \n",
              "10     float64  object    object    object     object         object   \n",
              "11     float64  object    object    object     object         object   \n",
              "12     float64  object    object    object     object         object   \n",
              "13     float64  object    object    object     object         object   \n",
              "14     float64  object    object    object     object         object   \n",
              "\n",
              "   annual_inc verification_status issue_d loan_status pymnt_plan purpose  \\\n",
              "0     float64              object  object      object     object  object   \n",
              "1     float64              object  object      object     object  object   \n",
              "2     float64              object  object      object     object  object   \n",
              "3     float64              object  object      object     object  object   \n",
              "4     float64              object  object      object     object  object   \n",
              "5     float64              object  object      object     object  object   \n",
              "6     float64              object  object      object     object  object   \n",
              "7     float64              object  object      object     object  object   \n",
              "8     float64              object  object      object     object  object   \n",
              "9     float64              object  object      object     object  object   \n",
              "10    float64              object  object      object     object  object   \n",
              "11    float64              object  object      object     object  object   \n",
              "12    float64              object  object      object     object  object   \n",
              "13    float64              object  object      object     object  object   \n",
              "14    float64              object  object      object     object  object   \n",
              "\n",
              "     title zip_code addr_state      dti delinq_2yrs earliest_cr_line  \\\n",
              "0   object   object     object  float64     float64           object   \n",
              "1   object   object     object  float64     float64           object   \n",
              "2   object   object     object  float64     float64           object   \n",
              "3   object   object     object  float64     float64           object   \n",
              "4   object   object     object  float64     float64           object   \n",
              "5   object   object     object  float64     float64           object   \n",
              "6   object   object     object  float64     float64           object   \n",
              "7   object   object     object  float64     float64           object   \n",
              "8   object   object     object  float64     float64           object   \n",
              "9   object   object     object  float64     float64           object   \n",
              "10  object   object     object  float64     float64           object   \n",
              "11  object   object     object  float64     float64           object   \n",
              "12  object   object     object  float64     float64           object   \n",
              "13  object   object     object  float64     float64           object   \n",
              "14  object   object     object  float64     float64           object   \n",
              "\n",
              "   inq_last_6mths open_acc  pub_rec revol_bal revol_util total_acc  \\\n",
              "0         float64  float64  float64   float64     object   float64   \n",
              "1         float64  float64  float64   float64     object   float64   \n",
              "2         float64  float64  float64   float64     object   float64   \n",
              "3         float64  float64  float64   float64     object   float64   \n",
              "4         float64  float64  float64   float64     object   float64   \n",
              "5         float64  float64  float64   float64     object   float64   \n",
              "6         float64  float64  float64   float64     object   float64   \n",
              "7         float64  float64  float64   float64     object   float64   \n",
              "8         float64  float64  float64   float64     object   float64   \n",
              "9         float64  float64  float64   float64     object   float64   \n",
              "10        float64  float64  float64   float64     object   float64   \n",
              "11        float64  float64  float64   float64     object   float64   \n",
              "12        float64  float64  float64   float64     object   float64   \n",
              "13        float64  float64  float64   float64     object   float64   \n",
              "14        float64  float64  float64   float64     object   float64   \n",
              "\n",
              "   initial_list_status out_prncp out_prncp_inv total_pymnt total_pymnt_inv  \\\n",
              "0               object   float64       float64     float64         float64   \n",
              "1               object   float64       float64     float64         float64   \n",
              "2               object   float64       float64     float64         float64   \n",
              "3               object   float64       float64     float64         float64   \n",
              "4               object   float64       float64     float64         float64   \n",
              "5               object   float64       float64     float64         float64   \n",
              "6               object   float64       float64     float64         float64   \n",
              "7               object   float64       float64     float64         float64   \n",
              "8               object   float64       float64     float64         float64   \n",
              "9               object   float64       float64     float64         float64   \n",
              "10              object   float64       float64     float64         float64   \n",
              "11              object   float64       float64     float64         float64   \n",
              "12              object   float64       float64     float64         float64   \n",
              "13              object   float64       float64     float64         float64   \n",
              "14              object   float64       float64     float64         float64   \n",
              "\n",
              "   total_rec_prncp total_rec_int total_rec_late_fee recoveries  \\\n",
              "0          float64       float64            float64    float64   \n",
              "1          float64       float64            float64    float64   \n",
              "2          float64       float64            float64    float64   \n",
              "3          float64       float64            float64    float64   \n",
              "4          float64       float64            float64    float64   \n",
              "5          float64       float64            float64    float64   \n",
              "6          float64       float64            float64    float64   \n",
              "7          float64       float64            float64    float64   \n",
              "8          float64       float64            float64    float64   \n",
              "9          float64       float64            float64    float64   \n",
              "10         float64       float64            float64    float64   \n",
              "11         float64       float64            float64    float64   \n",
              "12         float64       float64            float64    float64   \n",
              "13         float64       float64            float64    float64   \n",
              "14         float64       float64            float64    float64   \n",
              "\n",
              "   collection_recovery_fee last_pymnt_d last_pymnt_amnt last_credit_pull_d  \\\n",
              "0                  float64       object         float64             object   \n",
              "1                  float64       object         float64             object   \n",
              "2                  float64       object         float64             object   \n",
              "3                  float64       object         float64             object   \n",
              "4                  float64       object         float64             object   \n",
              "5                  float64       object         float64             object   \n",
              "6                  float64       object         float64             object   \n",
              "7                  float64       object         float64             object   \n",
              "8                  float64       object         float64             object   \n",
              "9                  float64       object         float64             object   \n",
              "10                 float64       object         float64             object   \n",
              "11                 float64       object         float64             object   \n",
              "12                 float64       object         float64             object   \n",
              "13                 float64       object         float64             object   \n",
              "14                 float64       object         float64             object   \n",
              "\n",
              "   collections_12_mths_ex_med policy_code application_type acc_now_delinq  \\\n",
              "0                     float64     float64           object        float64   \n",
              "1                     float64     float64           object        float64   \n",
              "2                     float64     float64           object        float64   \n",
              "3                     float64     float64           object        float64   \n",
              "4                     float64     float64           object        float64   \n",
              "5                     float64     float64           object        float64   \n",
              "6                     float64     float64           object        float64   \n",
              "7                     float64     float64           object        float64   \n",
              "8                     float64     float64           object        float64   \n",
              "9                     float64     float64           object        float64   \n",
              "10                    float64     float64           object        float64   \n",
              "11                    float64     float64           object        float64   \n",
              "12                    float64     float64           object        float64   \n",
              "13                    float64     float64           object        float64   \n",
              "14                    float64     float64           object        float64   \n",
              "\n",
              "   chargeoff_within_12_mths delinq_amnt pub_rec_bankruptcies tax_liens  \n",
              "0                   float64     float64              float64   float64  \n",
              "1                   float64     float64              float64   float64  \n",
              "2                   float64     float64              float64   float64  \n",
              "3                   float64     float64              float64   float64  \n",
              "4                   float64     float64              float64   float64  \n",
              "5                   float64     float64              float64   float64  \n",
              "6                   float64     float64              float64   float64  \n",
              "7                   float64     float64              float64   float64  \n",
              "8                   float64     float64              float64   float64  \n",
              "9                   float64     float64              float64   float64  \n",
              "10                  float64     float64              float64   float64  \n",
              "11                  float64     float64              float64   float64  \n",
              "12                  float64     float64              float64   float64  \n",
              "13                  float64     float64              float64   float64  \n",
              "14                  float64     float64              float64   float64  "
            ]
          },
          "execution_count": 7,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hPuOj5k-S-7Z"
      },
      "source": [
        "## What's wrong with the Id column ?\n",
        "\n",
        "From the above data exploration/analysis, it is found the Id column in last 2 chunk is not number (int64), need further explore the reason."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JFF9H7N2S-7a",
        "outputId": "3bab8438-4e74-4b45-97a1-2053bef218e3"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000)\n",
        "\n",
        "for chunkNum,loanChunk in enumerate(loanChunkIter):\n",
        "    if loanChunk['id'].dtypes=='object':\n",
        "        print('Chunk {} has problem for this row=>'.format(chunkNum))\n",
        "        strIDRow=loanChunk[loanChunk['id'].str.isdigit()==False].index\n",
        "        print(loanChunk.loc[list(strIDRow)])\n",
        "    else:\n",
        "        print('Chunk {} no problem'.format(chunkNum))\n"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Chunk 0 no problem\n",
            "Chunk 1 no problem\n",
            "Chunk 2 no problem\n",
            "Chunk 3 no problem\n",
            "Chunk 4 no problem\n",
            "Chunk 5 no problem\n",
            "Chunk 6 no problem\n",
            "Chunk 7 no problem\n",
            "Chunk 8 no problem\n",
            "Chunk 9 no problem\n",
            "Chunk 10 no problem\n",
            "Chunk 11 no problem\n",
            "Chunk 12 no problem\n",
            "Chunk 13 has problem for this row=>\n",
            "                                             id  member_id  loan_amnt  \\\n",
            "39786  Loans that do not meet the credit policy        NaN        NaN   \n",
            "\n",
            "       funded_amnt  funded_amnt_inv term int_rate  installment grade  \\\n",
            "39786          NaN              NaN  NaN      NaN          NaN   NaN   \n",
            "\n",
            "      sub_grade emp_title emp_length home_ownership  annual_inc  \\\n",
            "39786       NaN       NaN        NaN            NaN         NaN   \n",
            "\n",
            "      verification_status issue_d loan_status pymnt_plan purpose title  \\\n",
            "39786                 NaN     NaN         NaN        NaN     NaN   NaN   \n",
            "\n",
            "      zip_code addr_state  dti  delinq_2yrs earliest_cr_line  inq_last_6mths  \\\n",
            "39786      NaN        NaN  NaN          NaN              NaN             NaN   \n",
            "\n",
            "       open_acc  pub_rec  revol_bal revol_util  total_acc initial_list_status  \\\n",
            "39786       NaN      NaN        NaN        NaN        NaN                 NaN   \n",
            "\n",
            "       out_prncp  out_prncp_inv  total_pymnt  total_pymnt_inv  \\\n",
            "39786        NaN            NaN          NaN              NaN   \n",
            "\n",
            "       total_rec_prncp  total_rec_int  total_rec_late_fee  recoveries  \\\n",
            "39786              NaN            NaN                 NaN         NaN   \n",
            "\n",
            "       collection_recovery_fee last_pymnt_d  last_pymnt_amnt  \\\n",
            "39786                      NaN          NaN              NaN   \n",
            "\n",
            "      last_credit_pull_d  collections_12_mths_ex_med  policy_code  \\\n",
            "39786                NaN                         NaN          NaN   \n",
            "\n",
            "      application_type  acc_now_delinq  chargeoff_within_12_mths  delinq_amnt  \\\n",
            "39786              NaN             NaN                       NaN          NaN   \n",
            "\n",
            "       pub_rec_bankruptcies  tax_liens  \n",
            "39786                   NaN        NaN  \n",
            "Chunk 14 has problem for this row=>\n",
            "                                                    id  member_id  loan_amnt  \\\n",
            "42536  Total amount funded in policy code 1: 471701350        NaN        NaN   \n",
            "42537          Total amount funded in policy code 2: 0        NaN        NaN   \n",
            "\n",
            "       funded_amnt  funded_amnt_inv term int_rate  installment grade  \\\n",
            "42536          NaN              NaN  NaN      NaN          NaN   NaN   \n",
            "42537          NaN              NaN  NaN      NaN          NaN   NaN   \n",
            "\n",
            "      sub_grade emp_title emp_length home_ownership  annual_inc  \\\n",
            "42536       NaN       NaN        NaN            NaN         NaN   \n",
            "42537       NaN       NaN        NaN            NaN         NaN   \n",
            "\n",
            "      verification_status issue_d loan_status pymnt_plan purpose title  \\\n",
            "42536                 NaN     NaN         NaN        NaN     NaN   NaN   \n",
            "42537                 NaN     NaN         NaN        NaN     NaN   NaN   \n",
            "\n",
            "      zip_code addr_state  dti  delinq_2yrs earliest_cr_line  inq_last_6mths  \\\n",
            "42536      NaN        NaN  NaN          NaN              NaN             NaN   \n",
            "42537      NaN        NaN  NaN          NaN              NaN             NaN   \n",
            "\n",
            "       open_acc  pub_rec  revol_bal revol_util  total_acc initial_list_status  \\\n",
            "42536       NaN      NaN        NaN        NaN        NaN                 NaN   \n",
            "42537       NaN      NaN        NaN        NaN        NaN                 NaN   \n",
            "\n",
            "       out_prncp  out_prncp_inv  total_pymnt  total_pymnt_inv  \\\n",
            "42536        NaN            NaN          NaN              NaN   \n",
            "42537        NaN            NaN          NaN              NaN   \n",
            "\n",
            "       total_rec_prncp  total_rec_int  total_rec_late_fee  recoveries  \\\n",
            "42536              NaN            NaN                 NaN         NaN   \n",
            "42537              NaN            NaN                 NaN         NaN   \n",
            "\n",
            "       collection_recovery_fee last_pymnt_d  last_pymnt_amnt  \\\n",
            "42536                      NaN          NaN              NaN   \n",
            "42537                      NaN          NaN              NaN   \n",
            "\n",
            "      last_credit_pull_d  collections_12_mths_ex_med  policy_code  \\\n",
            "42536                NaN                         NaN          NaN   \n",
            "42537                NaN                         NaN          NaN   \n",
            "\n",
            "      application_type  acc_now_delinq  chargeoff_within_12_mths  delinq_amnt  \\\n",
            "42536              NaN             NaN                       NaN          NaN   \n",
            "42537              NaN             NaN                       NaN          NaN   \n",
            "\n",
            "       pub_rec_bankruptcies  tax_liens  \n",
            "42536                   NaN        NaN  \n",
            "42537                   NaN        NaN  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FytRJBjjS-7c"
      },
      "source": [
        "## 3 Rows are not containing loan information\n",
        "\n",
        "From the above analysis\n",
        "-  1 row in Chunk 13 is subheader, all other columns are NaN\n",
        "-  2 rows in Chunk 14 are subtotals, which are not individual loan record\n",
        "\n",
        "These rows needed to be removed.  Add A small function to remove \"non-loan\" rows in Chunk\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "212yj-c4S-7f"
      },
      "source": [
        "def removeProblematicRow(loanChunk):\n",
        "    if loanChunk['id'].dtypes=='object':\n",
        "        strIDRow=loanChunk[loanChunk['id'].str.isdigit()==False].index\n",
        "        loanChunk.drop(list(strIDRow),axis=0,inplace=True)\n",
        "        loanChunk['id']=loanChunk['id'].astype('int64')\n",
        "    return loanChunk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TewK2wrJS-7g"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000)\n",
        "\n",
        "loanChunkNumTxtColCounts=pd.DataFrame(columns=['Num Counts','Text Counts'])\n",
        "loanChunkTypes=pd.DataFrame(columns=loan.columns)\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    \n",
        "    numColCount=len(loanChunk.select_dtypes(include='number').columns)\n",
        "    txtColCount=len(loanChunk.select_dtypes(include='object').columns)\n",
        "    loanChunkNumTxtColCounts.loc[len(loanChunkNumTxtColCounts)]=[numColCount,txtColCount]\n",
        "    \n",
        "    loanChunkTypes.loc[len(loanChunkTypes)]=loanChunk.dtypes\n",
        "    "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K8cvBoNJS-7k",
        "outputId": "d6188c2f-c3ce-4b6b-be7c-d311c9ea0508"
      },
      "source": [
        "loanChunkNumTxtColCounts"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Num Counts</th>\n",
              "      <th>Text Counts</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>31</td>\n",
              "      <td>21</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   Num Counts Text Counts\n",
              "0          31          21\n",
              "1          31          21\n",
              "2          31          21\n",
              "3          31          21\n",
              "4          31          21\n",
              "5          31          21\n",
              "6          31          21\n",
              "7          31          21\n",
              "8          31          21\n",
              "9          31          21\n",
              "10         31          21\n",
              "11         31          21\n",
              "12         31          21\n",
              "13         31          21\n",
              "14         31          21"
            ]
          },
          "execution_count": 11,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ojp-KV7jS-7l",
        "outputId": "d10487a7-12ce-41d0-8c45-45bef93b71ec"
      },
      "source": [
        "loanChunkTypes"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>member_id</th>\n",
              "      <th>loan_amnt</th>\n",
              "      <th>funded_amnt</th>\n",
              "      <th>funded_amnt_inv</th>\n",
              "      <th>term</th>\n",
              "      <th>int_rate</th>\n",
              "      <th>installment</th>\n",
              "      <th>grade</th>\n",
              "      <th>sub_grade</th>\n",
              "      <th>emp_title</th>\n",
              "      <th>emp_length</th>\n",
              "      <th>home_ownership</th>\n",
              "      <th>annual_inc</th>\n",
              "      <th>verification_status</th>\n",
              "      <th>issue_d</th>\n",
              "      <th>loan_status</th>\n",
              "      <th>pymnt_plan</th>\n",
              "      <th>purpose</th>\n",
              "      <th>title</th>\n",
              "      <th>zip_code</th>\n",
              "      <th>addr_state</th>\n",
              "      <th>dti</th>\n",
              "      <th>delinq_2yrs</th>\n",
              "      <th>earliest_cr_line</th>\n",
              "      <th>inq_last_6mths</th>\n",
              "      <th>open_acc</th>\n",
              "      <th>pub_rec</th>\n",
              "      <th>revol_bal</th>\n",
              "      <th>revol_util</th>\n",
              "      <th>total_acc</th>\n",
              "      <th>initial_list_status</th>\n",
              "      <th>out_prncp</th>\n",
              "      <th>out_prncp_inv</th>\n",
              "      <th>total_pymnt</th>\n",
              "      <th>total_pymnt_inv</th>\n",
              "      <th>total_rec_prncp</th>\n",
              "      <th>total_rec_int</th>\n",
              "      <th>total_rec_late_fee</th>\n",
              "      <th>recoveries</th>\n",
              "      <th>collection_recovery_fee</th>\n",
              "      <th>last_pymnt_d</th>\n",
              "      <th>last_pymnt_amnt</th>\n",
              "      <th>last_credit_pull_d</th>\n",
              "      <th>collections_12_mths_ex_med</th>\n",
              "      <th>policy_code</th>\n",
              "      <th>application_type</th>\n",
              "      <th>acc_now_delinq</th>\n",
              "      <th>chargeoff_within_12_mths</th>\n",
              "      <th>delinq_amnt</th>\n",
              "      <th>pub_rec_bankruptcies</th>\n",
              "      <th>tax_liens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>object</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       id member_id loan_amnt funded_amnt funded_amnt_inv    term int_rate  \\\n",
              "0   int64   float64   float64     float64         float64  object   object   \n",
              "1   int64   float64   float64     float64         float64  object   object   \n",
              "2   int64   float64   float64     float64         float64  object   object   \n",
              "3   int64   float64   float64     float64         float64  object   object   \n",
              "4   int64   float64   float64     float64         float64  object   object   \n",
              "5   int64   float64   float64     float64         float64  object   object   \n",
              "6   int64   float64   float64     float64         float64  object   object   \n",
              "7   int64   float64   float64     float64         float64  object   object   \n",
              "8   int64   float64   float64     float64         float64  object   object   \n",
              "9   int64   float64   float64     float64         float64  object   object   \n",
              "10  int64   float64   float64     float64         float64  object   object   \n",
              "11  int64   float64   float64     float64         float64  object   object   \n",
              "12  int64   float64   float64     float64         float64  object   object   \n",
              "13  int64   float64   float64     float64         float64  object   object   \n",
              "14  int64   float64   float64     float64         float64  object   object   \n",
              "\n",
              "   installment   grade sub_grade emp_title emp_length home_ownership  \\\n",
              "0      float64  object    object    object     object         object   \n",
              "1      float64  object    object    object     object         object   \n",
              "2      float64  object    object    object     object         object   \n",
              "3      float64  object    object    object     object         object   \n",
              "4      float64  object    object    object     object         object   \n",
              "5      float64  object    object    object     object         object   \n",
              "6      float64  object    object    object     object         object   \n",
              "7      float64  object    object    object     object         object   \n",
              "8      float64  object    object    object     object         object   \n",
              "9      float64  object    object    object     object         object   \n",
              "10     float64  object    object    object     object         object   \n",
              "11     float64  object    object    object     object         object   \n",
              "12     float64  object    object    object     object         object   \n",
              "13     float64  object    object    object     object         object   \n",
              "14     float64  object    object    object     object         object   \n",
              "\n",
              "   annual_inc verification_status issue_d loan_status pymnt_plan purpose  \\\n",
              "0     float64              object  object      object     object  object   \n",
              "1     float64              object  object      object     object  object   \n",
              "2     float64              object  object      object     object  object   \n",
              "3     float64              object  object      object     object  object   \n",
              "4     float64              object  object      object     object  object   \n",
              "5     float64              object  object      object     object  object   \n",
              "6     float64              object  object      object     object  object   \n",
              "7     float64              object  object      object     object  object   \n",
              "8     float64              object  object      object     object  object   \n",
              "9     float64              object  object      object     object  object   \n",
              "10    float64              object  object      object     object  object   \n",
              "11    float64              object  object      object     object  object   \n",
              "12    float64              object  object      object     object  object   \n",
              "13    float64              object  object      object     object  object   \n",
              "14    float64              object  object      object     object  object   \n",
              "\n",
              "     title zip_code addr_state      dti delinq_2yrs earliest_cr_line  \\\n",
              "0   object   object     object  float64     float64           object   \n",
              "1   object   object     object  float64     float64           object   \n",
              "2   object   object     object  float64     float64           object   \n",
              "3   object   object     object  float64     float64           object   \n",
              "4   object   object     object  float64     float64           object   \n",
              "5   object   object     object  float64     float64           object   \n",
              "6   object   object     object  float64     float64           object   \n",
              "7   object   object     object  float64     float64           object   \n",
              "8   object   object     object  float64     float64           object   \n",
              "9   object   object     object  float64     float64           object   \n",
              "10  object   object     object  float64     float64           object   \n",
              "11  object   object     object  float64     float64           object   \n",
              "12  object   object     object  float64     float64           object   \n",
              "13  object   object     object  float64     float64           object   \n",
              "14  object   object     object  float64     float64           object   \n",
              "\n",
              "   inq_last_6mths open_acc  pub_rec revol_bal revol_util total_acc  \\\n",
              "0         float64  float64  float64   float64     object   float64   \n",
              "1         float64  float64  float64   float64     object   float64   \n",
              "2         float64  float64  float64   float64     object   float64   \n",
              "3         float64  float64  float64   float64     object   float64   \n",
              "4         float64  float64  float64   float64     object   float64   \n",
              "5         float64  float64  float64   float64     object   float64   \n",
              "6         float64  float64  float64   float64     object   float64   \n",
              "7         float64  float64  float64   float64     object   float64   \n",
              "8         float64  float64  float64   float64     object   float64   \n",
              "9         float64  float64  float64   float64     object   float64   \n",
              "10        float64  float64  float64   float64     object   float64   \n",
              "11        float64  float64  float64   float64     object   float64   \n",
              "12        float64  float64  float64   float64     object   float64   \n",
              "13        float64  float64  float64   float64     object   float64   \n",
              "14        float64  float64  float64   float64     object   float64   \n",
              "\n",
              "   initial_list_status out_prncp out_prncp_inv total_pymnt total_pymnt_inv  \\\n",
              "0               object   float64       float64     float64         float64   \n",
              "1               object   float64       float64     float64         float64   \n",
              "2               object   float64       float64     float64         float64   \n",
              "3               object   float64       float64     float64         float64   \n",
              "4               object   float64       float64     float64         float64   \n",
              "5               object   float64       float64     float64         float64   \n",
              "6               object   float64       float64     float64         float64   \n",
              "7               object   float64       float64     float64         float64   \n",
              "8               object   float64       float64     float64         float64   \n",
              "9               object   float64       float64     float64         float64   \n",
              "10              object   float64       float64     float64         float64   \n",
              "11              object   float64       float64     float64         float64   \n",
              "12              object   float64       float64     float64         float64   \n",
              "13              object   float64       float64     float64         float64   \n",
              "14              object   float64       float64     float64         float64   \n",
              "\n",
              "   total_rec_prncp total_rec_int total_rec_late_fee recoveries  \\\n",
              "0          float64       float64            float64    float64   \n",
              "1          float64       float64            float64    float64   \n",
              "2          float64       float64            float64    float64   \n",
              "3          float64       float64            float64    float64   \n",
              "4          float64       float64            float64    float64   \n",
              "5          float64       float64            float64    float64   \n",
              "6          float64       float64            float64    float64   \n",
              "7          float64       float64            float64    float64   \n",
              "8          float64       float64            float64    float64   \n",
              "9          float64       float64            float64    float64   \n",
              "10         float64       float64            float64    float64   \n",
              "11         float64       float64            float64    float64   \n",
              "12         float64       float64            float64    float64   \n",
              "13         float64       float64            float64    float64   \n",
              "14         float64       float64            float64    float64   \n",
              "\n",
              "   collection_recovery_fee last_pymnt_d last_pymnt_amnt last_credit_pull_d  \\\n",
              "0                  float64       object         float64             object   \n",
              "1                  float64       object         float64             object   \n",
              "2                  float64       object         float64             object   \n",
              "3                  float64       object         float64             object   \n",
              "4                  float64       object         float64             object   \n",
              "5                  float64       object         float64             object   \n",
              "6                  float64       object         float64             object   \n",
              "7                  float64       object         float64             object   \n",
              "8                  float64       object         float64             object   \n",
              "9                  float64       object         float64             object   \n",
              "10                 float64       object         float64             object   \n",
              "11                 float64       object         float64             object   \n",
              "12                 float64       object         float64             object   \n",
              "13                 float64       object         float64             object   \n",
              "14                 float64       object         float64             object   \n",
              "\n",
              "   collections_12_mths_ex_med policy_code application_type acc_now_delinq  \\\n",
              "0                     float64     float64           object        float64   \n",
              "1                     float64     float64           object        float64   \n",
              "2                     float64     float64           object        float64   \n",
              "3                     float64     float64           object        float64   \n",
              "4                     float64     float64           object        float64   \n",
              "5                     float64     float64           object        float64   \n",
              "6                     float64     float64           object        float64   \n",
              "7                     float64     float64           object        float64   \n",
              "8                     float64     float64           object        float64   \n",
              "9                     float64     float64           object        float64   \n",
              "10                    float64     float64           object        float64   \n",
              "11                    float64     float64           object        float64   \n",
              "12                    float64     float64           object        float64   \n",
              "13                    float64     float64           object        float64   \n",
              "14                    float64     float64           object        float64   \n",
              "\n",
              "   chargeoff_within_12_mths delinq_amnt pub_rec_bankruptcies tax_liens  \n",
              "0                   float64     float64              float64   float64  \n",
              "1                   float64     float64              float64   float64  \n",
              "2                   float64     float64              float64   float64  \n",
              "3                   float64     float64              float64   float64  \n",
              "4                   float64     float64              float64   float64  \n",
              "5                   float64     float64              float64   float64  \n",
              "6                   float64     float64              float64   float64  \n",
              "7                   float64     float64              float64   float64  \n",
              "8                   float64     float64              float64   float64  \n",
              "9                   float64     float64              float64   float64  \n",
              "10                  float64     float64              float64   float64  \n",
              "11                  float64     float64              float64   float64  \n",
              "12                  float64     float64              float64   float64  \n",
              "13                  float64     float64              float64   float64  \n",
              "14                  float64     float64              float64   float64  "
            ]
          },
          "execution_count": 12,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V1vJQkWTS-7m"
      },
      "source": [
        "## No More Inconsistencce Across Chunks After \"non-loan\" rows removal\n",
        "From the above, it can show that is no more inconsistence of column data types across all churnks after problematic rows are removed from concenred churnks\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m2E9ZkcMS-7n"
      },
      "source": [
        "## Any missing values ?\n",
        "\n",
        "The below code count the number of missing value for each columns for each chunk and then sum up missing value count for all chunks."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6juXRS8-S-7o",
        "outputId": "f1aba0af-a0e2-4eee-fbf3-fb13f73e468c"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000)\n",
        "\n",
        "\n",
        "loanChunkMissings=pd.DataFrame(columns=loan.columns)\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    \n",
        "    loanChunkMissing=[loanChunk[col].isnull().sum() for col in loanChunk.columns]\n",
        "    loanChunkMissings.loc[len(loanChunkMissings)]=loanChunkMissing\n",
        "\n",
        "loanChunkMissing=loanChunkMissings.sum(axis=0)\n",
        "\n",
        "loanChunkMissing"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "id                               0.0\n",
              "member_id                        0.0\n",
              "loan_amnt                        0.0\n",
              "funded_amnt                      0.0\n",
              "funded_amnt_inv                  0.0\n",
              "term                             0.0\n",
              "int_rate                         0.0\n",
              "installment                      0.0\n",
              "grade                            0.0\n",
              "sub_grade                        0.0\n",
              "emp_title                     2626.0\n",
              "emp_length                    1112.0\n",
              "home_ownership                   0.0\n",
              "annual_inc                       4.0\n",
              "verification_status              0.0\n",
              "issue_d                          0.0\n",
              "loan_status                      0.0\n",
              "pymnt_plan                       0.0\n",
              "purpose                          0.0\n",
              "title                           13.0\n",
              "zip_code                         0.0\n",
              "addr_state                       0.0\n",
              "dti                              0.0\n",
              "delinq_2yrs                     29.0\n",
              "earliest_cr_line                29.0\n",
              "inq_last_6mths                  29.0\n",
              "open_acc                        29.0\n",
              "pub_rec                         29.0\n",
              "revol_bal                        0.0\n",
              "revol_util                      90.0\n",
              "total_acc                       29.0\n",
              "initial_list_status              0.0\n",
              "out_prncp                        0.0\n",
              "out_prncp_inv                    0.0\n",
              "total_pymnt                      0.0\n",
              "total_pymnt_inv                  0.0\n",
              "total_rec_prncp                  0.0\n",
              "total_rec_int                    0.0\n",
              "total_rec_late_fee               0.0\n",
              "recoveries                       0.0\n",
              "collection_recovery_fee          0.0\n",
              "last_pymnt_d                    83.0\n",
              "last_pymnt_amnt                  0.0\n",
              "last_credit_pull_d               4.0\n",
              "collections_12_mths_ex_med     145.0\n",
              "policy_code                      0.0\n",
              "application_type                 0.0\n",
              "acc_now_delinq                  29.0\n",
              "chargeoff_within_12_mths       145.0\n",
              "delinq_amnt                     29.0\n",
              "pub_rec_bankruptcies          1365.0\n",
              "tax_liens                      105.0\n",
              "dtype: float64"
            ]
          },
          "execution_count": 13,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vnB3asdNS-7q"
      },
      "source": [
        "## Which float columns has no missing value and could be candidates for conversion to the integer type?\n",
        "\n",
        "Since we have checked data types for the same column in all chunk are the same afer \"non-loan\" rows are removed, the below code use the the column data type of last chunk from the previous step and missing value count to filter out float columns have no missing values. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3ett7DMzS-7r",
        "outputId": "df7fa53e-b285-42fd-c202-95da9cd954d5"
      },
      "source": [
        "colToInteger=loanChunk.columns[(loanChunk.dtypes=='float64') & (loanChunkMissing==0)]\n",
        "\n",
        "print(colToInteger)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Index(['member_id', 'loan_amnt', 'funded_amnt', 'funded_amnt_inv',\n",
            "       'installment', 'dti', 'revol_bal', 'out_prncp', 'out_prncp_inv',\n",
            "       'total_pymnt', 'total_pymnt_inv', 'total_rec_prncp', 'total_rec_int',\n",
            "       'total_rec_late_fee', 'recoveries', 'collection_recovery_fee',\n",
            "       'last_pymnt_amnt', 'policy_code'],\n",
            "      dtype='object')\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xae0U6UTS-7s"
      },
      "source": [
        "## Check candidate of Integer columns are only integer value\n",
        "\n",
        "Check across all chunk to see candidate integer columns only contain integer value across all chunk, if not remove the column from the candidate list.  The integer column list is saved for later use."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z2nRqqpvS-7u",
        "outputId": "630edbc4-cd3d-4201-9a04-7a8804270432"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000)\n",
        "\n",
        "colIsFloat=set()\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    \n",
        "    for col in loanChunk.columns:\n",
        "        if col in colToInteger:\n",
        "            allRowsInteger=all([x.is_integer() for x in loanChunk[col]])\n",
        "            if allRowsInteger==False:\n",
        "                colIsFloat.add(col)\n",
        "\n",
        "colToInteger=[x for x in colToInteger if x not in colIsFloat]\n",
        "colToInteger"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['member_id', 'loan_amnt', 'funded_amnt', 'revol_bal', 'policy_code']"
            ]
          },
          "execution_count": 15,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8jMRZmGqS-7w"
      },
      "source": [
        "## Which string columns contain values that are less than 50% unique?\n",
        "\n",
        "To answer the above we first create a list of dict to hold unique values of string (object) columns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ws47wVVbS-7x"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000)\n",
        "\n",
        "uniqueDicts=[{'col': col, 'non-numeric':False,'values':{}} for col in loan.columns]\n",
        "totalNumberOfRows=0\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    totalNumberOfRows+=loanChunk.shape[0]\n",
        "\n",
        "    for idx,col in enumerate(loanChunk):\n",
        "        if loanChunk[col].dtypes=='object':\n",
        "            uniqueDicts[idx]['non-numeric']=True\n",
        "            for cell in loanChunk[col]:\n",
        "                if cell in uniqueDicts[idx]['values']:\n",
        "                    uniqueDicts[idx]['values'][cell]+=1\n",
        "                else:\n",
        "                    uniqueDicts[idx]['values'][cell]=1\n",
        "               "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KxJ_0AGSS-7z"
      },
      "source": [
        "## But wait ... not all string columns are strings ...\n",
        "Explore the dict and found 7th and 30th column (int_rate, revol_util column) are holding percentage value, which should be of float type instead of string.  These column should be clean up first."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "b1IxML43S-7z",
        "outputId": "b4c946d1-4445-4aca-af33-fb572e7a5eb6"
      },
      "source": [
        "pp.pprint(uniqueDicts[6])\n",
        "pp.pprint(uniqueDicts[29])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'col': 'int_rate',\n",
            " 'non-numeric': True,\n",
            " 'values': {'  5.42%': 573,\n",
            "            '  5.79%': 410,\n",
            "            '  5.99%': 347,\n",
            "            '  6.00%': 19,\n",
            "            '  6.03%': 447,\n",
            "            '  6.17%': 252,\n",
            "            '  6.39%': 58,\n",
            "            '  6.54%': 307,\n",
            "            '  6.62%': 396,\n",
            "            '  6.76%': 168,\n",
            "            '  6.91%': 310,\n",
            "            '  6.92%': 204,\n",
            "            '  6.99%': 336,\n",
            "            '  7.05%': 23,\n",
            "            '  7.12%': 9,\n",
            "            '  7.14%': 342,\n",
            "            '  7.29%': 397,\n",
            "            '  7.37%': 32,\n",
            "            '  7.40%': 72,\n",
            "            '  7.42%': 7,\n",
            "            '  7.43%': 33,\n",
            "            '  7.49%': 656,\n",
            "            '  7.51%': 787,\n",
            "            '  7.66%': 292,\n",
            "            '  7.68%': 94,\n",
            "            '  7.74%': 154,\n",
            "            '  7.75%': 27,\n",
            "            '  7.88%': 742,\n",
            "            '  7.90%': 582,\n",
            "            '  7.91%': 14,\n",
            "            '  8.00%': 198,\n",
            "            '  8.07%': 26,\n",
            "            '  8.32%': 49,\n",
            "            '  8.38%': 26,\n",
            "            '  8.49%': 445,\n",
            "            '  8.59%': 187,\n",
            "            '  8.63%': 44,\n",
            "            '  8.70%': 17,\n",
            "            '  8.88%': 168,\n",
            "            '  8.90%': 402,\n",
            "            '  8.94%': 299,\n",
            "            '  9.01%': 17,\n",
            "            '  9.07%': 15,\n",
            "            '  9.20%': 15,\n",
            "            '  9.25%': 141,\n",
            "            '  9.32%': 187,\n",
            "            '  9.33%': 24,\n",
            "            '  9.38%': 23,\n",
            "            '  9.45%': 47,\n",
            "            '  9.51%': 10,\n",
            "            '  9.62%': 195,\n",
            "            '  9.63%': 384,\n",
            "            '  9.64%': 29,\n",
            "            '  9.67%': 7,\n",
            "            '  9.70%': 9,\n",
            "            '  9.76%': 77,\n",
            "            '  9.83%': 11,\n",
            "            '  9.88%': 186,\n",
            "            '  9.91%': 377,\n",
            "            '  9.96%': 30,\n",
            "            '  9.99%': 607,\n",
            "            ' 10.00%': 251,\n",
            "            ' 10.01%': 8,\n",
            "            ' 10.08%': 55,\n",
            "            ' 10.14%': 11,\n",
            "            ' 10.20%': 18,\n",
            "            ' 10.25%': 228,\n",
            "            ' 10.28%': 32,\n",
            "            ' 10.33%': 9,\n",
            "            ' 10.36%': 256,\n",
            "            ' 10.37%': 470,\n",
            "            ' 10.38%': 227,\n",
            "            ' 10.39%': 57,\n",
            "            ' 10.46%': 12,\n",
            "            ' 10.51%': 22,\n",
            "            ' 10.59%': 400,\n",
            "            ' 10.62%': 241,\n",
            "            ' 10.64%': 5,\n",
            "            ' 10.65%': 435,\n",
            "            ' 10.71%': 58,\n",
            "            ' 10.74%': 312,\n",
            "            ' 10.75%': 260,\n",
            "            ' 10.78%': 23,\n",
            "            ' 10.83%': 31,\n",
            "            ' 10.91%': 31,\n",
            "            ' 10.95%': 82,\n",
            "            ' 10.96%': 15,\n",
            "            ' 10.99%': 970,\n",
            "            ' 11.03%': 64,\n",
            "            ' 11.09%': 8,\n",
            "            ' 11.11%': 332,\n",
            "            ' 11.12%': 272,\n",
            "            ' 11.14%': 215,\n",
            "            ' 11.22%': 33,\n",
            "            ' 11.26%': 95,\n",
            "            ' 11.28%': 19,\n",
            "            ' 11.34%': 56,\n",
            "            ' 11.36%': 258,\n",
            "            ' 11.41%': 17,\n",
            "            ' 11.46%': 27,\n",
            "            ' 11.48%': 193,\n",
            "            ' 11.49%': 837,\n",
            "            ' 11.54%': 21,\n",
            "            ' 11.58%': 107,\n",
            "            ' 11.59%': 12,\n",
            "            ' 11.63%': 8,\n",
            "            ' 11.66%': 50,\n",
            "            ' 11.71%': 609,\n",
            "            ' 11.72%': 12,\n",
            "            ' 11.78%': 28,\n",
            "            ' 11.83%': 243,\n",
            "            ' 11.86%': 418,\n",
            "            ' 11.89%': 165,\n",
            "            ' 11.91%': 16,\n",
            "            ' 11.97%': 51,\n",
            "            ' 11.99%': 535,\n",
            "            ' 12.04%': 11,\n",
            "            ' 12.09%': 20,\n",
            "            ' 12.12%': 5,\n",
            "            ' 12.17%': 17,\n",
            "            ' 12.18%': 216,\n",
            "            ' 12.21%': 183,\n",
            "            ' 12.22%': 16,\n",
            "            ' 12.23%': 143,\n",
            "            ' 12.29%': 53,\n",
            "            ' 12.36%': 11,\n",
            "            ' 12.41%': 22,\n",
            "            ' 12.42%': 443,\n",
            "            ' 12.49%': 22,\n",
            "            ' 12.53%': 356,\n",
            "            ' 12.54%': 9,\n",
            "            ' 12.61%': 224,\n",
            "            ' 12.62%': 3,\n",
            "            ' 12.67%': 5,\n",
            "            ' 12.68%': 251,\n",
            "            ' 12.69%': 492,\n",
            "            ' 12.72%': 18,\n",
            "            ' 12.73%': 205,\n",
            "            ' 12.80%': 25,\n",
            "            ' 12.84%': 152,\n",
            "            ' 12.86%': 11,\n",
            "            ' 12.87%': 206,\n",
            "            ' 12.92%': 44,\n",
            "            ' 12.98%': 121,\n",
            "            ' 12.99%': 456,\n",
            "            ' 13.04%': 23,\n",
            "            ' 13.06%': 220,\n",
            "            ' 13.11%': 203,\n",
            "            ' 13.12%': 23,\n",
            "            ' 13.16%': 143,\n",
            "            ' 13.17%': 11,\n",
            "            ' 13.22%': 159,\n",
            "            ' 13.23%': 277,\n",
            "            ' 13.24%': 56,\n",
            "            ' 13.30%': 17,\n",
            "            ' 13.35%': 136,\n",
            "            ' 13.36%': 12,\n",
            "            ' 13.43%': 157,\n",
            "            ' 13.47%': 146,\n",
            "            ' 13.48%': 175,\n",
            "            ' 13.49%': 832,\n",
            "            ' 13.55%': 50,\n",
            "            ' 13.57%': 153,\n",
            "            ' 13.61%': 313,\n",
            "            ' 13.62%': 9,\n",
            "            ' 13.67%': 20,\n",
            "            ' 13.72%': 125,\n",
            "            ' 13.75%': 24,\n",
            "            ' 13.79%': 125,\n",
            "            ' 13.80%': 138,\n",
            "            ' 13.84%': 2,\n",
            "            ' 13.85%': 152,\n",
            "            ' 13.87%': 40,\n",
            "            ' 13.92%': 142,\n",
            "            ' 13.93%': 10,\n",
            "            ' 13.98%': 312,\n",
            "            ' 13.99%': 257,\n",
            "            ' 14.07%': 18,\n",
            "            ' 14.09%': 73,\n",
            "            ' 14.11%': 113,\n",
            "            ' 14.12%': 10,\n",
            "            ' 14.17%': 122,\n",
            "            ' 14.18%': 39,\n",
            "            ' 14.22%': 132,\n",
            "            ' 14.25%': 10,\n",
            "            ' 14.26%': 138,\n",
            "            ' 14.27%': 391,\n",
            "            ' 14.30%': 8,\n",
            "            ' 14.35%': 138,\n",
            "            ' 14.38%': 35,\n",
            "            ' 14.42%': 106,\n",
            "            ' 14.43%': 4,\n",
            "            ' 14.46%': 118,\n",
            "            ' 14.50%': 41,\n",
            "            ' 14.54%': 75,\n",
            "            ' 14.57%': 3,\n",
            "            ' 14.59%': 132,\n",
            "            ' 14.61%': 114,\n",
            "            ' 14.62%': 4,\n",
            "            ' 14.65%': 222,\n",
            "            ' 14.67%': 3,\n",
            "            ' 14.70%': 13,\n",
            "            ' 14.72%': 125,\n",
            "            ' 14.74%': 84,\n",
            "            ' 14.75%': 9,\n",
            "            ' 14.79%': 194,\n",
            "            ' 14.82%': 43,\n",
            "            ' 14.83%': 105,\n",
            "            ' 14.84%': 131,\n",
            "            ' 14.88%': 7,\n",
            "            ' 14.91%': 172,\n",
            "            ' 14.93%': 8,\n",
            "            ' 14.96%': 249,\n",
            "            ' 15.01%': 17,\n",
            "            ' 15.05%': 76,\n",
            "            ' 15.07%': 9,\n",
            "            ' 15.13%': 22,\n",
            "            ' 15.20%': 86,\n",
            "            ' 15.21%': 220,\n",
            "            ' 15.23%': 216,\n",
            "            ' 15.25%': 7,\n",
            "            ' 15.27%': 200,\n",
            "            ' 15.28%': 147,\n",
            "            ' 15.31%': 113,\n",
            "            ' 15.33%': 135,\n",
            "            ' 15.37%': 52,\n",
            "            ' 15.38%': 6,\n",
            "            ' 15.45%': 24,\n",
            "            ' 15.51%': 5,\n",
            "            ' 15.57%': 92,\n",
            "            ' 15.58%': 201,\n",
            "            ' 15.62%': 171,\n",
            "            ' 15.65%': 232,\n",
            "            ' 15.68%': 50,\n",
            "            ' 15.70%': 99,\n",
            "            ' 15.76%': 16,\n",
            "            ' 15.83%': 7,\n",
            "            ' 15.88%': 6,\n",
            "            ' 15.95%': 240,\n",
            "            ' 15.96%': 193,\n",
            "            ' 15.99%': 287,\n",
            "            ' 16.00%': 127,\n",
            "            ' 16.01%': 6,\n",
            "            ' 16.02%': 150,\n",
            "            ' 16.07%': 76,\n",
            "            ' 16.08%': 15,\n",
            "            ' 16.11%': 7,\n",
            "            ' 16.15%': 5,\n",
            "            ' 16.20%': 1,\n",
            "            ' 16.28%': 10,\n",
            "            ' 16.29%': 142,\n",
            "            ' 16.32%': 233,\n",
            "            ' 16.33%': 2,\n",
            "            ' 16.35%': 49,\n",
            "            ' 16.40%': 154,\n",
            "            ' 16.45%': 193,\n",
            "            ' 16.46%': 3,\n",
            "            ' 16.49%': 245,\n",
            "            ' 16.59%': 9,\n",
            "            ' 16.63%': 23,\n",
            "            ' 16.65%': 4,\n",
            "            ' 16.69%': 65,\n",
            "            ' 16.70%': 43,\n",
            "            ' 16.71%': 7,\n",
            "            ' 16.77%': 329,\n",
            "            ' 16.82%': 162,\n",
            "            ' 16.83%': 1,\n",
            "            ' 16.89%': 213,\n",
            "            ' 16.91%': 5,\n",
            "            ' 16.95%': 24,\n",
            "            ' 16.96%': 4,\n",
            "            ' 17.03%': 8,\n",
            "            ' 17.04%': 32,\n",
            "            ' 17.06%': 36,\n",
            "            ' 17.09%': 2,\n",
            "            ' 17.14%': 108,\n",
            "            ' 17.15%': 4,\n",
            "            ' 17.19%': 131,\n",
            "            ' 17.22%': 8,\n",
            "            ' 17.26%': 17,\n",
            "            ' 17.27%': 204,\n",
            "            ' 17.28%': 2,\n",
            "            ' 17.34%': 13,\n",
            "            ' 17.39%': 22,\n",
            "            ' 17.41%': 1,\n",
            "            ' 17.43%': 32,\n",
            "            ' 17.44%': 1,\n",
            "            ' 17.46%': 1,\n",
            "            ' 17.49%': 190,\n",
            "            ' 17.51%': 65,\n",
            "            ' 17.54%': 11,\n",
            "            ' 17.56%': 125,\n",
            "            ' 17.58%': 191,\n",
            "            ' 17.59%': 2,\n",
            "            ' 17.66%': 8,\n",
            "            ' 17.72%': 1,\n",
            "            ' 17.74%': 25,\n",
            "            ' 17.78%': 1,\n",
            "            ' 17.80%': 26,\n",
            "            ' 17.86%': 9,\n",
            "            ' 17.88%': 66,\n",
            "            ' 17.90%': 6,\n",
            "            ' 17.91%': 2,\n",
            "            ' 17.93%': 96,\n",
            "            ' 17.97%': 6,\n",
            "            ' 17.99%': 150,\n",
            "            ' 18.04%': 6,\n",
            "            ' 18.07%': 9,\n",
            "            ' 18.09%': 11,\n",
            "            ' 18.17%': 30,\n",
            "            ' 18.21%': 9,\n",
            "            ' 18.25%': 233,\n",
            "            ' 18.29%': 13,\n",
            "            ' 18.30%': 68,\n",
            "            ' 18.36%': 5,\n",
            "            ' 18.39%': 155,\n",
            "            ' 18.43%': 21,\n",
            "            ' 18.49%': 3,\n",
            "            ' 18.53%': 7,\n",
            "            ' 18.54%': 24,\n",
            "            ' 18.61%': 16,\n",
            "            ' 18.62%': 49,\n",
            "            ' 18.64%': 143,\n",
            "            ' 18.67%': 51,\n",
            "            ' 18.72%': 1,\n",
            "            ' 18.78%': 7,\n",
            "            ' 18.79%': 106,\n",
            "            ' 18.84%': 6,\n",
            "            ' 18.86%': 3,\n",
            "            ' 18.91%': 12,\n",
            "            ' 18.99%': 30,\n",
            "            ' 19.03%': 126,\n",
            "            ' 19.04%': 43,\n",
            "            ' 19.13%': 17,\n",
            "            ' 19.16%': 5,\n",
            "            ' 19.29%': 120,\n",
            "            ' 19.36%': 34,\n",
            "            ' 19.39%': 4,\n",
            "            ' 19.41%': 32,\n",
            "            ' 19.42%': 118,\n",
            "            ' 19.47%': 15,\n",
            "            ' 19.66%': 11,\n",
            "            ' 19.69%': 108,\n",
            "            ' 19.74%': 30,\n",
            "            ' 19.79%': 25,\n",
            "            ' 19.82%': 8,\n",
            "            ' 19.91%': 91,\n",
            "            ' 20.03%': 13,\n",
            "            ' 20.11%': 33,\n",
            "            ' 20.16%': 20,\n",
            "            ' 20.17%': 6,\n",
            "            ' 20.20%': 2,\n",
            "            ' 20.25%': 77,\n",
            "            ' 20.30%': 89,\n",
            "            ' 20.40%': 4,\n",
            "            ' 20.48%': 22,\n",
            "            ' 20.52%': 4,\n",
            "            ' 20.53%': 20,\n",
            "            ' 20.62%': 59,\n",
            "            ' 20.69%': 3,\n",
            "            ' 20.77%': 12,\n",
            "            ' 20.85%': 16,\n",
            "            ' 20.86%': 6,\n",
            "            ' 20.89%': 72,\n",
            "            ' 20.90%': 18,\n",
            "            ' 20.99%': 55,\n",
            "            ' 21.14%': 7,\n",
            "            ' 21.21%': 7,\n",
            "            ' 21.22%': 7,\n",
            "            ' 21.27%': 16,\n",
            "            ' 21.28%': 43,\n",
            "            ' 21.36%': 45,\n",
            "            ' 21.48%': 1,\n",
            "            ' 21.59%': 8,\n",
            "            ' 21.64%': 12,\n",
            "            ' 21.67%': 42,\n",
            "            ' 21.74%': 31,\n",
            "            ' 21.82%': 2,\n",
            "            ' 22.06%': 37,\n",
            "            ' 22.11%': 31,\n",
            "            ' 22.35%': 25,\n",
            "            ' 22.48%': 16,\n",
            "            ' 22.64%': 1,\n",
            "            ' 22.74%': 20,\n",
            "            ' 22.85%': 8,\n",
            "            ' 22.94%': 2,\n",
            "            ' 23.13%': 9,\n",
            "            ' 23.22%': 9,\n",
            "            ' 23.52%': 9,\n",
            "            ' 23.59%': 4,\n",
            "            ' 23.91%': 11,\n",
            "            ' 24.11%': 3,\n",
            "            ' 24.40%': 1,\n",
            "            ' 24.59%': 1}}\n",
            "{'col': 'revol_util',\n",
            " 'non-numeric': True,\n",
            " 'values': {nan: 90,\n",
            "            '0%': 1070,\n",
            "            '0.01%': 1,\n",
            "            '0.03%': 1,\n",
            "            '0.04%': 1,\n",
            "            '0.05%': 1,\n",
            "            '0.1%': 61,\n",
            "            '0.12%': 1,\n",
            "            '0.16%': 1,\n",
            "            '0.2%': 64,\n",
            "            '0.3%': 43,\n",
            "            '0.4%': 45,\n",
            "            '0.46%': 1,\n",
            "            '0.49%': 1,\n",
            "            '0.5%': 49,\n",
            "            '0.54%': 1,\n",
            "            '0.6%': 39,\n",
            "            '0.7%': 47,\n",
            "            '0.75%': 1,\n",
            "            '0.8%': 43,\n",
            "            '0.83%': 1,\n",
            "            '0.86%': 1,\n",
            "            '0.9%': 41,\n",
            "            '1%': 51,\n",
            "            '1.1%': 39,\n",
            "            '1.2%': 37,\n",
            "            '1.3%': 36,\n",
            "            '1.4%': 32,\n",
            "            '1.5%': 33,\n",
            "            '1.6%': 24,\n",
            "            '1.7%': 30,\n",
            "            '1.8%': 40,\n",
            "            '1.88%': 1,\n",
            "            '1.9%': 31,\n",
            "            '10%': 38,\n",
            "            '10.08%': 1,\n",
            "            '10.1%': 36,\n",
            "            '10.17%': 1,\n",
            "            '10.2%': 30,\n",
            "            '10.3%': 33,\n",
            "            '10.4%': 33,\n",
            "            '10.5%': 34,\n",
            "            '10.6%': 20,\n",
            "            '10.61%': 1,\n",
            "            '10.7%': 27,\n",
            "            '10.8%': 31,\n",
            "            '10.9%': 36,\n",
            "            '100%': 2,\n",
            "            '100.5%': 1,\n",
            "            '100.6%': 1,\n",
            "            '100.7%': 2,\n",
            "            '101.4%': 3,\n",
            "            '101.5%': 1,\n",
            "            '101.8%': 2,\n",
            "            '102.7%': 1,\n",
            "            '102.8%': 1,\n",
            "            '103.1%': 1,\n",
            "            '103.2%': 1,\n",
            "            '103.5%': 1,\n",
            "            '103.9%': 1,\n",
            "            '104.2%': 1,\n",
            "            '104.3%': 1,\n",
            "            '105.2%': 1,\n",
            "            '105.3%': 1,\n",
            "            '105.7%': 1,\n",
            "            '106.1%': 1,\n",
            "            '106.2%': 1,\n",
            "            '106.4%': 1,\n",
            "            '106.5%': 1,\n",
            "            '108.8%': 1,\n",
            "            '11%': 38,\n",
            "            '11.1%': 29,\n",
            "            '11.2%': 31,\n",
            "            '11.3%': 38,\n",
            "            '11.4%': 35,\n",
            "            '11.5%': 33,\n",
            "            '11.6%': 38,\n",
            "            '11.62%': 1,\n",
            "            '11.63%': 1,\n",
            "            '11.7%': 31,\n",
            "            '11.8%': 44,\n",
            "            '11.9%': 29,\n",
            "            '119%': 1,\n",
            "            '12%': 38,\n",
            "            '12.1%': 44,\n",
            "            '12.2%': 32,\n",
            "            '12.3%': 28,\n",
            "            '12.4%': 42,\n",
            "            '12.42%': 1,\n",
            "            '12.5%': 31,\n",
            "            '12.6%': 36,\n",
            "            '12.7%': 32,\n",
            "            '12.8%': 46,\n",
            "            '12.9%': 36,\n",
            "            '13%': 32,\n",
            "            '13.1%': 36,\n",
            "            '13.2%': 30,\n",
            "            '13.3%': 29,\n",
            "            '13.4%': 35,\n",
            "            '13.5%': 26,\n",
            "            '13.56%': 1,\n",
            "            '13.6%': 24,\n",
            "            '13.7%': 36,\n",
            "            '13.8%': 34,\n",
            "            '13.9%': 35,\n",
            "            '14%': 33,\n",
            "            '14.1%': 39,\n",
            "            '14.2%': 45,\n",
            "            '14.3%': 42,\n",
            "            '14.4%': 40,\n",
            "            '14.5%': 27,\n",
            "            '14.6%': 35,\n",
            "            '14.7%': 37,\n",
            "            '14.8%': 43,\n",
            "            '14.9%': 45,\n",
            "            '15%': 32,\n",
            "            '15.1%': 42,\n",
            "            '15.2%': 32,\n",
            "            '15.3%': 37,\n",
            "            '15.4%': 30,\n",
            "            '15.5%': 41,\n",
            "            '15.6%': 46,\n",
            "            '15.7%': 31,\n",
            "            '15.8%': 40,\n",
            "            '15.9%': 35,\n",
            "            '16%': 35,\n",
            "            '16.02%': 1,\n",
            "            '16.1%': 29,\n",
            "            '16.2%': 32,\n",
            "            '16.3%': 32,\n",
            "            '16.4%': 31,\n",
            "            '16.5%': 47,\n",
            "            '16.6%': 32,\n",
            "            '16.7%': 42,\n",
            "            '16.8%': 33,\n",
            "            '16.9%': 26,\n",
            "            '17%': 31,\n",
            "            '17.1%': 44,\n",
            "            '17.2%': 41,\n",
            "            '17.3%': 36,\n",
            "            '17.4%': 34,\n",
            "            '17.5%': 42,\n",
            "            '17.6%': 38,\n",
            "            '17.67%': 1,\n",
            "            '17.7%': 37,\n",
            "            '17.71%': 1,\n",
            "            '17.78%': 1,\n",
            "            '17.8%': 34,\n",
            "            '17.9%': 43,\n",
            "            '18%': 30,\n",
            "            '18.1%': 34,\n",
            "            '18.2%': 47,\n",
            "            '18.3%': 34,\n",
            "            '18.4%': 43,\n",
            "            '18.5%': 33,\n",
            "            '18.6%': 38,\n",
            "            '18.7%': 39,\n",
            "            '18.8%': 37,\n",
            "            '18.82%': 1,\n",
            "            '18.9%': 31,\n",
            "            '19%': 44,\n",
            "            '19.1%': 27,\n",
            "            '19.2%': 36,\n",
            "            '19.3%': 38,\n",
            "            '19.4%': 44,\n",
            "            '19.5%': 35,\n",
            "            '19.6%': 32,\n",
            "            '19.7%': 30,\n",
            "            '19.8%': 51,\n",
            "            '19.9%': 28,\n",
            "            '2%': 35,\n",
            "            '2.1%': 32,\n",
            "            '2.2%': 35,\n",
            "            '2.3%': 40,\n",
            "            '2.4%': 40,\n",
            "            '2.5%': 30,\n",
            "            '2.6%': 31,\n",
            "            '2.64%': 1,\n",
            "            '2.7%': 32,\n",
            "            '2.8%': 33,\n",
            "            '2.9%': 32,\n",
            "            '20%': 45,\n",
            "            '20.1%': 25,\n",
            "            '20.2%': 36,\n",
            "            '20.3%': 41,\n",
            "            '20.4%': 31,\n",
            "            '20.5%': 35,\n",
            "            '20.6%': 37,\n",
            "            '20.7%': 27,\n",
            "            '20.8%': 36,\n",
            "            '20.9%': 29,\n",
            "            '21%': 46,\n",
            "            '21.1%': 43,\n",
            "            '21.2%': 47,\n",
            "            '21.3%': 38,\n",
            "            '21.4%': 40,\n",
            "            '21.5%': 52,\n",
            "            '21.59%': 1,\n",
            "            '21.6%': 38,\n",
            "            '21.7%': 31,\n",
            "            '21.72%': 1,\n",
            "            '21.8%': 35,\n",
            "            '21.9%': 34,\n",
            "            '21.92%': 1,\n",
            "            '22%': 45,\n",
            "            '22.1%': 48,\n",
            "            '22.2%': 39,\n",
            "            '22.3%': 44,\n",
            "            '22.4%': 32,\n",
            "            '22.5%': 36,\n",
            "            '22.6%': 39,\n",
            "            '22.7%': 35,\n",
            "            '22.8%': 36,\n",
            "            '22.9%': 40,\n",
            "            '23%': 54,\n",
            "            '23.1%': 43,\n",
            "            '23.11%': 1,\n",
            "            '23.2%': 43,\n",
            "            '23.3%': 31,\n",
            "            '23.4%': 34,\n",
            "            '23.5%': 46,\n",
            "            '23.6%': 28,\n",
            "            '23.7%': 52,\n",
            "            '23.8%': 51,\n",
            "            '23.9%': 40,\n",
            "            '24%': 44,\n",
            "            '24.1%': 41,\n",
            "            '24.2%': 44,\n",
            "            '24.3%': 48,\n",
            "            '24.4%': 42,\n",
            "            '24.5%': 35,\n",
            "            '24.6%': 41,\n",
            "            '24.63%': 1,\n",
            "            '24.65%': 1,\n",
            "            '24.66%': 1,\n",
            "            '24.7%': 48,\n",
            "            '24.8%': 34,\n",
            "            '24.89%': 1,\n",
            "            '24.9%': 43,\n",
            "            '25%': 46,\n",
            "            '25.1%': 28,\n",
            "            '25.2%': 39,\n",
            "            '25.3%': 47,\n",
            "            '25.33%': 1,\n",
            "            '25.4%': 31,\n",
            "            '25.5%': 58,\n",
            "            '25.6%': 37,\n",
            "            '25.7%': 44,\n",
            "            '25.74%': 1,\n",
            "            '25.8%': 40,\n",
            "            '25.9%': 31,\n",
            "            '26%': 49,\n",
            "            '26.1%': 34,\n",
            "            '26.2%': 37,\n",
            "            '26.3%': 32,\n",
            "            '26.32%': 1,\n",
            "            '26.33%': 1,\n",
            "            '26.4%': 42,\n",
            "            '26.5%': 41,\n",
            "            '26.6%': 47,\n",
            "            '26.7%': 51,\n",
            "            '26.8%': 41,\n",
            "            '26.9%': 42,\n",
            "            '27%': 40,\n",
            "            '27.1%': 42,\n",
            "            '27.2%': 54,\n",
            "            '27.3%': 46,\n",
            "            '27.4%': 47,\n",
            "            '27.5%': 37,\n",
            "            '27.6%': 44,\n",
            "            '27.7%': 54,\n",
            "            '27.8%': 40,\n",
            "            '27.81%': 1,\n",
            "            '27.9%': 38,\n",
            "            '28%': 37,\n",
            "            '28.1%': 35,\n",
            "            '28.2%': 40,\n",
            "            '28.3%': 38,\n",
            "            '28.4%': 47,\n",
            "            '28.41%': 1,\n",
            "            '28.5%': 32,\n",
            "            '28.6%': 32,\n",
            "            '28.7%': 30,\n",
            "            '28.8%': 47,\n",
            "            '28.9%': 46,\n",
            "            '29%': 35,\n",
            "            '29.1%': 54,\n",
            "            '29.2%': 45,\n",
            "            '29.3%': 46,\n",
            "            '29.4%': 41,\n",
            "            '29.5%': 26,\n",
            "            '29.53%': 1,\n",
            "            '29.6%': 42,\n",
            "            '29.7%': 35,\n",
            "            '29.77%': 1,\n",
            "            '29.8%': 45,\n",
            "            '29.9%': 43,\n",
            "            '3%': 27,\n",
            "            '3.1%': 37,\n",
            "            '3.18%': 1,\n",
            "            '3.2%': 39,\n",
            "            '3.3%': 44,\n",
            "            '3.4%': 27,\n",
            "            '3.5%': 33,\n",
            "            '3.6%': 35,\n",
            "            '3.7%': 35,\n",
            "            '3.8%': 31,\n",
            "            '3.9%': 24,\n",
            "            '30%': 35,\n",
            "            '30.1%': 50,\n",
            "            '30.19%': 1,\n",
            "            '30.2%': 37,\n",
            "            '30.3%': 40,\n",
            "            '30.4%': 33,\n",
            "            '30.5%': 49,\n",
            "            '30.6%': 43,\n",
            "            '30.7%': 51,\n",
            "            '30.8%': 46,\n",
            "            '30.9%': 56,\n",
            "            '31%': 37,\n",
            "            '31.1%': 49,\n",
            "            '31.2%': 59,\n",
            "            '31.3%': 39,\n",
            "            '31.4%': 40,\n",
            "            '31.5%': 28,\n",
            "            '31.6%': 42,\n",
            "            '31.7%': 39,\n",
            "            '31.8%': 31,\n",
            "            '31.9%': 43,\n",
            "            '32%': 55,\n",
            "            '32.04%': 1,\n",
            "            '32.1%': 38,\n",
            "            '32.2%': 43,\n",
            "            '32.3%': 42,\n",
            "            '32.4%': 50,\n",
            "            '32.5%': 35,\n",
            "            '32.6%': 45,\n",
            "            '32.7%': 39,\n",
            "            '32.71%': 1,\n",
            "            '32.8%': 39,\n",
            "            '32.9%': 41,\n",
            "            '33%': 36,\n",
            "            '33.1%': 44,\n",
            "            '33.14%': 1,\n",
            "            '33.2%': 50,\n",
            "            '33.26%': 1,\n",
            "            '33.29%': 1,\n",
            "            '33.3%': 54,\n",
            "            '33.39%': 1,\n",
            "            '33.4%': 43,\n",
            "            '33.5%': 45,\n",
            "            '33.6%': 35,\n",
            "            '33.7%': 54,\n",
            "            '33.8%': 51,\n",
            "            '33.9%': 48,\n",
            "            '34%': 43,\n",
            "            '34.1%': 42,\n",
            "            '34.2%': 47,\n",
            "            '34.3%': 41,\n",
            "            '34.4%': 57,\n",
            "            '34.5%': 48,\n",
            "            '34.6%': 32,\n",
            "            '34.7%': 42,\n",
            "            '34.8%': 44,\n",
            "            '34.89%': 1,\n",
            "            '34.9%': 41,\n",
            "            '35%': 49,\n",
            "            '35.1%': 41,\n",
            "            '35.2%': 47,\n",
            "            '35.3%': 59,\n",
            "            '35.4%': 40,\n",
            "            '35.5%': 47,\n",
            "            '35.6%': 32,\n",
            "            '35.7%': 53,\n",
            "            '35.8%': 38,\n",
            "            '35.9%': 41,\n",
            "            '36%': 41,\n",
            "            '36.1%': 49,\n",
            "            '36.2%': 34,\n",
            "            '36.3%': 28,\n",
            "            '36.4%': 42,\n",
            "            '36.5%': 54,\n",
            "            '36.6%': 39,\n",
            "            '36.7%': 47,\n",
            "            '36.78%': 1,\n",
            "            '36.8%': 51,\n",
            "            '36.88%': 1,\n",
            "            '36.9%': 48,\n",
            "            '36.94%': 1,\n",
            "            '37%': 36,\n",
            "            '37.1%': 42,\n",
            "            '37.2%': 38,\n",
            "            '37.3%': 36,\n",
            "            '37.4%': 48,\n",
            "            '37.5%': 44,\n",
            "            '37.6%': 60,\n",
            "            '37.63%': 1,\n",
            "            '37.7%': 42,\n",
            "            '37.73%': 1,\n",
            "            '37.8%': 47,\n",
            "            '37.9%': 50,\n",
            "            '38%': 45,\n",
            "            '38.1%': 48,\n",
            "            '38.2%': 40,\n",
            "            '38.3%': 45,\n",
            "            '38.4%': 51,\n",
            "            '38.5%': 46,\n",
            "            '38.6%': 48,\n",
            "            '38.7%': 39,\n",
            "            '38.77%': 1,\n",
            "            '38.8%': 45,\n",
            "            '38.9%': 46,\n",
            "            '39%': 45,\n",
            "            '39.1%': 47,\n",
            "            '39.2%': 45,\n",
            "            '39.3%': 39,\n",
            "            '39.4%': 40,\n",
            "            '39.5%': 48,\n",
            "            '39.6%': 33,\n",
            "            '39.7%': 49,\n",
            "            '39.8%': 49,\n",
            "            '39.9%': 41,\n",
            "            '39.95%': 1,\n",
            "            '4%': 37,\n",
            "            '4.1%': 30,\n",
            "            '4.2%': 32,\n",
            "            '4.3%': 37,\n",
            "            '4.4%': 31,\n",
            "            '4.5%': 37,\n",
            "            '4.6%': 39,\n",
            "            '4.7%': 29,\n",
            "            '4.8%': 36,\n",
            "            '4.85%': 1,\n",
            "            '4.9%': 17,\n",
            "            '40%': 39,\n",
            "            '40.1%': 40,\n",
            "            '40.2%': 52,\n",
            "            '40.3%': 52,\n",
            "            '40.4%': 41,\n",
            "            '40.5%': 45,\n",
            "            '40.6%': 45,\n",
            "            '40.7%': 65,\n",
            "            '40.8%': 38,\n",
            "            '40.9%': 38,\n",
            "            '41%': 56,\n",
            "            '41.1%': 41,\n",
            "            '41.2%': 47,\n",
            "            '41.3%': 34,\n",
            "            '41.4%': 53,\n",
            "            '41.5%': 37,\n",
            "            '41.6%': 54,\n",
            "            '41.7%': 48,\n",
            "            '41.8%': 44,\n",
            "            '41.85%': 1,\n",
            "            '41.9%': 47,\n",
            "            '42%': 42,\n",
            "            '42.1%': 44,\n",
            "            '42.2%': 47,\n",
            "            '42.3%': 45,\n",
            "            '42.4%': 41,\n",
            "            '42.5%': 48,\n",
            "            '42.6%': 39,\n",
            "            '42.7%': 44,\n",
            "            '42.8%': 43,\n",
            "            '42.9%': 50,\n",
            "            '43%': 30,\n",
            "            '43.1%': 40,\n",
            "            '43.2%': 39,\n",
            "            '43.3%': 44,\n",
            "            '43.38%': 1,\n",
            "            '43.4%': 55,\n",
            "            '43.5%': 44,\n",
            "            '43.6%': 54,\n",
            "            '43.61%': 1,\n",
            "            '43.7%': 45,\n",
            "            '43.8%': 49,\n",
            "            '43.9%': 42,\n",
            "            '44%': 35,\n",
            "            '44.1%': 51,\n",
            "            '44.2%': 44,\n",
            "            '44.3%': 42,\n",
            "            '44.4%': 44,\n",
            "            '44.5%': 46,\n",
            "            '44.6%': 51,\n",
            "            '44.7%': 50,\n",
            "            '44.8%': 45,\n",
            "            '44.9%': 47,\n",
            "            '45%': 44,\n",
            "            '45.1%': 44,\n",
            "            '45.2%': 44,\n",
            "            '45.3%': 44,\n",
            "            '45.4%': 50,\n",
            "            '45.5%': 42,\n",
            "            '45.6%': 56,\n",
            "            '45.7%': 29,\n",
            "            '45.8%': 42,\n",
            "            '45.9%': 53,\n",
            "            '46%': 53,\n",
            "            '46.1%': 39,\n",
            "            '46.2%': 49,\n",
            "            '46.3%': 44,\n",
            "            '46.4%': 59,\n",
            "            '46.5%': 44,\n",
            "            '46.6%': 58,\n",
            "            '46.7%': 49,\n",
            "            '46.74%': 1,\n",
            "            '46.8%': 42,\n",
            "            '46.9%': 43,\n",
            "            '47%': 40,\n",
            "            '47.1%': 50,\n",
            "            '47.2%': 41,\n",
            "            '47.3%': 40,\n",
            "            '47.36%': 1,\n",
            "            '47.4%': 52,\n",
            "            '47.5%': 40,\n",
            "            '47.6%': 51,\n",
            "            '47.7%': 48,\n",
            "            '47.8%': 56,\n",
            "            '47.9%': 39,\n",
            "            '48%': 56,\n",
            "            '48.1%': 44,\n",
            "            '48.2%': 55,\n",
            "            '48.3%': 41,\n",
            "            '48.4%': 32,\n",
            "            '48.5%': 52,\n",
            "            '48.6%': 47,\n",
            "            '48.7%': 53,\n",
            "            '48.8%': 49,\n",
            "            '48.9%': 57,\n",
            "            '49%': 40,\n",
            "            '49.1%': 53,\n",
            "            '49.2%': 46,\n",
            "            '49.3%': 48,\n",
            "            '49.4%': 40,\n",
            "            '49.5%': 48,\n",
            "            '49.6%': 53,\n",
            "            '49.63%': 1,\n",
            "            '49.69%': 1,\n",
            "            '49.7%': 47,\n",
            "            '49.8%': 57,\n",
            "            '49.9%': 44,\n",
            "            '5%': 41,\n",
            "            '5.1%': 32,\n",
            "            '5.2%': 34,\n",
            "            '5.3%': 37,\n",
            "            '5.33%': 1,\n",
            "            '5.34%': 1,\n",
            "            '5.4%': 35,\n",
            "            '5.5%': 32,\n",
            "            '5.6%': 45,\n",
            "            '5.7%': 36,\n",
            "            '5.79%': 1,\n",
            "            '5.8%': 36,\n",
            "            '5.9%': 36,\n",
            "            '50%': 51,\n",
            "            '50.1%': 43,\n",
            "            '50.2%': 43,\n",
            "            '50.3%': 42,\n",
            "            '50.4%': 48,\n",
            "            '50.5%': 40,\n",
            "            '50.6%': 47,\n",
            "            '50.7%': 41,\n",
            "            '50.8%': 36,\n",
            "            '50.9%': 43,\n",
            "            '51%': 40,\n",
            "            '51.1%': 55,\n",
            "            '51.2%': 48,\n",
            "            '51.3%': 38,\n",
            "            '51.4%': 48,\n",
            "            '51.5%': 53,\n",
            "            '51.6%': 56,\n",
            "            '51.7%': 41,\n",
            "            '51.8%': 48,\n",
            "            '51.9%': 48,\n",
            "            '52%': 53,\n",
            "            '52.1%': 37,\n",
            "            '52.2%': 43,\n",
            "            '52.3%': 52,\n",
            "            '52.4%': 52,\n",
            "            '52.5%': 35,\n",
            "            '52.58%': 1,\n",
            "            '52.6%': 43,\n",
            "            '52.7%': 49,\n",
            "            '52.8%': 39,\n",
            "            '52.9%': 38,\n",
            "            '53%': 47,\n",
            "            '53.1%': 40,\n",
            "            '53.2%': 31,\n",
            "            '53.3%': 44,\n",
            "            '53.4%': 40,\n",
            "            '53.5%': 49,\n",
            "            '53.6%': 45,\n",
            "            '53.7%': 50,\n",
            "            '53.8%': 37,\n",
            "            '53.9%': 54,\n",
            "            '54%': 50,\n",
            "            '54.1%': 56,\n",
            "            '54.2%': 52,\n",
            "            '54.22%': 1,\n",
            "            '54.3%': 56,\n",
            "            '54.4%': 41,\n",
            "            '54.5%': 47,\n",
            "            '54.6%': 56,\n",
            "            '54.7%': 47,\n",
            "            '54.8%': 51,\n",
            "            '54.9%': 48,\n",
            "            '55%': 47,\n",
            "            '55.1%': 52,\n",
            "            '55.2%': 56,\n",
            "            '55.3%': 44,\n",
            "            '55.4%': 49,\n",
            "            '55.5%': 56,\n",
            "            '55.6%': 45,\n",
            "            '55.7%': 54,\n",
            "            '55.8%': 58,\n",
            "            '55.9%': 38,\n",
            "            '56%': 43,\n",
            "            '56.1%': 49,\n",
            "            '56.2%': 44,\n",
            "            '56.26%': 1,\n",
            "            '56.3%': 43,\n",
            "            '56.4%': 48,\n",
            "            '56.5%': 46,\n",
            "            '56.6%': 45,\n",
            "            '56.7%': 52,\n",
            "            '56.8%': 48,\n",
            "            '56.9%': 41,\n",
            "            '57%': 40,\n",
            "            '57.1%': 39,\n",
            "            '57.2%': 54,\n",
            "            '57.3%': 45,\n",
            "            '57.4%': 58,\n",
            "            '57.5%': 55,\n",
            "            '57.56%': 1,\n",
            "            '57.6%': 43,\n",
            "            '57.7%': 56,\n",
            "            '57.8%': 52,\n",
            "            '57.9%': 53,\n",
            "            '58%': 46,\n",
            "            '58.1%': 52,\n",
            "            '58.19%': 1,\n",
            "            '58.2%': 49,\n",
            "            '58.3%': 41,\n",
            "            '58.4%': 43,\n",
            "            '58.5%': 43,\n",
            "            '58.6%': 34,\n",
            "            '58.7%': 52,\n",
            "            '58.77%': 1,\n",
            "            '58.8%': 39,\n",
            "            '58.9%': 41,\n",
            "            '59%': 45,\n",
            "            '59.1%': 44,\n",
            "            '59.2%': 49,\n",
            "            '59.3%': 52,\n",
            "            '59.4%': 41,\n",
            "            '59.5%': 37,\n",
            "            '59.6%': 53,\n",
            "            '59.7%': 43,\n",
            "            '59.8%': 47,\n",
            "            '59.9%': 48,\n",
            "            '6%': 37,\n",
            "            '6.1%': 36,\n",
            "            '6.2%': 36,\n",
            "            '6.3%': 26,\n",
            "            '6.4%': 39,\n",
            "            '6.5%': 37,\n",
            "            '6.6%': 35,\n",
            "            '6.7%': 27,\n",
            "            '6.75%': 1,\n",
            "            '6.8%': 38,\n",
            "            '6.9%': 33,\n",
            "            '60%': 46,\n",
            "            '60.1%': 49,\n",
            "            '60.2%': 54,\n",
            "            '60.3%': 51,\n",
            "            '60.4%': 43,\n",
            "            '60.5%': 39,\n",
            "            '60.6%': 38,\n",
            "            '60.69%': 1,\n",
            "            '60.7%': 46,\n",
            "            '60.8%': 47,\n",
            "            '60.9%': 46,\n",
            "            '61%': 58,\n",
            "            '61.1%': 47,\n",
            "            '61.2%': 44,\n",
            "            '61.3%': 46,\n",
            "            '61.4%': 42,\n",
            "            '61.5%': 49,\n",
            "            '61.6%': 42,\n",
            "            '61.7%': 41,\n",
            "            '61.8%': 46,\n",
            "            '61.9%': 50,\n",
            "            '62%': 51,\n",
            "            '62.1%': 41,\n",
            "            '62.2%': 39,\n",
            "            '62.3%': 51,\n",
            "            '62.31%': 1,\n",
            "            '62.4%': 39,\n",
            "            '62.5%': 45,\n",
            "            '62.6%': 44,\n",
            "            '62.7%': 37,\n",
            "            '62.8%': 55,\n",
            "            '62.9%': 47,\n",
            "            '63%': 63,\n",
            "            '63.1%': 39,\n",
            "            '63.2%': 41,\n",
            "            '63.3%': 47,\n",
            "            '63.4%': 44,\n",
            "            '63.5%': 43,\n",
            "            '63.6%': 41,\n",
            "            '63.7%': 48,\n",
            "            '63.8%': 40,\n",
            "            '63.9%': 46,\n",
            "            '64%': 35,\n",
            "            '64.1%': 49,\n",
            "            '64.2%': 48,\n",
            "            '64.3%': 53,\n",
            "            '64.4%': 31,\n",
            "            '64.5%': 44,\n",
            "            '64.6%': 60,\n",
            "            '64.7%': 50,\n",
            "            '64.8%': 56,\n",
            "            '64.9%': 46,\n",
            "            '65%': 51,\n",
            "            '65.1%': 35,\n",
            "            '65.2%': 45,\n",
            "            '65.3%': 40,\n",
            "            '65.4%': 49,\n",
            "            '65.5%': 40,\n",
            "            '65.6%': 44,\n",
            "            '65.7%': 42,\n",
            "            '65.8%': 44,\n",
            "            '65.9%': 57,\n",
            "            '66%': 43,\n",
            "            '66.1%': 51,\n",
            "            '66.2%': 51,\n",
            "            '66.3%': 50,\n",
            "            '66.4%': 29,\n",
            "            '66.5%': 46,\n",
            "            '66.6%': 62,\n",
            "            '66.7%': 59,\n",
            "            '66.8%': 51,\n",
            "            '66.9%': 57,\n",
            "            '67%': 51,\n",
            "            '67.1%': 41,\n",
            "            '67.2%': 41,\n",
            "            '67.3%': 42,\n",
            "            '67.4%': 45,\n",
            "            '67.5%': 43,\n",
            "            '67.6%': 49,\n",
            "            '67.7%': 52,\n",
            "            '67.8%': 39,\n",
            "            '67.9%': 41,\n",
            "            '68%': 43,\n",
            "            '68.1%': 39,\n",
            "            '68.2%': 31,\n",
            "            '68.3%': 40,\n",
            "            '68.4%': 42,\n",
            "            '68.5%': 47,\n",
            "            '68.6%': 57,\n",
            "            '68.7%': 54,\n",
            "            '68.8%': 48,\n",
            "            '68.9%': 44,\n",
            "            '69%': 42,\n",
            "            '69.1%': 48,\n",
            "            '69.14%': 1,\n",
            "            '69.2%': 40,\n",
            "            '69.3%': 48,\n",
            "            '69.4%': 49,\n",
            "            '69.5%': 42,\n",
            "            '69.6%': 47,\n",
            "            '69.7%': 37,\n",
            "            '69.8%': 34,\n",
            "            '69.9%': 44,\n",
            "            '69.98%': 1,\n",
            "            '7%': 33,\n",
            "            '7.1%': 35,\n",
            "            '7.2%': 50,\n",
            "            '7.28%': 1,\n",
            "            '7.3%': 36,\n",
            "            '7.4%': 46,\n",
            "            '7.43%': 1,\n",
            "            '7.5%': 35,\n",
            "            '7.6%': 43,\n",
            "            '7.64%': 1,\n",
            "            '7.7%': 39,\n",
            "            '7.8%': 37,\n",
            "            '7.9%': 36,\n",
            "            '70%': 53,\n",
            "            '70.1%': 56,\n",
            "            '70.2%': 41,\n",
            "            '70.26%': 1,\n",
            "            '70.3%': 53,\n",
            "            '70.4%': 61,\n",
            "            '70.5%': 43,\n",
            "            '70.6%': 32,\n",
            "            '70.7%': 42,\n",
            "            '70.8%': 55,\n",
            "            '70.9%': 45,\n",
            "            '70.94%': 1,\n",
            "            '71%': 30,\n",
            "            '71.1%': 46,\n",
            "            '71.2%': 46,\n",
            "            '71.3%': 43,\n",
            "            '71.4%': 41,\n",
            "            '71.5%': 45,\n",
            "            '71.6%': 41,\n",
            "            '71.7%': 51,\n",
            "            '71.8%': 39,\n",
            "            '71.9%': 50,\n",
            "            '72%': 42,\n",
            "            '72.1%': 57,\n",
            "            '72.2%': 43,\n",
            "            '72.3%': 40,\n",
            "            '72.4%': 49,\n",
            "            '72.5%': 47,\n",
            "            '72.6%': 45,\n",
            "            '72.7%': 44,\n",
            "            '72.8%': 33,\n",
            "            '72.9%': 37,\n",
            "            '73%': 38,\n",
            "            '73.1%': 39,\n",
            "            '73.2%': 50,\n",
            "            '73.3%': 43,\n",
            "            '73.4%': 49,\n",
            "            '73.5%': 40,\n",
            "            '73.6%': 38,\n",
            "            '73.7%': 53,\n",
            "            '73.8%': 50,\n",
            "            '73.9%': 45,\n",
            "            '74%': 34,\n",
            "            '74.1%': 26,\n",
            "            '74.2%': 35,\n",
            "            '74.3%': 43,\n",
            "            '74.4%': 49,\n",
            "            '74.5%': 46,\n",
            "            '74.6%': 33,\n",
            "            '74.7%': 46,\n",
            "            '74.8%': 40,\n",
            "            '74.9%': 55,\n",
            "            '75%': 44,\n",
            "            '75.1%': 43,\n",
            "            '75.2%': 43,\n",
            "            '75.3%': 39,\n",
            "            '75.4%': 38,\n",
            "            '75.5%': 54,\n",
            "            '75.6%': 41,\n",
            "            '75.7%': 44,\n",
            "            '75.8%': 40,\n",
            "            '75.9%': 37,\n",
            "            '76%': 53,\n",
            "            '76.1%': 36,\n",
            "            '76.2%': 39,\n",
            "            '76.3%': 38,\n",
            "            '76.4%': 35,\n",
            "            '76.5%': 36,\n",
            "            '76.6%': 58,\n",
            "            '76.7%': 49,\n",
            "            '76.8%': 50,\n",
            "            '76.9%': 39,\n",
            "            '77%': 46,\n",
            "            '77.1%': 42,\n",
            "            '77.2%': 49,\n",
            "            '77.3%': 49,\n",
            "            '77.4%': 47,\n",
            "            '77.5%': 40,\n",
            "            '77.6%': 38,\n",
            "            '77.63%': 1,\n",
            "            '77.7%': 38,\n",
            "            '77.8%': 35,\n",
            "            '77.9%': 47,\n",
            "            '78%': 40,\n",
            "            '78.1%': 39,\n",
            "            '78.2%': 41,\n",
            "            '78.3%': 43,\n",
            "            '78.4%': 38,\n",
            "            '78.5%': 45,\n",
            "            '78.6%': 40,\n",
            "            '78.7%': 58,\n",
            "            '78.8%': 47,\n",
            "            '78.9%': 36,\n",
            "            '79%': 46,\n",
            "            '79.1%': 35,\n",
            "            '79.2%': 44,\n",
            "            '79.3%': 32,\n",
            "            '79.4%': 48,\n",
            "            '79.5%': 51,\n",
            "            '79.6%': 44,\n",
            "            '79.7%': 49,\n",
            "            '79.8%': 44,\n",
            "            '79.9%': 40,\n",
            "            '8%': 26,\n",
            "            '8.01%': 1,\n",
            "            '8.1%': 39,\n",
            "            '8.2%': 40,\n",
            "            '8.3%': 38,\n",
            "            '8.4%': 40,\n",
            "            '8.46%': 1,\n",
            "            '8.49%': 1,\n",
            "            '8.5%': 38,\n",
            "            '8.58%': 1,\n",
            "            '8.6%': 39,\n",
            "            '8.7%': 54,\n",
            "            '8.8%': 40,\n",
            "            '8.9%': 45,\n",
            "            '80%': 48,\n",
            "            '80.1%': 49,\n",
            "            '80.2%': 44,\n",
            "            '80.3%': 45,\n",
            "            '80.4%': 38,\n",
            "            '80.5%': 44,\n",
            "            '80.6%': 50,\n",
            "            '80.7%': 51,\n",
            "            '80.8%': 47,\n",
            "            '80.9%': 40,\n",
            "            '81%': 49,\n",
            "            '81.1%': 53,\n",
            "            '81.2%': 41,\n",
            "            '81.3%': 40,\n",
            "            '81.31%': 1,\n",
            "            '81.4%': 38,\n",
            "            '81.5%': 44,\n",
            "            '81.6%': 45,\n",
            "            '81.7%': 37,\n",
            "            '81.8%': 35,\n",
            "            '81.9%': 38,\n",
            "            '82%': 41,\n",
            "            '82.1%': 34,\n",
            "            '82.2%': 44,\n",
            "            '82.3%': 43,\n",
            "            '82.4%': 41,\n",
            "            '82.5%': 44,\n",
            "            '82.6%': 34,\n",
            "            '82.7%': 47,\n",
            "            '82.8%': 43,\n",
            "            '82.9%': 50,\n",
            "            '83%': 45,\n",
            "            '83.1%': 42,\n",
            "            '83.2%': 36,\n",
            "            '83.3%': 48,\n",
            "            '83.4%': 32,\n",
            "            '83.5%': 46,\n",
            "            '83.6%': 41,\n",
            "            '83.7%': 47,\n",
            "            '83.8%': 35,\n",
            "            '83.9%': 37,\n",
            "            '84%': 41,\n",
            "            '84.1%': 43,\n",
            "            '84.2%': 34,\n",
            "            '84.3%': 45,\n",
            "            '84.4%': 37,\n",
            "            '84.5%': 39,\n",
            "            '84.6%': 46,\n",
            "            '84.7%': 37,\n",
            "            '84.8%': 48,\n",
            "            '84.9%': 36,\n",
            "            '85%': 41,\n",
            "            '85.1%': 44,\n",
            "            '85.2%': 46,\n",
            "            '85.3%': 35,\n",
            "            '85.4%': 45,\n",
            "            '85.5%': 36,\n",
            "            '85.6%': 38,\n",
            "            '85.7%': 48,\n",
            "            '85.8%': 31,\n",
            "            '85.9%': 23,\n",
            "            '86%': 35,\n",
            "            '86.1%': 44,\n",
            "            '86.2%': 38,\n",
            "            '86.3%': 37,\n",
            "            '86.4%': 42,\n",
            "            '86.5%': 32,\n",
            "            '86.6%': 33,\n",
            "            '86.7%': 39,\n",
            "            '86.8%': 32,\n",
            "            '86.9%': 23,\n",
            "            '87%': 33,\n",
            "            '87.1%': 29,\n",
            "            '87.2%': 31,\n",
            "            '87.3%': 35,\n",
            "            '87.4%': 48,\n",
            "            '87.5%': 25,\n",
            "            '87.6%': 50,\n",
            "            '87.7%': 43,\n",
            "            '87.8%': 40,\n",
            "            '87.9%': 34,\n",
            "            '88%': 51,\n",
            "            '88.1%': 39,\n",
            "            '88.2%': 34,\n",
            "            '88.3%': 32,\n",
            "            '88.4%': 27,\n",
            "            '88.48%': 1,\n",
            "            '88.5%': 44,\n",
            "            '88.6%': 38,\n",
            "            '88.7%': 34,\n",
            "            '88.8%': 30,\n",
            "            '88.9%': 38,\n",
            "            '89%': 43,\n",
            "            '89.1%': 39,\n",
            "            '89.2%': 31,\n",
            "            '89.3%': 50,\n",
            "            '89.4%': 48,\n",
            "            '89.5%': 58,\n",
            "            '89.6%': 39,\n",
            "            '89.7%': 27,\n",
            "            '89.8%': 39,\n",
            "            '89.9%': 55,\n",
            "            '9%': 39,\n",
            "            '9.1%': 34,\n",
            "            '9.2%': 47,\n",
            "            '9.3%': 36,\n",
            "            '9.34%': 1,\n",
            "            '9.4%': 41,\n",
            "            '9.5%': 36,\n",
            "            '9.6%': 35,\n",
            "            '9.7%': 40,\n",
            "            '9.71%': 1,\n",
            "            '9.8%': 24,\n",
            "            '9.9%': 33,\n",
            "            '90%': 41,\n",
            "            '90.1%': 37,\n",
            "            '90.2%': 37,\n",
            "            '90.3%': 47,\n",
            "            '90.4%': 33,\n",
            "            '90.5%': 31,\n",
            "            '90.6%': 38,\n",
            "            '90.7%': 40,\n",
            "            '90.8%': 35,\n",
            "            '90.9%': 34,\n",
            "            '91%': 30,\n",
            "            '91.1%': 37,\n",
            "            '91.2%': 39,\n",
            "            '91.3%': 42,\n",
            "            '91.4%': 24,\n",
            "            '91.5%': 27,\n",
            "            '91.6%': 32,\n",
            "            '91.7%': 34,\n",
            "            '91.8%': 41,\n",
            "            '91.9%': 26,\n",
            "            '92%': 35,\n",
            "            '92.1%': 35,\n",
            "            '92.2%': 48,\n",
            "            '92.3%': 41,\n",
            "            '92.4%': 33,\n",
            "            '92.5%': 36,\n",
            "            '92.6%': 41,\n",
            "            '92.7%': 43,\n",
            "            '92.8%': 22,\n",
            "            '92.9%': 41,\n",
            "            '93%': 39,\n",
            "            '93.1%': 37,\n",
            "            '93.2%': 44,\n",
            "            '93.3%': 36,\n",
            "            '93.4%': 35,\n",
            "            '93.5%': 40,\n",
            "            '93.6%': 39,\n",
            "            '93.7%': 35,\n",
            "            '93.8%': 46,\n",
            "            '93.9%': 41,\n",
            "            '94%': 35,\n",
            "            '94.1%': 34,\n",
            "            '94.2%': 36,\n",
            "            '94.3%': 36,\n",
            "            '94.4%': 35,\n",
            "            '94.46%': 1,\n",
            "            '94.5%': 42,\n",
            "            '94.6%': 45,\n",
            "            '94.7%': 38,\n",
            "            '94.8%': 35,\n",
            "            '94.9%': 36,\n",
            "            '95%': 34,\n",
            "            '95.1%': 31,\n",
            "            '95.2%': 27,\n",
            "            '95.3%': 43,\n",
            "            '95.4%': 33,\n",
            "            '95.5%': 44,\n",
            "            '95.6%': 32,\n",
            "            '95.7%': 28,\n",
            "            '95.8%': 46,\n",
            "            '95.9%': 37,\n",
            "            '96%': 38,\n",
            "            '96.1%': 35,\n",
            "            '96.2%': 36,\n",
            "            '96.3%': 31,\n",
            "            '96.4%': 34,\n",
            "            '96.5%': 32,\n",
            "            '96.6%': 30,\n",
            "            '96.7%': 34,\n",
            "            '96.8%': 33,\n",
            "            '96.9%': 32,\n",
            "            '97%': 31,\n",
            "            '97.1%': 39,\n",
            "            '97.2%': 32,\n",
            "            '97.3%': 36,\n",
            "            '97.4%': 33,\n",
            "            '97.5%': 35,\n",
            "            '97.6%': 35,\n",
            "            '97.7%': 45,\n",
            "            '97.8%': 38,\n",
            "            '97.9%': 27,\n",
            "            '98%': 28,\n",
            "            '98.1%': 25,\n",
            "            '98.2%': 25,\n",
            "            '98.3%': 36,\n",
            "            '98.4%': 31,\n",
            "            '98.5%': 29,\n",
            "            '98.6%': 36,\n",
            "            '98.7%': 27,\n",
            "            '98.8%': 38,\n",
            "            '98.9%': 20,\n",
            "            '99%': 35,\n",
            "            '99.1%': 33,\n",
            "            '99.2%': 22,\n",
            "            '99.3%': 32,\n",
            "            '99.4%': 24,\n",
            "            '99.5%': 24,\n",
            "            '99.6%': 27,\n",
            "            '99.7%': 32,\n",
            "            '99.8%': 25,\n",
            "            '99.9%': 29}}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OF-M6SwgS-70"
      },
      "source": [
        "## How about the term columns, change to integer ?\n",
        "The term column only has two distinct values '36 months' and '60 months'.  Prefer to treat it as categorical rather than numeric. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "CHdSINnoS-71",
        "outputId": "45d1a513-73bc-4e5c-cf55-80ac18e2da95"
      },
      "source": [
        "pp.pprint(uniqueDicts[5])"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "{'col': 'term',\n",
            " 'non-numeric': True,\n",
            " 'values': {' 36 months': 31534, ' 60 months': 11001}}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9LUF_KNeS-72"
      },
      "source": [
        "## Recall date columns is identified from the first 5 rows\n",
        "They should be read as date instead of string and not categorical. Put the column name in a list and pass the list to parse_date parmeter of read_csv"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "352YJH4-S-73"
      },
      "source": [
        "dateCols=['issue_d','earliest_cr_line','last_pymnt_d','last_credit_pull_d']"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Tu6zpDDDS-74"
      },
      "source": [
        "## Let's clean the int_rate column and look for string columns contain values that are less than 50% unique again\n",
        "\n",
        "Add a small function to replace % char in int_rate column and set the column type to float, then build the list of unique values dict again"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-6AkYTp7S-74"
      },
      "source": [
        "def cleanRateCol(loanChunk):\n",
        "    loanChunk['int_rate']=loanChunk['int_rate'].str.replace('%','')\n",
        "    loanChunk['int_rate']=loanChunk['int_rate'].astype('float')\n",
        "    loanChunk['revol_util']=loanChunk['revol_util'].str.replace('%','')\n",
        "    loanChunk['revol_util']=loanChunk['revol_util'].astype('float')\n",
        "\n",
        "\n",
        "    \n",
        "    return loanChunk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FbgChfNxS-75",
        "outputId": "4d7017da-7a1f-4022-a94b-6a034ac92121"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "totalNumberOfRows=0\n",
        "uniqueDicts=[{'col': col, 'non-numeric':False,'values':{}} for col in loan.columns]\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    totalNumberOfRows+=loanChunk.shape[0]\n",
        "    \n",
        "    for idx,col in enumerate(loanChunk):\n",
        "        if loanChunk[col].dtypes=='object':\n",
        "            uniqueDicts[idx]['non-numeric']=True\n",
        "            for cell in loanChunk[col]:\n",
        "                if cell in uniqueDicts[idx]['values']:\n",
        "                    uniqueDicts[idx]['values'][cell]+=1\n",
        "                else:\n",
        "                    uniqueDicts[idx]['values'][cell]=1\n",
        "    \n",
        "\n",
        "isCatCols=[False for x in uniqueDicts]\n",
        "\n",
        "for idx,uniqueDict in enumerate(uniqueDicts):\n",
        "    if uniqueDict['non-numeric']==True:\n",
        "        noOfUniqueValues=len(uniqueDict['values'].keys())\n",
        "        if noOfUniqueValues/totalNumberOfRows<0.5:\n",
        "            isCatCols[idx]=True        \n",
        "\n",
        "catCols=[y  for x,y in zip(isCatCols,loanChunk.columns) if x==True]\n",
        "catCols"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['term',\n",
              " 'grade',\n",
              " 'sub_grade',\n",
              " 'emp_length',\n",
              " 'home_ownership',\n",
              " 'verification_status',\n",
              " 'loan_status',\n",
              " 'pymnt_plan',\n",
              " 'purpose',\n",
              " 'title',\n",
              " 'zip_code',\n",
              " 'addr_state',\n",
              " 'initial_list_status',\n",
              " 'application_type']"
            ]
          },
          "execution_count": 21,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TYc7pAZsS-76"
      },
      "source": [
        "## Column purpose, title are categorical ?  Probablly not  ...\n",
        "\n",
        "For the initial exploration of the first 5 rows of the loan data set, puropse and titile look likie descriptive text columns.Descrpitive text columns are not common candidiates of 'categorical' as they are free text columns.  These two columns need further explore why unique values are taking up below 50%.\n",
        "\n",
        "Print out the unique values of purpose column and value of title column that occur more than 10 times.  It is found that title has repetitive text of same meaning of different form (e.g 'wedding','wedding loan','Wedding Fund') and the purpose column already summarize the repetitive meaning of title.  So, the title column can be dropped and keep the purpose column as categorical columns."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DzxWS7lIS-76",
        "outputId": "c19f439b-5e1b-4a56-c21a-903f2ac841d9"
      },
      "source": [
        "uniqueDicts[18]"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'col': 'purpose',\n",
              " 'non-numeric': True,\n",
              " 'values': {'car': 1615,\n",
              "  'credit_card': 5477,\n",
              "  'debt_consolidation': 19776,\n",
              "  'educational': 422,\n",
              "  'home_improvement': 3199,\n",
              "  'house': 426,\n",
              "  'major_purchase': 2311,\n",
              "  'medical': 753,\n",
              "  'moving': 629,\n",
              "  'other': 4425,\n",
              "  'renewable_energy': 106,\n",
              "  'small_business': 1992,\n",
              "  'vacation': 400,\n",
              "  'wedding': 1004}}"
            ]
          },
          "execution_count": 22,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5052qONLS-77",
        "outputId": "aa57009d-62b9-4220-ddb3-98dadb840dd2"
      },
      "source": [
        "print(uniqueDicts[19]['col'])\n",
        "for value in uniqueDicts[19]['values'].items():\n",
        "    if value[1]>10:\n",
        "        print(value)\n",
        "    "
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "title\n",
            "(nan, 13)\n",
            "('Dept Consolidation', 13)\n",
            "('car', 30)\n",
            "('HOME IMPROVEMENT', 15)\n",
            "('My loan', 34)\n",
            "('Business Expansion', 13)\n",
            "('Freedom Loan', 21)\n",
            "('Other Loan', 192)\n",
            "('pool loan', 13)\n",
            "('Pay off', 17)\n",
            "('Medical Bills', 11)\n",
            "('Boat Loan', 25)\n",
            "('Consolidate', 141)\n",
            "('Home Repairs', 11)\n",
            "('CC Refinance', 27)\n",
            "('Consolidating Credit Card Debt', 14)\n",
            "('Pay off credit card', 11)\n",
            "('Debit Consolidation', 20)\n",
            "('moving', 15)\n",
            "('Purchase', 12)\n",
            "('Lending Club Loan', 33)\n",
            "('Bill Consolidation', 48)\n",
            "('loan1', 11)\n",
            "('Debt loan', 13)\n",
            "('Motorcycle loan', 16)\n",
            "('truck', 11)\n",
            "('Consolidate Bills', 12)\n",
            "('Loan Consolidation', 37)\n",
            "('consolidate debt', 25)\n",
            "('debt_consolidation', 68)\n",
            "('Consolidate Credit Cards', 16)\n",
            "('business', 16)\n",
            "('medical expenses', 12)\n",
            "('Fresh Start', 19)\n",
            "('Engagement Ring', 60)\n",
            "('Debt Relief', 17)\n",
            "('Credit Card Pay Off', 21)\n",
            "('Pay off credit cards', 30)\n",
            "('Pool loan', 12)\n",
            "('Personal', 330)\n",
            "('get out of debt', 21)\n",
            "('Home Improvement Loan', 255)\n",
            "('CONSOLIDATION', 14)\n",
            "('Pay Off Credit Cards', 20)\n",
            "('motorcycle', 35)\n",
            "('Card Consolidation', 14)\n",
            "('Moving Expenses', 22)\n",
            "('Help', 14)\n",
            "('home', 24)\n",
            "('Consolidating Debt', 19)\n",
            "('personal ', 11)\n",
            "('consolidation loan', 45)\n",
            "('medical', 28)\n",
            "('Loan 1', 49)\n",
            "('Debt Consolidation ', 67)\n",
            "('Debt Loan', 64)\n",
            "('boat', 18)\n",
            "('credit_card', 14)\n",
            "('credit card payoff', 25)\n",
            "('Debt Consolidate', 20)\n",
            "('refinance', 15)\n",
            "('swimming pool', 12)\n",
            "('Consolidate Debt', 64)\n",
            "('Personal Loan ', 16)\n",
            "('Credit Card Refinance Loan', 20)\n",
            "('Car Loan', 206)\n",
            "('credit card pay off', 11)\n",
            "('Consolidation ', 19)\n",
            "('My Personal Loan', 19)\n",
            "('Credit Card loan', 13)\n",
            "('Debt Consolidation 2011', 17)\n",
            "('Pay Off', 22)\n",
            "('Motorcycle Loan', 46)\n",
            "('Student Loan', 27)\n",
            "('Moving', 54)\n",
            "('Major Purchase', 66)\n",
            "('LOAN', 17)\n",
            "('Home Improvements', 48)\n",
            "('auto', 14)\n",
            "('Consolidated', 11)\n",
            "('PERSONAL LOAN', 21)\n",
            "('Financial Freedom', 15)\n",
            "('Car Repair', 18)\n",
            "('Loan', 218)\n",
            "('Bike', 12)\n",
            "('Pool Loan', 42)\n",
            "('lending club loan', 11)\n",
            "('my loan', 53)\n",
            "('auto loan', 15)\n",
            "('Medical Loan', 64)\n",
            "('debt consolidation', 532)\n",
            "('Payoff', 46)\n",
            "('Boat', 15)\n",
            "('House', 18)\n",
            "('credit', 13)\n",
            "('credit card debt', 13)\n",
            "('My Loan', 96)\n",
            "('Loan 2', 13)\n",
            "('Major purchase', 11)\n",
            "('Bills', 29)\n",
            "('Credit Card Refinancing', 32)\n",
            "('Debt Consolidation', 2259)\n",
            "('home improvement', 117)\n",
            "('Debt Freedom', 11)\n",
            "('debt consolidation ', 20)\n",
            "('Loan consolidation', 15)\n",
            "('Consolidate credit card debt', 13)\n",
            "('Debt consolidation', 347)\n",
            "('Wedding ', 11)\n",
            "('Relocation Loan', 14)\n",
            "('Consolidation loan', 41)\n",
            "('Credit Cards', 95)\n",
            "('House Loan', 41)\n",
            "('Credit cards', 20)\n",
            "('Debt consolidation loan', 48)\n",
            "('Credit Card Debt Consolidation', 63)\n",
            "('bill consolidation', 20)\n",
            "('debt', 56)\n",
            "('Debt Consolidation Loan', 1760)\n",
            "('Paying off credit cards', 17)\n",
            "('Home improvement', 66)\n",
            "('Wedding Fund', 11)\n",
            "('Credit Card', 124)\n",
            "('Business', 33)\n",
            "('New Start', 14)\n",
            "('credit card refinance', 28)\n",
            "('CC Debt', 12)\n",
            "('credit cards', 40)\n",
            "('pool', 20)\n",
            "('New Roof', 11)\n",
            "('CC Consolidation', 54)\n",
            "('Payoff Credit Cards', 13)\n",
            "('loan 1', 19)\n",
            "('vacation loan', 15)\n",
            "('Credit Card consolidation', 20)\n",
            "('Consolidate Credit Card Debt', 27)\n",
            "('Education', 16)\n",
            "('Credit card consolidation', 45)\n",
            "('Home Improvement', 373)\n",
            "('debt free', 26)\n",
            "('Small_Business Loan', 14)\n",
            "('Lending Club', 29)\n",
            "('Freedom', 81)\n",
            "('Debt Consolodation', 27)\n",
            "('Credit Card Refi', 37)\n",
            "('car loan', 29)\n",
            "('Auto', 20)\n",
            "('Pool', 41)\n",
            "('major purchase', 26)\n",
            "('Loan1', 24)\n",
            "('Debt consolidation ', 14)\n",
            "('consolidate', 80)\n",
            "('Home improvement loan', 12)\n",
            "('Home_Improvement Loan', 11)\n",
            "('Credit Card Consolidation Loan', 52)\n",
            "('freedom', 48)\n",
            "('Credit card loan', 12)\n",
            "('Debt consolidation loan ', 15)\n",
            "('wedding loan', 27)\n",
            "('Credit Card Loan', 323)\n",
            "('Refinancing', 13)\n",
            "('Home', 38)\n",
            "('Pay off Debt', 15)\n",
            "('Credit Card Consolidation', 370)\n",
            "('Credit Card Refinance', 147)\n",
            "('Vacation Loan', 38)\n",
            "('Debt Consol', 11)\n",
            "('CC', 14)\n",
            "('loan', 90)\n",
            "('Get out of debt', 23)\n",
            "('Personal ', 14)\n",
            "('Debt', 89)\n",
            "('Wedding', 160)\n",
            "('Medical Expenses', 23)\n",
            "('LendingClub Loan', 12)\n",
            "('Motorcycle', 66)\n",
            "('credit card', 36)\n",
            "('credit card loan', 18)\n",
            "('Credit card payoff', 21)\n",
            "('Wedding Expenses', 52)\n",
            "('Truck Loan', 11)\n",
            "('Auto Loan', 42)\n",
            "('Wedding Loan', 217)\n",
            "('Car loan', 31)\n",
            "('credit card consolidation', 58)\n",
            "('Ring', 18)\n",
            "('Roof', 19)\n",
            "('Car', 32)\n",
            "('consolidation', 217)\n",
            "('Debt Payoff', 37)\n",
            "('Consolidate debt', 40)\n",
            "('motorcycle loan', 12)\n",
            "('pay off', 20)\n",
            "('Medical', 124)\n",
            "('other', 13)\n",
            "('business loan', 19)\n",
            "('Credit Refinance', 11)\n",
            "('Debt ', 13)\n",
            "('Truck', 13)\n",
            "('Vacation', 54)\n",
            "('Debt Free', 47)\n",
            "('Credit Card payoff', 12)\n",
            "('Consolidation', 547)\n",
            "('credit card refinance loan', 11)\n",
            "('Investment', 11)\n",
            "('Credit Consolidation', 51)\n",
            "('Pay off debt', 18)\n",
            "('Credit card refinance', 24)\n",
            "('home improvement loan', 16)\n",
            "('Moving Loan', 78)\n",
            "('Credit Card Payoff Loan', 18)\n",
            "('Debt Consolidation loan ', 22)\n",
            "('Swimming Pool', 23)\n",
            "('DEBT CONSOLIDATION', 59)\n",
            "('Personal Loan', 708)\n",
            "('Small Business', 48)\n",
            "('Debt Consolidation loan', 24)\n",
            "('vacation', 34)\n",
            "('2010', 11)\n",
            "('Working Capital', 15)\n",
            "('Small Business loan', 24)\n",
            "('Debt Reduction', 18)\n",
            "('Major Purchase Loan', 146)\n",
            "('pay off credit cards', 26)\n",
            "('Surgery', 11)\n",
            "('Credit Card Debt', 54)\n",
            "('CC Loan', 23)\n",
            "('personal', 247)\n",
            "('Business Loan', 88)\n",
            "('moving loan', 17)\n",
            "('home improvements', 20)\n",
            "('Kitchen Remodel', 18)\n",
            "('PERSONAL', 14)\n",
            "('personal loan', 266)\n",
            "('Pay off Credit Cards', 21)\n",
            "('CC Payoff', 37)\n",
            "('payoff', 49)\n",
            "('Consolidation Loan', 264)\n",
            "('CC Refi', 19)\n",
            "('Personal loan', 80)\n",
            "('help', 13)\n",
            "('Business loan', 12)\n",
            "('Small Business Loan', 333)\n",
            "('debt relief', 14)\n",
            "('Refinance', 48)\n",
            "('debt consolidation loan', 59)\n",
            "('Credit Card Payoff', 164)\n",
            "('Home Loan', 24)\n",
            "('Taxes', 20)\n",
            "('wedding', 38)\n",
            "('credit consolidation', 15)\n",
            "('Other', 59)\n",
            "('Debt payoff', 14)\n",
            "('debt loan', 31)\n",
            "('bills', 27)\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cYg9uhobS-78"
      },
      "source": [
        "## Rebulid the list of categorical columns and unique values dict of categorical columns again\n",
        "\n",
        "So we write a small function to drop the titile column , rebulid the list of categorical columns and unique values dict of categorical columns agaian after dropping the title column\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5_ePdcM8S-78"
      },
      "source": [
        "def dropTitleCol(loanChunk):\n",
        "    loanChunk.drop(columns=['title'],axis=1,inplace=True)\n",
        "    \n",
        "    return loanChunk"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rq3ipw9oS-79",
        "outputId": "00f830b3-4378-47d1-83b1-d75e8ee4c210"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "totalNumberOfRows=0\n",
        "uniqueDicts=[{'col': col, 'non-numeric':False,'values':{}} for col in loan.columns]\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    loanChunk=dropTitleCol(loanChunk)\n",
        "    \n",
        "    totalNumberOfRows+=loanChunk.shape[0]\n",
        "    \n",
        "    for idx,col in enumerate(loanChunk):\n",
        "        if loanChunk[col].dtypes=='object':\n",
        "            uniqueDicts[idx]['non-numeric']=True\n",
        "            for cell in loanChunk[col]:\n",
        "                if cell in uniqueDicts[idx]['values']:\n",
        "                    uniqueDicts[idx]['values'][cell]+=1\n",
        "                else:\n",
        "                    uniqueDicts[idx]['values'][cell]=1\n",
        "\n",
        "isCatCols=[False for x in uniqueDicts]\n",
        "\n",
        "for idx,uniqueDict in enumerate(uniqueDicts):\n",
        "    if uniqueDict['non-numeric']==True:\n",
        "        noOfUniqueValues=len(uniqueDict['values'].keys())\n",
        "        if noOfUniqueValues/totalNumberOfRows<0.5:\n",
        "            isCatCols[idx]=True        \n",
        "\n",
        "catCols=[y  for x,y in zip(isCatCols,loanChunk.columns) if x==True]\n",
        "catCols    "
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['term',\n",
              " 'grade',\n",
              " 'sub_grade',\n",
              " 'emp_length',\n",
              " 'home_ownership',\n",
              " 'verification_status',\n",
              " 'loan_status',\n",
              " 'pymnt_plan',\n",
              " 'purpose',\n",
              " 'zip_code',\n",
              " 'addr_state',\n",
              " 'initial_list_status',\n",
              " 'application_type']"
            ]
          },
          "execution_count": 25,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Ba6MyNGS-7-"
      },
      "source": [
        "## For each identified category column, build a unique value list across all chunk\n",
        "\n",
        "Write a small function to create a CategoricalDtype for each column, holding the unique values across all chunks of that column."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Mn1xZq9JS-7_"
      },
      "source": [
        "from pandas.api.types import CategoricalDtype\n",
        "\n",
        "def categorizeLoanChunk(loanChunk):\n",
        "    for idx,col in enumerate(loanChunk.columns):\n",
        "        if col in catCols:\n",
        "            catValues=[value for value in uniqueDicts[idx]['values'].keys() if str(value)!='nan']\n",
        "            cat_type = CategoricalDtype(categories=catValues)            \n",
        "            loanChunk[col]=loanChunk[col].astype(cat_type)\n",
        "    \n",
        "    return loanChunk        \n",
        "            "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kR85lr0KS-8B"
      },
      "source": [
        "## A two pass startegy to categorized all category column\n",
        "We want to use the same categoryization scheme for each cateogry column across all chunks.  To achieve this goal, a two pass startegy is emplyeed to categorize all category column across all chunks.\n",
        "\n",
        "- First build the category column list and unqiue values dict across all chunks\n",
        "- Then use the unique values dict to create a CategoryDtype for each category column to categorize the column.\n",
        "\n",
        "To verify the result, list the datatypes of each chunks and sample check a category column of the last loanChunk."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "irfRNY88S-8C",
        "outputId": "c112518e-2a3e-4485-d543-ee457d3d166e"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "totalNumberOfRows=0\n",
        "uniqueDicts=[{'col': col, 'non-numeric':False,'values':{}} for col in loan.columns]\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    loanChunk=dropTitleCol(loanChunk)\n",
        "    \n",
        "    totalNumberOfRows+=loanChunk.shape[0]\n",
        "    \n",
        "    for idx,col in enumerate(loanChunk):\n",
        "        if loanChunk[col].dtypes=='object':\n",
        "            uniqueDicts[idx]['non-numeric']=True\n",
        "            for cell in loanChunk[col]:\n",
        "                if cell in uniqueDicts[idx]['values']:\n",
        "                    uniqueDicts[idx]['values'][cell]+=1\n",
        "                else:\n",
        "                    uniqueDicts[idx]['values'][cell]=1\n",
        "\n",
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "loanChunkTypes=pd.DataFrame(columns=loanChunk.columns)\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    loanChunk=dropTitleCol(loanChunk)\n",
        "    loanChunk=categorizeLoanChunk(loanChunk)\n",
        "\n",
        "    loanChunkTypes.loc[len(loanChunkTypes)]=loanChunk.dtypes\n",
        "    \n",
        "loanChunkTypes"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>member_id</th>\n",
              "      <th>loan_amnt</th>\n",
              "      <th>funded_amnt</th>\n",
              "      <th>funded_amnt_inv</th>\n",
              "      <th>term</th>\n",
              "      <th>int_rate</th>\n",
              "      <th>installment</th>\n",
              "      <th>grade</th>\n",
              "      <th>sub_grade</th>\n",
              "      <th>emp_title</th>\n",
              "      <th>emp_length</th>\n",
              "      <th>home_ownership</th>\n",
              "      <th>annual_inc</th>\n",
              "      <th>verification_status</th>\n",
              "      <th>issue_d</th>\n",
              "      <th>loan_status</th>\n",
              "      <th>pymnt_plan</th>\n",
              "      <th>purpose</th>\n",
              "      <th>zip_code</th>\n",
              "      <th>addr_state</th>\n",
              "      <th>dti</th>\n",
              "      <th>delinq_2yrs</th>\n",
              "      <th>earliest_cr_line</th>\n",
              "      <th>inq_last_6mths</th>\n",
              "      <th>open_acc</th>\n",
              "      <th>pub_rec</th>\n",
              "      <th>revol_bal</th>\n",
              "      <th>revol_util</th>\n",
              "      <th>total_acc</th>\n",
              "      <th>initial_list_status</th>\n",
              "      <th>out_prncp</th>\n",
              "      <th>out_prncp_inv</th>\n",
              "      <th>total_pymnt</th>\n",
              "      <th>total_pymnt_inv</th>\n",
              "      <th>total_rec_prncp</th>\n",
              "      <th>total_rec_int</th>\n",
              "      <th>total_rec_late_fee</th>\n",
              "      <th>recoveries</th>\n",
              "      <th>collection_recovery_fee</th>\n",
              "      <th>last_pymnt_d</th>\n",
              "      <th>last_pymnt_amnt</th>\n",
              "      <th>last_credit_pull_d</th>\n",
              "      <th>collections_12_mths_ex_med</th>\n",
              "      <th>policy_code</th>\n",
              "      <th>application_type</th>\n",
              "      <th>acc_now_delinq</th>\n",
              "      <th>chargeoff_within_12_mths</th>\n",
              "      <th>delinq_amnt</th>\n",
              "      <th>pub_rec_bankruptcies</th>\n",
              "      <th>tax_liens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       id member_id loan_amnt funded_amnt funded_amnt_inv      term int_rate  \\\n",
              "0   int64   float64   float64     float64         float64  category  float64   \n",
              "1   int64   float64   float64     float64         float64  category  float64   \n",
              "2   int64   float64   float64     float64         float64  category  float64   \n",
              "3   int64   float64   float64     float64         float64  category  float64   \n",
              "4   int64   float64   float64     float64         float64  category  float64   \n",
              "5   int64   float64   float64     float64         float64  category  float64   \n",
              "6   int64   float64   float64     float64         float64  category  float64   \n",
              "7   int64   float64   float64     float64         float64  category  float64   \n",
              "8   int64   float64   float64     float64         float64  category  float64   \n",
              "9   int64   float64   float64     float64         float64  category  float64   \n",
              "10  int64   float64   float64     float64         float64  category  float64   \n",
              "11  int64   float64   float64     float64         float64  category  float64   \n",
              "12  int64   float64   float64     float64         float64  category  float64   \n",
              "13  int64   float64   float64     float64         float64  category  float64   \n",
              "14  int64   float64   float64     float64         float64  category  float64   \n",
              "\n",
              "   installment     grade sub_grade emp_title emp_length home_ownership  \\\n",
              "0      float64  category  category    object   category       category   \n",
              "1      float64  category  category    object   category       category   \n",
              "2      float64  category  category    object   category       category   \n",
              "3      float64  category  category    object   category       category   \n",
              "4      float64  category  category    object   category       category   \n",
              "5      float64  category  category    object   category       category   \n",
              "6      float64  category  category    object   category       category   \n",
              "7      float64  category  category    object   category       category   \n",
              "8      float64  category  category    object   category       category   \n",
              "9      float64  category  category    object   category       category   \n",
              "10     float64  category  category    object   category       category   \n",
              "11     float64  category  category    object   category       category   \n",
              "12     float64  category  category    object   category       category   \n",
              "13     float64  category  category    object   category       category   \n",
              "14     float64  category  category    object   category       category   \n",
              "\n",
              "   annual_inc verification_status         issue_d loan_status pymnt_plan  \\\n",
              "0     float64            category  datetime64[ns]    category   category   \n",
              "1     float64            category  datetime64[ns]    category   category   \n",
              "2     float64            category  datetime64[ns]    category   category   \n",
              "3     float64            category  datetime64[ns]    category   category   \n",
              "4     float64            category  datetime64[ns]    category   category   \n",
              "5     float64            category  datetime64[ns]    category   category   \n",
              "6     float64            category  datetime64[ns]    category   category   \n",
              "7     float64            category  datetime64[ns]    category   category   \n",
              "8     float64            category  datetime64[ns]    category   category   \n",
              "9     float64            category  datetime64[ns]    category   category   \n",
              "10    float64            category  datetime64[ns]    category   category   \n",
              "11    float64            category  datetime64[ns]    category   category   \n",
              "12    float64            category  datetime64[ns]    category   category   \n",
              "13    float64            category  datetime64[ns]    category   category   \n",
              "14    float64            category  datetime64[ns]    category   category   \n",
              "\n",
              "     purpose  zip_code addr_state      dti delinq_2yrs earliest_cr_line  \\\n",
              "0   category  category   category  float64     float64   datetime64[ns]   \n",
              "1   category  category   category  float64     float64   datetime64[ns]   \n",
              "2   category  category   category  float64     float64   datetime64[ns]   \n",
              "3   category  category   category  float64     float64   datetime64[ns]   \n",
              "4   category  category   category  float64     float64   datetime64[ns]   \n",
              "5   category  category   category  float64     float64   datetime64[ns]   \n",
              "6   category  category   category  float64     float64   datetime64[ns]   \n",
              "7   category  category   category  float64     float64   datetime64[ns]   \n",
              "8   category  category   category  float64     float64   datetime64[ns]   \n",
              "9   category  category   category  float64     float64   datetime64[ns]   \n",
              "10  category  category   category  float64     float64   datetime64[ns]   \n",
              "11  category  category   category  float64     float64   datetime64[ns]   \n",
              "12  category  category   category  float64     float64   datetime64[ns]   \n",
              "13  category  category   category  float64     float64   datetime64[ns]   \n",
              "14  category  category   category  float64     float64   datetime64[ns]   \n",
              "\n",
              "   inq_last_6mths open_acc  pub_rec revol_bal revol_util total_acc  \\\n",
              "0         float64  float64  float64   float64    float64   float64   \n",
              "1         float64  float64  float64   float64    float64   float64   \n",
              "2         float64  float64  float64   float64    float64   float64   \n",
              "3         float64  float64  float64   float64    float64   float64   \n",
              "4         float64  float64  float64   float64    float64   float64   \n",
              "5         float64  float64  float64   float64    float64   float64   \n",
              "6         float64  float64  float64   float64    float64   float64   \n",
              "7         float64  float64  float64   float64    float64   float64   \n",
              "8         float64  float64  float64   float64    float64   float64   \n",
              "9         float64  float64  float64   float64    float64   float64   \n",
              "10        float64  float64  float64   float64    float64   float64   \n",
              "11        float64  float64  float64   float64    float64   float64   \n",
              "12        float64  float64  float64   float64    float64   float64   \n",
              "13        float64  float64  float64   float64    float64   float64   \n",
              "14        float64  float64  float64   float64    float64   float64   \n",
              "\n",
              "   initial_list_status out_prncp out_prncp_inv total_pymnt total_pymnt_inv  \\\n",
              "0             category   float64       float64     float64         float64   \n",
              "1             category   float64       float64     float64         float64   \n",
              "2             category   float64       float64     float64         float64   \n",
              "3             category   float64       float64     float64         float64   \n",
              "4             category   float64       float64     float64         float64   \n",
              "5             category   float64       float64     float64         float64   \n",
              "6             category   float64       float64     float64         float64   \n",
              "7             category   float64       float64     float64         float64   \n",
              "8             category   float64       float64     float64         float64   \n",
              "9             category   float64       float64     float64         float64   \n",
              "10            category   float64       float64     float64         float64   \n",
              "11            category   float64       float64     float64         float64   \n",
              "12            category   float64       float64     float64         float64   \n",
              "13            category   float64       float64     float64         float64   \n",
              "14            category   float64       float64     float64         float64   \n",
              "\n",
              "   total_rec_prncp total_rec_int total_rec_late_fee recoveries  \\\n",
              "0          float64       float64            float64    float64   \n",
              "1          float64       float64            float64    float64   \n",
              "2          float64       float64            float64    float64   \n",
              "3          float64       float64            float64    float64   \n",
              "4          float64       float64            float64    float64   \n",
              "5          float64       float64            float64    float64   \n",
              "6          float64       float64            float64    float64   \n",
              "7          float64       float64            float64    float64   \n",
              "8          float64       float64            float64    float64   \n",
              "9          float64       float64            float64    float64   \n",
              "10         float64       float64            float64    float64   \n",
              "11         float64       float64            float64    float64   \n",
              "12         float64       float64            float64    float64   \n",
              "13         float64       float64            float64    float64   \n",
              "14         float64       float64            float64    float64   \n",
              "\n",
              "   collection_recovery_fee    last_pymnt_d last_pymnt_amnt last_credit_pull_d  \\\n",
              "0                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "1                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "2                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "3                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "4                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "5                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "6                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "7                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "8                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "9                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "10                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "11                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "12                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "13                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "14                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "\n",
              "   collections_12_mths_ex_med policy_code application_type acc_now_delinq  \\\n",
              "0                     float64     float64         category        float64   \n",
              "1                     float64     float64         category        float64   \n",
              "2                     float64     float64         category        float64   \n",
              "3                     float64     float64         category        float64   \n",
              "4                     float64     float64         category        float64   \n",
              "5                     float64     float64         category        float64   \n",
              "6                     float64     float64         category        float64   \n",
              "7                     float64     float64         category        float64   \n",
              "8                     float64     float64         category        float64   \n",
              "9                     float64     float64         category        float64   \n",
              "10                    float64     float64         category        float64   \n",
              "11                    float64     float64         category        float64   \n",
              "12                    float64     float64         category        float64   \n",
              "13                    float64     float64         category        float64   \n",
              "14                    float64     float64         category        float64   \n",
              "\n",
              "   chargeoff_within_12_mths delinq_amnt pub_rec_bankruptcies tax_liens  \n",
              "0                   float64     float64              float64   float64  \n",
              "1                   float64     float64              float64   float64  \n",
              "2                   float64     float64              float64   float64  \n",
              "3                   float64     float64              float64   float64  \n",
              "4                   float64     float64              float64   float64  \n",
              "5                   float64     float64              float64   float64  \n",
              "6                   float64     float64              float64   float64  \n",
              "7                   float64     float64              float64   float64  \n",
              "8                   float64     float64              float64   float64  \n",
              "9                   float64     float64              float64   float64  \n",
              "10                  float64     float64              float64   float64  \n",
              "11                  float64     float64              float64   float64  \n",
              "12                  float64     float64              float64   float64  \n",
              "13                  float64     float64              float64   float64  \n",
              "14                  float64     float64              float64   float64  "
            ]
          },
          "execution_count": 27,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xAZyd3LJS-8E",
        "outputId": "b5d75882-1865-4076-aeb3-24cc5a60dc39"
      },
      "source": [
        "print('No of distinct addr_state values in last chunk => {}'.format(len(loanChunk['addr_state'].cat.codes.unique())))\n",
        "\n",
        "print('No of category value used to categorized addr_state => {}'.format(len(loanChunk['addr_state'].cat.categories)))"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "No of distinct addr_state values in last chunk => 42\n",
            "No of category value used to categorized addr_state => 50\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7gRZRXejS-8G"
      },
      "source": [
        "## Process integer columns\n",
        "After the categorical column is processed, it is time to process numeric columns.  Recall that we have already identified columns that can be converted to integer column and saved the integer column list in colToInteger.\n",
        "\n",
        "A small function is created to covert the concerned column from float to integer.  The datatype of all columns across all chunks are then listed again to verify result."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bx_hH7qfS-8H",
        "outputId": "720dd1d3-0b74-4df2-a72e-9c525331519c"
      },
      "source": [
        "colToInteger"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "['member_id', 'loan_amnt', 'funded_amnt', 'revol_bal', 'policy_code']"
            ]
          },
          "execution_count": 29,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "2D3ubWGzS-8I"
      },
      "source": [
        "def convertColToInteger(loanChunk,colToInteger):\n",
        "    for col in loanChunk.columns:\n",
        "        if col in colToInteger:\n",
        "            loanChunk[col]=loanChunk[col].astype('int64')\n",
        "    return loanChunk        "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i5xwvCFDS-8J",
        "outputId": "0346cb84-e661-43bf-f7c9-5953d0b1dc9d"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "totalNumberOfRows=0\n",
        "uniqueDicts=[{'col': col, 'non-numeric':False,'values':{}} for col in loan.columns]\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    loanChunk=dropTitleCol(loanChunk)\n",
        "    \n",
        "    totalNumberOfRows+=loanChunk.shape[0]\n",
        "    \n",
        "    for idx,col in enumerate(loanChunk):\n",
        "        if loanChunk[col].dtypes=='object':\n",
        "            uniqueDicts[idx]['non-numeric']=True\n",
        "            for cell in loanChunk[col]:\n",
        "                if cell in uniqueDicts[idx]['values']:\n",
        "                    uniqueDicts[idx]['values'][cell]+=1\n",
        "                else:\n",
        "                    uniqueDicts[idx]['values'][cell]=1\n",
        "\n",
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "loanChunkTypes=pd.DataFrame(columns=loanChunk.columns)\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    loanChunk=dropTitleCol(loanChunk)\n",
        "    loanChunk=categorizeLoanChunk(loanChunk)\n",
        "    loanChunk=convertColToInteger(loanChunk,colToInteger)\n",
        "   \n",
        "    loanChunkTypes.loc[len(loanChunkTypes)]=loanChunk.dtypes\n",
        "\n",
        "loanChunkTypes"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>member_id</th>\n",
              "      <th>loan_amnt</th>\n",
              "      <th>funded_amnt</th>\n",
              "      <th>funded_amnt_inv</th>\n",
              "      <th>term</th>\n",
              "      <th>int_rate</th>\n",
              "      <th>installment</th>\n",
              "      <th>grade</th>\n",
              "      <th>sub_grade</th>\n",
              "      <th>emp_title</th>\n",
              "      <th>emp_length</th>\n",
              "      <th>home_ownership</th>\n",
              "      <th>annual_inc</th>\n",
              "      <th>verification_status</th>\n",
              "      <th>issue_d</th>\n",
              "      <th>loan_status</th>\n",
              "      <th>pymnt_plan</th>\n",
              "      <th>purpose</th>\n",
              "      <th>zip_code</th>\n",
              "      <th>addr_state</th>\n",
              "      <th>dti</th>\n",
              "      <th>delinq_2yrs</th>\n",
              "      <th>earliest_cr_line</th>\n",
              "      <th>inq_last_6mths</th>\n",
              "      <th>open_acc</th>\n",
              "      <th>pub_rec</th>\n",
              "      <th>revol_bal</th>\n",
              "      <th>revol_util</th>\n",
              "      <th>total_acc</th>\n",
              "      <th>initial_list_status</th>\n",
              "      <th>out_prncp</th>\n",
              "      <th>out_prncp_inv</th>\n",
              "      <th>total_pymnt</th>\n",
              "      <th>total_pymnt_inv</th>\n",
              "      <th>total_rec_prncp</th>\n",
              "      <th>total_rec_int</th>\n",
              "      <th>total_rec_late_fee</th>\n",
              "      <th>recoveries</th>\n",
              "      <th>collection_recovery_fee</th>\n",
              "      <th>last_pymnt_d</th>\n",
              "      <th>last_pymnt_amnt</th>\n",
              "      <th>last_credit_pull_d</th>\n",
              "      <th>collections_12_mths_ex_med</th>\n",
              "      <th>policy_code</th>\n",
              "      <th>application_type</th>\n",
              "      <th>acc_now_delinq</th>\n",
              "      <th>chargeoff_within_12_mths</th>\n",
              "      <th>delinq_amnt</th>\n",
              "      <th>pub_rec_bankruptcies</th>\n",
              "      <th>tax_liens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float64</td>\n",
              "      <td>int64</td>\n",
              "      <td>category</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "      <td>float64</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       id member_id loan_amnt funded_amnt funded_amnt_inv      term int_rate  \\\n",
              "0   int64     int64     int64       int64         float64  category  float64   \n",
              "1   int64     int64     int64       int64         float64  category  float64   \n",
              "2   int64     int64     int64       int64         float64  category  float64   \n",
              "3   int64     int64     int64       int64         float64  category  float64   \n",
              "4   int64     int64     int64       int64         float64  category  float64   \n",
              "5   int64     int64     int64       int64         float64  category  float64   \n",
              "6   int64     int64     int64       int64         float64  category  float64   \n",
              "7   int64     int64     int64       int64         float64  category  float64   \n",
              "8   int64     int64     int64       int64         float64  category  float64   \n",
              "9   int64     int64     int64       int64         float64  category  float64   \n",
              "10  int64     int64     int64       int64         float64  category  float64   \n",
              "11  int64     int64     int64       int64         float64  category  float64   \n",
              "12  int64     int64     int64       int64         float64  category  float64   \n",
              "13  int64     int64     int64       int64         float64  category  float64   \n",
              "14  int64     int64     int64       int64         float64  category  float64   \n",
              "\n",
              "   installment     grade sub_grade emp_title emp_length home_ownership  \\\n",
              "0      float64  category  category    object   category       category   \n",
              "1      float64  category  category    object   category       category   \n",
              "2      float64  category  category    object   category       category   \n",
              "3      float64  category  category    object   category       category   \n",
              "4      float64  category  category    object   category       category   \n",
              "5      float64  category  category    object   category       category   \n",
              "6      float64  category  category    object   category       category   \n",
              "7      float64  category  category    object   category       category   \n",
              "8      float64  category  category    object   category       category   \n",
              "9      float64  category  category    object   category       category   \n",
              "10     float64  category  category    object   category       category   \n",
              "11     float64  category  category    object   category       category   \n",
              "12     float64  category  category    object   category       category   \n",
              "13     float64  category  category    object   category       category   \n",
              "14     float64  category  category    object   category       category   \n",
              "\n",
              "   annual_inc verification_status         issue_d loan_status pymnt_plan  \\\n",
              "0     float64            category  datetime64[ns]    category   category   \n",
              "1     float64            category  datetime64[ns]    category   category   \n",
              "2     float64            category  datetime64[ns]    category   category   \n",
              "3     float64            category  datetime64[ns]    category   category   \n",
              "4     float64            category  datetime64[ns]    category   category   \n",
              "5     float64            category  datetime64[ns]    category   category   \n",
              "6     float64            category  datetime64[ns]    category   category   \n",
              "7     float64            category  datetime64[ns]    category   category   \n",
              "8     float64            category  datetime64[ns]    category   category   \n",
              "9     float64            category  datetime64[ns]    category   category   \n",
              "10    float64            category  datetime64[ns]    category   category   \n",
              "11    float64            category  datetime64[ns]    category   category   \n",
              "12    float64            category  datetime64[ns]    category   category   \n",
              "13    float64            category  datetime64[ns]    category   category   \n",
              "14    float64            category  datetime64[ns]    category   category   \n",
              "\n",
              "     purpose  zip_code addr_state      dti delinq_2yrs earliest_cr_line  \\\n",
              "0   category  category   category  float64     float64   datetime64[ns]   \n",
              "1   category  category   category  float64     float64   datetime64[ns]   \n",
              "2   category  category   category  float64     float64   datetime64[ns]   \n",
              "3   category  category   category  float64     float64   datetime64[ns]   \n",
              "4   category  category   category  float64     float64   datetime64[ns]   \n",
              "5   category  category   category  float64     float64   datetime64[ns]   \n",
              "6   category  category   category  float64     float64   datetime64[ns]   \n",
              "7   category  category   category  float64     float64   datetime64[ns]   \n",
              "8   category  category   category  float64     float64   datetime64[ns]   \n",
              "9   category  category   category  float64     float64   datetime64[ns]   \n",
              "10  category  category   category  float64     float64   datetime64[ns]   \n",
              "11  category  category   category  float64     float64   datetime64[ns]   \n",
              "12  category  category   category  float64     float64   datetime64[ns]   \n",
              "13  category  category   category  float64     float64   datetime64[ns]   \n",
              "14  category  category   category  float64     float64   datetime64[ns]   \n",
              "\n",
              "   inq_last_6mths open_acc  pub_rec revol_bal revol_util total_acc  \\\n",
              "0         float64  float64  float64     int64    float64   float64   \n",
              "1         float64  float64  float64     int64    float64   float64   \n",
              "2         float64  float64  float64     int64    float64   float64   \n",
              "3         float64  float64  float64     int64    float64   float64   \n",
              "4         float64  float64  float64     int64    float64   float64   \n",
              "5         float64  float64  float64     int64    float64   float64   \n",
              "6         float64  float64  float64     int64    float64   float64   \n",
              "7         float64  float64  float64     int64    float64   float64   \n",
              "8         float64  float64  float64     int64    float64   float64   \n",
              "9         float64  float64  float64     int64    float64   float64   \n",
              "10        float64  float64  float64     int64    float64   float64   \n",
              "11        float64  float64  float64     int64    float64   float64   \n",
              "12        float64  float64  float64     int64    float64   float64   \n",
              "13        float64  float64  float64     int64    float64   float64   \n",
              "14        float64  float64  float64     int64    float64   float64   \n",
              "\n",
              "   initial_list_status out_prncp out_prncp_inv total_pymnt total_pymnt_inv  \\\n",
              "0             category   float64       float64     float64         float64   \n",
              "1             category   float64       float64     float64         float64   \n",
              "2             category   float64       float64     float64         float64   \n",
              "3             category   float64       float64     float64         float64   \n",
              "4             category   float64       float64     float64         float64   \n",
              "5             category   float64       float64     float64         float64   \n",
              "6             category   float64       float64     float64         float64   \n",
              "7             category   float64       float64     float64         float64   \n",
              "8             category   float64       float64     float64         float64   \n",
              "9             category   float64       float64     float64         float64   \n",
              "10            category   float64       float64     float64         float64   \n",
              "11            category   float64       float64     float64         float64   \n",
              "12            category   float64       float64     float64         float64   \n",
              "13            category   float64       float64     float64         float64   \n",
              "14            category   float64       float64     float64         float64   \n",
              "\n",
              "   total_rec_prncp total_rec_int total_rec_late_fee recoveries  \\\n",
              "0          float64       float64            float64    float64   \n",
              "1          float64       float64            float64    float64   \n",
              "2          float64       float64            float64    float64   \n",
              "3          float64       float64            float64    float64   \n",
              "4          float64       float64            float64    float64   \n",
              "5          float64       float64            float64    float64   \n",
              "6          float64       float64            float64    float64   \n",
              "7          float64       float64            float64    float64   \n",
              "8          float64       float64            float64    float64   \n",
              "9          float64       float64            float64    float64   \n",
              "10         float64       float64            float64    float64   \n",
              "11         float64       float64            float64    float64   \n",
              "12         float64       float64            float64    float64   \n",
              "13         float64       float64            float64    float64   \n",
              "14         float64       float64            float64    float64   \n",
              "\n",
              "   collection_recovery_fee    last_pymnt_d last_pymnt_amnt last_credit_pull_d  \\\n",
              "0                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "1                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "2                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "3                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "4                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "5                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "6                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "7                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "8                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "9                  float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "10                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "11                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "12                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "13                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "14                 float64  datetime64[ns]         float64     datetime64[ns]   \n",
              "\n",
              "   collections_12_mths_ex_med policy_code application_type acc_now_delinq  \\\n",
              "0                     float64       int64         category        float64   \n",
              "1                     float64       int64         category        float64   \n",
              "2                     float64       int64         category        float64   \n",
              "3                     float64       int64         category        float64   \n",
              "4                     float64       int64         category        float64   \n",
              "5                     float64       int64         category        float64   \n",
              "6                     float64       int64         category        float64   \n",
              "7                     float64       int64         category        float64   \n",
              "8                     float64       int64         category        float64   \n",
              "9                     float64       int64         category        float64   \n",
              "10                    float64       int64         category        float64   \n",
              "11                    float64       int64         category        float64   \n",
              "12                    float64       int64         category        float64   \n",
              "13                    float64       int64         category        float64   \n",
              "14                    float64       int64         category        float64   \n",
              "\n",
              "   chargeoff_within_12_mths delinq_amnt pub_rec_bankruptcies tax_liens  \n",
              "0                   float64     float64              float64   float64  \n",
              "1                   float64     float64              float64   float64  \n",
              "2                   float64     float64              float64   float64  \n",
              "3                   float64     float64              float64   float64  \n",
              "4                   float64     float64              float64   float64  \n",
              "5                   float64     float64              float64   float64  \n",
              "6                   float64     float64              float64   float64  \n",
              "7                   float64     float64              float64   float64  \n",
              "8                   float64     float64              float64   float64  \n",
              "9                   float64     float64              float64   float64  \n",
              "10                  float64     float64              float64   float64  \n",
              "11                  float64     float64              float64   float64  \n",
              "12                  float64     float64              float64   float64  \n",
              "13                  float64     float64              float64   float64  \n",
              "14                  float64     float64              float64   float64  "
            ]
          },
          "execution_count": 31,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HpNFkCDrS-8K"
      },
      "source": [
        "## Further reduce chunk size by using the apporpirate number of bits for integer or float type\n",
        "\n",
        "64 bit is the default size used to represent integer and float in DataFrame.  The chunk size can be further reduce if we know the min and max value of numeric columns across all chunks and use the proper number of bits to represent integer and float value.\n",
        "\n",
        "To determine the min and max of numeric value across all chunks, all chunks are read again to determine the min/max value and the result are stored in a dictionary as tuple.  "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y1rxVofrS-8L"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "totalNumberOfRows=0\n",
        "uniqueDicts=[{'col': col, 'non-numeric':False,'values':{}} for col in loan.columns]\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    loanChunk=dropTitleCol(loanChunk)\n",
        "    \n",
        "    totalNumberOfRows+=loanChunk.shape[0]\n",
        "    \n",
        "    for idx,col in enumerate(loanChunk):\n",
        "        if loanChunk[col].dtypes=='object':\n",
        "            uniqueDicts[idx]['non-numeric']=True\n",
        "            for cell in loanChunk[col]:\n",
        "                if cell in uniqueDicts[idx]['values']:\n",
        "                    uniqueDicts[idx]['values'][cell]+=1\n",
        "                else:\n",
        "                    uniqueDicts[idx]['values'][cell]=1\n",
        "\n",
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "loanChunkTypes=pd.DataFrame(columns=loanChunk.columns)\n",
        "\n",
        "\n",
        "numColMinMaxDict={}\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    loanChunk=dropTitleCol(loanChunk)\n",
        "    loanChunk=categorizeLoanChunk(loanChunk)\n",
        "    loanChunk=convertColToInteger(loanChunk,colToInteger)\n",
        "   \n",
        "    for numCol in loanChunk.select_dtypes(include='number'):\n",
        "        if numCol not in numColMinMaxDict.keys():\n",
        "            numColMinMaxDict[numCol]=(loanChunk[numCol].min(),loanChunk[numCol].max())\n",
        "        else:\n",
        "            colMin=loanChunk[numCol].min()\n",
        "            colMax=loanChunk[numCol].max()\n",
        "            if colMin<numColMinMaxDict[numCol][0]:\n",
        "                numColMinMaxDict[numCol]=(colMin,numColMinMaxDict[numCol][1])\n",
        "            if colMax>numColMinMaxDict[numCol][1]:\n",
        "                numColMinMaxDict[numCol]=(numColMinMaxDict[numCol][0],colMax)\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fK6eoohLS-8M",
        "outputId": "cc019bc1-5c89-4ba7-e988-d8ce39d69d49"
      },
      "source": [
        "numColMinMaxDict"
      ],
      "execution_count": null,
      "outputs": [
        {
          "data": {
            "text/plain": [
              "{'acc_now_delinq': (0.0, 1.0),\n",
              " 'annual_inc': (1896.0, 6000000.0),\n",
              " 'chargeoff_within_12_mths': (0.0, 0.0),\n",
              " 'collection_recovery_fee': (0.0, 7002.19),\n",
              " 'collections_12_mths_ex_med': (0.0, 0.0),\n",
              " 'delinq_2yrs': (0.0, 13.0),\n",
              " 'delinq_amnt': (0.0, 6053.0),\n",
              " 'dti': (0.0, 29.99),\n",
              " 'funded_amnt': (500, 35000),\n",
              " 'funded_amnt_inv': (0.0, 35000.0),\n",
              " 'id': (54734, 1077501),\n",
              " 'inq_last_6mths': (0.0, 33.0),\n",
              " 'installment': (15.67, 1305.19),\n",
              " 'int_rate': (5.42, 24.59),\n",
              " 'last_pymnt_amnt': (0.0, 36115.2),\n",
              " 'loan_amnt': (500, 35000),\n",
              " 'member_id': (70473, 1314167),\n",
              " 'open_acc': (1.0, 47.0),\n",
              " 'out_prncp': (0.0, 5794.29),\n",
              " 'out_prncp_inv': (0.0, 5794.29),\n",
              " 'policy_code': (1, 1),\n",
              " 'pub_rec': (0.0, 5.0),\n",
              " 'pub_rec_bankruptcies': (0.0, 2.0),\n",
              " 'recoveries': (0.0, 29623.35),\n",
              " 'revol_bal': (0, 1207359),\n",
              " 'revol_util': (0.0, 119.0),\n",
              " 'tax_liens': (0.0, 1.0),\n",
              " 'total_acc': (1.0, 90.0),\n",
              " 'total_pymnt': (0.0, 58563.6799293133),\n",
              " 'total_pymnt_inv': (0.0, 58563.68),\n",
              " 'total_rec_int': (0.0, 23611.1),\n",
              " 'total_rec_late_fee': (0.0, 209.000000006491),\n",
              " 'total_rec_prncp': (0.0, 35000.02)}"
            ]
          },
          "execution_count": 33,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7kRtSmqiS-8N"
      },
      "source": [
        "## The min/max dict shows there is no -ve value ...\n",
        "\n",
        "The min/max dictionary shows that there is no -ve value in all numeric columns.  So unsigned integer can be used for integer values, and there is no need to check for -ve minimun value in determine the no. of bits to store numeric values.\n",
        "\n",
        "Two procdure are created to size the integer (unsigned) columns and float columns using the np.iinfo() and np.finfo() to check the min and max value that can be represented with different numner of bits.  There two functions are then called to converted numeric columns using the apporpriate no. of bits that could hold the min/max of different numeric columns.\n",
        "\n",
        "The final size of each chunk is then displayed and the data type of each column for all chunks are also displayed for verification."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dB-6D01DS-8O"
      },
      "source": [
        "def sizeIntCol(loanChunk):\n",
        "    for col in loanChunk.select_dtypes('integer'):\n",
        "        if numColMinMaxDict[col][1]<np.iinfo(np.uint8).max:\n",
        "            loanChunk[col]=loanChunk[col].astype('uint8')\n",
        "        elif numColMinMaxDict[col][1]<np.iinfo(np.uint16).max:     \n",
        "            loanChunk[col]=loanChunk[col].astype('uint16')\n",
        "        elif numColMinMaxDict[col][1]<np.iinfo(np.uint32).max:     \n",
        "            loanChunk[col]=loanChunk[col].astype('uint32')\n",
        "        elif numColMinMaxDict[col][1]<np.iinfo(np.uint64).max:     \n",
        "            loanChunk[col]=loanChunk[col].astype('uint64')\n",
        "    \n",
        "    return loanChunk \n",
        "        "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cK9DYWyHS-8P"
      },
      "source": [
        "def sizeFloatCol(loanChunk):\n",
        "    for col in loanChunk.select_dtypes('float'):\n",
        "        if numColMinMaxDict[col][1]<np.finfo(np.float16).max:     \n",
        "            loanChunk[col]=loanChunk[col].astype('float16')\n",
        "        elif numColMinMaxDict[col][1]<np.finfo(np.float32).max:     \n",
        "            loanChunk[col]=loanChunk[col].astype('float32')\n",
        "        elif numColMinMaxDict[col][1]<np.finfo(np.float64).max:     \n",
        "            loanChunk[col]=loanChunk[col].astype('float64')\n",
        "    \n",
        "    return loanChunk \n",
        "        "
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EswhUdkCS-8R",
        "outputId": "d2d66248-2cbc-4285-9fc5-9be78f75298c"
      },
      "source": [
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "totalNumberOfRows=0\n",
        "uniqueDicts=[{'col': col, 'non-numeric':False,'values':{}} for col in loan.columns]\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    loanChunk=dropTitleCol(loanChunk)\n",
        "    \n",
        "    totalNumberOfRows+=loanChunk.shape[0]\n",
        "    \n",
        "    for idx,col in enumerate(loanChunk):\n",
        "        if loanChunk[col].dtypes=='object':\n",
        "            uniqueDicts[idx]['non-numeric']=True\n",
        "            for cell in loanChunk[col]:\n",
        "                if cell in uniqueDicts[idx]['values']:\n",
        "                    uniqueDicts[idx]['values'][cell]+=1\n",
        "                else:\n",
        "                    uniqueDicts[idx]['values'][cell]=1\n",
        "\n",
        "loanChunkIter = pd.read_csv(\"loans_2007.csv\", chunksize=3000,parse_dates=dateCols)\n",
        "\n",
        "loanChunkTypes=pd.DataFrame(columns=loanChunk.columns)\n",
        "\n",
        "for loanChunk in loanChunkIter:\n",
        "    loanChunk=removeProblematicRow(loanChunk)\n",
        "    loanChunk=cleanRateCol(loanChunk)\n",
        "    loanChunk=dropTitleCol(loanChunk)\n",
        "    loanChunk=categorizeLoanChunk(loanChunk)\n",
        "    loanChunk=convertColToInteger(loanChunk,colToInteger)\n",
        "    loacnChunk=sizeIntCol(loanChunk)\n",
        "    loacnChunk=sizeFloatCol(loanChunk)\n",
        "   \n",
        "   \n",
        "    loanChunkTypes.loc[len(loanChunkTypes)]=loanChunk.dtypes\n",
        "    print(loanChunk.memory_usage(deep=True).sum()/(1024*1024))\n",
        "    \n",
        "loanChunkTypes"
      ],
      "execution_count": null,
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "0.6524324417114258\n",
            "0.6490945816040039\n",
            "0.6510372161865234\n",
            "0.6513290405273438\n",
            "0.6495151519775391\n",
            "0.6511297225952148\n",
            "0.649439811706543\n",
            "0.6513757705688477\n",
            "0.6512508392333984\n",
            "0.6525535583496094\n",
            "0.6515703201293945\n",
            "0.6487932205200195\n",
            "0.6552047729492188\n",
            "0.6734914779663086\n",
            "0.20389747619628906\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>id</th>\n",
              "      <th>member_id</th>\n",
              "      <th>loan_amnt</th>\n",
              "      <th>funded_amnt</th>\n",
              "      <th>funded_amnt_inv</th>\n",
              "      <th>term</th>\n",
              "      <th>int_rate</th>\n",
              "      <th>installment</th>\n",
              "      <th>grade</th>\n",
              "      <th>sub_grade</th>\n",
              "      <th>emp_title</th>\n",
              "      <th>emp_length</th>\n",
              "      <th>home_ownership</th>\n",
              "      <th>annual_inc</th>\n",
              "      <th>verification_status</th>\n",
              "      <th>issue_d</th>\n",
              "      <th>loan_status</th>\n",
              "      <th>pymnt_plan</th>\n",
              "      <th>purpose</th>\n",
              "      <th>zip_code</th>\n",
              "      <th>addr_state</th>\n",
              "      <th>dti</th>\n",
              "      <th>delinq_2yrs</th>\n",
              "      <th>earliest_cr_line</th>\n",
              "      <th>inq_last_6mths</th>\n",
              "      <th>open_acc</th>\n",
              "      <th>pub_rec</th>\n",
              "      <th>revol_bal</th>\n",
              "      <th>revol_util</th>\n",
              "      <th>total_acc</th>\n",
              "      <th>initial_list_status</th>\n",
              "      <th>out_prncp</th>\n",
              "      <th>out_prncp_inv</th>\n",
              "      <th>total_pymnt</th>\n",
              "      <th>total_pymnt_inv</th>\n",
              "      <th>total_rec_prncp</th>\n",
              "      <th>total_rec_int</th>\n",
              "      <th>total_rec_late_fee</th>\n",
              "      <th>recoveries</th>\n",
              "      <th>collection_recovery_fee</th>\n",
              "      <th>last_pymnt_d</th>\n",
              "      <th>last_pymnt_amnt</th>\n",
              "      <th>last_credit_pull_d</th>\n",
              "      <th>collections_12_mths_ex_med</th>\n",
              "      <th>policy_code</th>\n",
              "      <th>application_type</th>\n",
              "      <th>acc_now_delinq</th>\n",
              "      <th>chargeoff_within_12_mths</th>\n",
              "      <th>delinq_amnt</th>\n",
              "      <th>pub_rec_bankruptcies</th>\n",
              "      <th>tax_liens</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>uint32</td>\n",
              "      <td>uint32</td>\n",
              "      <td>uint16</td>\n",
              "      <td>uint16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>object</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float32</td>\n",
              "      <td>category</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint32</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>datetime64[ns]</td>\n",
              "      <td>float16</td>\n",
              "      <td>uint8</td>\n",
              "      <td>category</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "      <td>float16</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        id member_id loan_amnt funded_amnt funded_amnt_inv      term int_rate  \\\n",
              "0   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "1   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "2   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "3   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "4   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "5   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "6   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "7   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "8   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "9   uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "10  uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "11  uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "12  uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "13  uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "14  uint32    uint32    uint16      uint16         float16  category  float16   \n",
              "\n",
              "   installment     grade sub_grade emp_title emp_length home_ownership  \\\n",
              "0      float16  category  category    object   category       category   \n",
              "1      float16  category  category    object   category       category   \n",
              "2      float16  category  category    object   category       category   \n",
              "3      float16  category  category    object   category       category   \n",
              "4      float16  category  category    object   category       category   \n",
              "5      float16  category  category    object   category       category   \n",
              "6      float16  category  category    object   category       category   \n",
              "7      float16  category  category    object   category       category   \n",
              "8      float16  category  category    object   category       category   \n",
              "9      float16  category  category    object   category       category   \n",
              "10     float16  category  category    object   category       category   \n",
              "11     float16  category  category    object   category       category   \n",
              "12     float16  category  category    object   category       category   \n",
              "13     float16  category  category    object   category       category   \n",
              "14     float16  category  category    object   category       category   \n",
              "\n",
              "   annual_inc verification_status         issue_d loan_status pymnt_plan  \\\n",
              "0     float32            category  datetime64[ns]    category   category   \n",
              "1     float32            category  datetime64[ns]    category   category   \n",
              "2     float32            category  datetime64[ns]    category   category   \n",
              "3     float32            category  datetime64[ns]    category   category   \n",
              "4     float32            category  datetime64[ns]    category   category   \n",
              "5     float32            category  datetime64[ns]    category   category   \n",
              "6     float32            category  datetime64[ns]    category   category   \n",
              "7     float32            category  datetime64[ns]    category   category   \n",
              "8     float32            category  datetime64[ns]    category   category   \n",
              "9     float32            category  datetime64[ns]    category   category   \n",
              "10    float32            category  datetime64[ns]    category   category   \n",
              "11    float32            category  datetime64[ns]    category   category   \n",
              "12    float32            category  datetime64[ns]    category   category   \n",
              "13    float32            category  datetime64[ns]    category   category   \n",
              "14    float32            category  datetime64[ns]    category   category   \n",
              "\n",
              "     purpose  zip_code addr_state      dti delinq_2yrs earliest_cr_line  \\\n",
              "0   category  category   category  float16     float16   datetime64[ns]   \n",
              "1   category  category   category  float16     float16   datetime64[ns]   \n",
              "2   category  category   category  float16     float16   datetime64[ns]   \n",
              "3   category  category   category  float16     float16   datetime64[ns]   \n",
              "4   category  category   category  float16     float16   datetime64[ns]   \n",
              "5   category  category   category  float16     float16   datetime64[ns]   \n",
              "6   category  category   category  float16     float16   datetime64[ns]   \n",
              "7   category  category   category  float16     float16   datetime64[ns]   \n",
              "8   category  category   category  float16     float16   datetime64[ns]   \n",
              "9   category  category   category  float16     float16   datetime64[ns]   \n",
              "10  category  category   category  float16     float16   datetime64[ns]   \n",
              "11  category  category   category  float16     float16   datetime64[ns]   \n",
              "12  category  category   category  float16     float16   datetime64[ns]   \n",
              "13  category  category   category  float16     float16   datetime64[ns]   \n",
              "14  category  category   category  float16     float16   datetime64[ns]   \n",
              "\n",
              "   inq_last_6mths open_acc  pub_rec revol_bal revol_util total_acc  \\\n",
              "0         float16  float16  float16    uint32    float16   float16   \n",
              "1         float16  float16  float16    uint32    float16   float16   \n",
              "2         float16  float16  float16    uint32    float16   float16   \n",
              "3         float16  float16  float16    uint32    float16   float16   \n",
              "4         float16  float16  float16    uint32    float16   float16   \n",
              "5         float16  float16  float16    uint32    float16   float16   \n",
              "6         float16  float16  float16    uint32    float16   float16   \n",
              "7         float16  float16  float16    uint32    float16   float16   \n",
              "8         float16  float16  float16    uint32    float16   float16   \n",
              "9         float16  float16  float16    uint32    float16   float16   \n",
              "10        float16  float16  float16    uint32    float16   float16   \n",
              "11        float16  float16  float16    uint32    float16   float16   \n",
              "12        float16  float16  float16    uint32    float16   float16   \n",
              "13        float16  float16  float16    uint32    float16   float16   \n",
              "14        float16  float16  float16    uint32    float16   float16   \n",
              "\n",
              "   initial_list_status out_prncp out_prncp_inv total_pymnt total_pymnt_inv  \\\n",
              "0             category   float16       float16     float16         float16   \n",
              "1             category   float16       float16     float16         float16   \n",
              "2             category   float16       float16     float16         float16   \n",
              "3             category   float16       float16     float16         float16   \n",
              "4             category   float16       float16     float16         float16   \n",
              "5             category   float16       float16     float16         float16   \n",
              "6             category   float16       float16     float16         float16   \n",
              "7             category   float16       float16     float16         float16   \n",
              "8             category   float16       float16     float16         float16   \n",
              "9             category   float16       float16     float16         float16   \n",
              "10            category   float16       float16     float16         float16   \n",
              "11            category   float16       float16     float16         float16   \n",
              "12            category   float16       float16     float16         float16   \n",
              "13            category   float16       float16     float16         float16   \n",
              "14            category   float16       float16     float16         float16   \n",
              "\n",
              "   total_rec_prncp total_rec_int total_rec_late_fee recoveries  \\\n",
              "0          float16       float16            float16    float16   \n",
              "1          float16       float16            float16    float16   \n",
              "2          float16       float16            float16    float16   \n",
              "3          float16       float16            float16    float16   \n",
              "4          float16       float16            float16    float16   \n",
              "5          float16       float16            float16    float16   \n",
              "6          float16       float16            float16    float16   \n",
              "7          float16       float16            float16    float16   \n",
              "8          float16       float16            float16    float16   \n",
              "9          float16       float16            float16    float16   \n",
              "10         float16       float16            float16    float16   \n",
              "11         float16       float16            float16    float16   \n",
              "12         float16       float16            float16    float16   \n",
              "13         float16       float16            float16    float16   \n",
              "14         float16       float16            float16    float16   \n",
              "\n",
              "   collection_recovery_fee    last_pymnt_d last_pymnt_amnt last_credit_pull_d  \\\n",
              "0                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "1                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "2                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "3                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "4                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "5                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "6                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "7                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "8                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "9                  float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "10                 float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "11                 float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "12                 float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "13                 float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "14                 float16  datetime64[ns]         float16     datetime64[ns]   \n",
              "\n",
              "   collections_12_mths_ex_med policy_code application_type acc_now_delinq  \\\n",
              "0                     float16       uint8         category        float16   \n",
              "1                     float16       uint8         category        float16   \n",
              "2                     float16       uint8         category        float16   \n",
              "3                     float16       uint8         category        float16   \n",
              "4                     float16       uint8         category        float16   \n",
              "5                     float16       uint8         category        float16   \n",
              "6                     float16       uint8         category        float16   \n",
              "7                     float16       uint8         category        float16   \n",
              "8                     float16       uint8         category        float16   \n",
              "9                     float16       uint8         category        float16   \n",
              "10                    float16       uint8         category        float16   \n",
              "11                    float16       uint8         category        float16   \n",
              "12                    float16       uint8         category        float16   \n",
              "13                    float16       uint8         category        float16   \n",
              "14                    float16       uint8         category        float16   \n",
              "\n",
              "   chargeoff_within_12_mths delinq_amnt pub_rec_bankruptcies tax_liens  \n",
              "0                   float16     float16              float16   float16  \n",
              "1                   float16     float16              float16   float16  \n",
              "2                   float16     float16              float16   float16  \n",
              "3                   float16     float16              float16   float16  \n",
              "4                   float16     float16              float16   float16  \n",
              "5                   float16     float16              float16   float16  \n",
              "6                   float16     float16              float16   float16  \n",
              "7                   float16     float16              float16   float16  \n",
              "8                   float16     float16              float16   float16  \n",
              "9                   float16     float16              float16   float16  \n",
              "10                  float16     float16              float16   float16  \n",
              "11                  float16     float16              float16   float16  \n",
              "12                  float16     float16              float16   float16  \n",
              "13                  float16     float16              float16   float16  \n",
              "14                  float16     float16              float16   float16  "
            ]
          },
          "execution_count": 36,
          "metadata": {},
          "output_type": "execute_result"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "collapsed": true,
        "id": "QpvaqzfhS-8S"
      },
      "source": [
        "## Processing Result\n",
        "\n",
        "We have effectively reduce the memory foot print of each chunk from 4.6M to 0.6M by\n",
        "- read in date column as date type using date parse capability provided by pd.read_csv() function\n",
        "- convert percentage value to float by removing the % character in the CSV\n",
        "- categorize string columns for columns that the no. of unique value is less than half (50%) of total record in the loan CSV\n",
        "- drop the free text column title\n",
        "- convert numeric column to integer and float accordingly and store them in the apporpirate format in minimun no. of bits that could whole the min/max value"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5o9UjiPYS-8T"
      },
      "source": [
        "## Further processing ...\n",
        "\n",
        "In exploring the data during the exerice, the below can further be processed ...\n",
        "- remove 'months' from the terms column and categorize the column as numeric instead of text column\n",
        "- similary remove 'years' in column emp_length and categorize the column as numeric instead opf text column  \n",
        "- remove columns that are single valued across all chunk, as single valued column cannot be analyzed\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "collapsed": true,
        "id": "tQOCJT47S-8U"
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}