01.1_stanwifi-evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aa9c40c9-05f1-4566-91ed-b8fe9a7cf2f3",
   "metadata": {},
   "source": [
    "# Evaluation on public dataset: StanWiFi\n",
    "\n",
    "This notebook uses the StanWiFi dataset to evaluate a preprocessing pipeline and a CNN-based classification model. The StanWiFi dataset was collected in\n",
    "\n",
    "> S. Yousefi, H. Narui, S. Dayal, S. Ermon and S. Valaee, \"A Survey on Behavior Recognition Using WiFi Channel State Information,\" in IEEE Communications Magazine, vol. 55, no. 10, pp. 98-104, Oct. 2017, doi: 10.1109/MCOM.2017.1700082.\n",
    "\n",
    "The results obtained from the evaluation are stored in `02_RESULTS/01_STANWIFI/01_MODEL-REPORTS/reports.json` and can be loaded in the [Section 1.6](#summary) of this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "135496ab-a805-4b10-a929-bd5001e8556e",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9e471fc3-fe72-4420-9ec3-f8133892e3f0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "from tensorflow import keras\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "from functions.filters import dbscan_filtering, wavelet_filtering\n",
    "from functions.json import load_json, save_json\n",
    "from functions.ml import clear_backend_and_seeds, cross_validation\n",
    "from functions.report_metrics import metrics_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "974983ae-f305-4df5-876d-3bc0cf73d2d5",
   "metadata": {},
   "source": [
    "## Constants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e7d4d49c-7aea-48bb-a16f-afd3afd3b667",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "RESULTS = '02_RESULTS'\n",
    "\n",
    "STANWIFI_REPORTS = os.path.join(RESULTS, '01_STANWIFI', '01_MODEL-REPORTS', 'reports.json')\n",
    "STANWIFI_LABELS = ['LIE DOWN', 'FALL', 'WALK', 'RUN', 'SITDOWN', 'STANDUP']\n",
    "\n",
    "FOLDS = 10\n",
    "BATCH_SIZE = 256\n",
    "EPOCHS = 30"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc38b091-be6a-442d-8956-b97478c7ee38",
   "metadata": {},
   "source": [
    "## Data loading\n",
    "\n",
    "Follow the instructions provided in [`01_DATA/01_STANWIFI/README.md`](./01_DATA/01_STANWIFI/README.md) and then execute the following cell to load the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1e1e974-bcb2-4689-8456-a30ea0199e4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from cross_vali_input_data import csv_import\n",
    "\n",
    "x_bed, x_fall, x_pickup, x_run, x_sitdown, x_standup, x_walk, \\\n",
    "y_bed, y_fall, y_pickup, y_run, y_sitdown, y_standup, y_walk = csv_import()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c28ff59-42d6-4f45-9cdd-0187f8c94bf5",
   "metadata": {},
   "source": [
    "## Data processing\n",
    "\n",
    "- Apply DBSCAN filtering + DWT filtering\n",
    "- Remove \"pickup\" activity from one-hot vectors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9059ee5d-56e4-48f6-9540-e6d00fbf7b99",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_bed_proc = np.apply_along_axis(lambda x: wavelet_filtering(dbscan_filtering(x)),1, x_bed)\n",
    "x_fall_proc = np.apply_along_axis(lambda x: wavelet_filtering(dbscan_filtering(x)),1, x_fall)\n",
    "x_run_proc = np.apply_along_axis(lambda x: wavelet_filtering(dbscan_filtering(x)),1, x_run)\n",
    "x_sitdown_proc = np.apply_along_axis(lambda x: wavelet_filtering(dbscan_filtering(x)),1, x_sitdown)\n",
    "x_standup_proc = np.apply_along_axis(lambda x: wavelet_filtering(dbscan_filtering(x)),1, x_standup)\n",
    "x_walk_proc = np.apply_along_axis(lambda x: wavelet_filtering(dbscan_filtering(x)),1, x_walk)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6be4de3-1bc2-44ef-809c-82ae8b95187a",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_bed_proc = np.delete(y_bed, [0,4], axis=1)\n",
    "y_fall_proc = np.delete(y_fall, [0,4], axis=1)\n",
    "y_run_proc = np.delete(y_run, [0,4], axis=1)\n",
    "y_sitdown_proc = np.delete(y_sitdown, [0,4], axis=1)\n",
    "y_standup_proc = np.delete(y_standup, [0,4], axis=1)\n",
    "y_walk_proc = np.delete(y_walk, [0,4], axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88bd1f52-284e-464c-abc7-a5a6c7e35ab6",
   "metadata": {},
   "outputs": [],
   "source": [
    "x = np.vstack((x_bed_proc, x_fall_proc, x_run_proc, x_sitdown_proc, x_standup_proc, x_walk_proc))\n",
    "x = np.transpose(x, axes=(0,2,1))\n",
    "y = np.vstack((y_bed_proc, y_fall_proc, y_run_proc, y_sitdown_proc, y_standup_proc, y_walk_proc))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "747c5063-1bd5-43eb-90ec-fea2f64cbd2f",
   "metadata": {},
   "source": [
    "## 10-fold cross validation\n",
    "\n",
    "> **WARNING**: Its execution can last several hours. You can instead load the results obtained by the authors in [Section 1.6](#summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0b19ce8e-2b41-4d12-8620-563ffa17fcfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "def build_model():\n",
    "    clear_backend_and_seeds()\n",
    "    \n",
    "    model = keras.Sequential([\n",
    "        layers.Conv2D(filters=8, kernel_size=(3,10), input_shape=(90, 500, 1)),\n",
    "        layers.BatchNormalization(),\n",
    "        layers.Activation('relu'),\n",
    "        layers.MaxPooling2D(),\n",
    "        \n",
    "        layers.Conv2D(filters=8, kernel_size=(3,10)),\n",
    "        layers.BatchNormalization(),\n",
    "        layers.Activation('relu'),\n",
    "        layers.MaxPooling2D(),\n",
    "        \n",
    "        layers.Flatten(),\n",
    "        \n",
    "        layers.Dense(512, activation='relu'),\n",
    "        layers.Dropout(0.2),\n",
    "        layers.Dense(512, activation='relu'),\n",
    "        layers.Dense(512, activation='relu'),\n",
    "        layers.Dense(6, activation='softmax')\n",
    "    ])\n",
    "\n",
    "    model.compile(loss='categorical_crossentropy', optimizer=keras.optimizers.Adam(learning_rate=0.0001), metrics=['accuracy'])\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d83c4fca-0382-49d6-be0e-482d8a5ee9d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "reports = cross_validation(build_model, x, y, folds=FOLDS, batch_size=BATCH_SIZE, epochs=EPOCHS, labels=STANWIFI_LABELS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bc170678-34bc-4f70-8ecb-b2742d71df70",
   "metadata": {},
   "outputs": [],
   "source": [
    "save_json(reports, STANWIFI_REPORTS)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad5d890e-35f4-4fe6-9965-d5fb90f9bea6",
   "metadata": {},
   "source": [
    "<a id='summary'></a>\n",
    "## Summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d24542de-28e5-40dc-b7c3-ebf8ef2774e5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "reports = load_json(STANWIFI_REPORTS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b6cbfe29-02cd-4346-bc6f-cdbeab65a7cc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>accuracy</th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>f1-score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>StanWiFi CV</th>\n",
       "      <td>0.967177</td>\n",
       "      <td>0.968634</td>\n",
       "      <td>0.967177</td>\n",
       "      <td>0.966725</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "             accuracy  precision    recall  f1-score\n",
       "StanWiFi CV  0.967177   0.968634  0.967177  0.966725"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics_summary([reports], ['StanWiFi CV'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e37189a-b2a5-461b-8ecb-85fcbe563e70",
   "metadata": {},
   "source": [
    "The information presented in the above table corresponds to the one included in the **Table III** (StanWiFi > This work) of the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66bfd988-7001-475b-aa9a-b3b27f7440e0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "tf310-gpu",
   "language": "python",
   "name": "tf310-gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "toc-autonumbering": true
 },
 "nbformat": 4,
 "nbformat_minor": 5
}