SuaveDispatch-Beautified.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Welcome to the SuAVE Jupyter Notebook Server\n",
    "\n",
    "This is the Jupyter Notebook Dispatcher module of the SuAVE platform. This environment allows you to execute scripts to process survey data in SuAVE and update the datasets for visual analysis. In a typical case, the scripts will generate one or more additional variables which will be added to a dataset. \n",
    "\n",
    "You can execute cells in sequence, by clicking 'Run' on the toolbar above, or pressing 'Shift-Enter'. From  this module (\"dispatcher\") you can launch other notebooks to perform computations, image processing,  modeling and statistical tasks.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Accepting survey parameters from SuAVE \n",
    "\n",
    "This cell will retrieve survey parameters from the URL. Execute it to continue "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "function getQueryStringValue (key)\n",
       "{  \n",
       "    return unescape(window.location.search.replace(new RegExp(\"^(?:.*[&\\\\?]\" + escape(key).replace(/[\\.\\+\\*]/g, \"\\\\$&\") + \"(?:\\\\=([^&]*))?)?.*$\", \"i\"), \"$1\"));\n",
       "}\n",
       "IPython.notebook.kernel.execute(\"survey_url='\".concat(getQueryStringValue(\"surveyurl\")).concat(\"'\"));\n",
       "IPython.notebook.kernel.execute(\"user='\".concat(getQueryStringValue(\"user\")).concat(\"'\"));\n",
       "IPython.notebook.kernel.execute(\"csv_file='\".concat(getQueryStringValue(\"csv\")).concat(\"'\")); \n",
       "IPython.notebook.kernel.execute(\"dzc_file='\".concat(getQueryStringValue(\"dzc\")).concat(\"'\")); \n",
       "IPython.notebook.kernel.execute(\"params='\".concat(getQueryStringValue(\"params\")).concat(\"'\")); \n",
       "IPython.notebook.kernel.execute(\"active_object='\".concat(getQueryStringValue(\"activeobject\")).concat(\"'\")); \n",
       "IPython.notebook.kernel.execute(\"full_notebook_url='\" + window.location + \"'\"); "
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%javascript\n",
    "function getQueryStringValue (key)\n",
    "{  \n",
    "    return unescape(window.location.search.replace(new RegExp(\"^(?:.*[&\\\\?]\" + escape(key).replace(/[\\.\\+\\*]/g, \"\\\\$&\") + \"(?:\\\\=([^&]*))?)?.*$\", \"i\"), \"$1\"));\n",
    "}\n",
    "IPython.notebook.kernel.execute(\"survey_url='\".concat(getQueryStringValue(\"surveyurl\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"user='\".concat(getQueryStringValue(\"user\")).concat(\"'\"));\n",
    "IPython.notebook.kernel.execute(\"csv_file='\".concat(getQueryStringValue(\"csv\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"dzc_file='\".concat(getQueryStringValue(\"dzc\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"params='\".concat(getQueryStringValue(\"params\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"active_object='\".concat(getQueryStringValue(\"activeobject\")).concat(\"'\")); \n",
    "IPython.notebook.kernel.execute(\"full_notebook_url='\" + window.location + \"'\"); \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Check if the passed parameters are correct ###"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Base Survey URL:  https://suave-dev.sdsc.edu/main/file=public_S_E_Articles_Timeline_clone_.csv\n",
      "User ID:  undefined\n",
      "Additional Parameters:  none\n",
      "Data File:  public_S_E_Articles_Timeline_clone_.csv\n",
      "Image Tile Collection:  https://maxim.ucsd.edu/dzgen/uploads/275b305e3f83cad50623f9c6946abb4b/content.dzc\n",
      "Active Object:  null\n",
      "Jupyter Hub URL:  http://suave-jupyterhub.com/user/zeppelin-v/notebooks\n"
     ]
    }
   ],
   "source": [
    "# Check if the parameters are correct\n",
    "import ntpath\n",
    "url_partitioned = full_notebook_url.partition('/SuaveDispatch')\n",
    "base_url = url_partitioned[0];\n",
    "if dzc_file == \"undefined\":\n",
    "    dzc_file = \"\"\n",
    "print(\"Base Survey URL: \", survey_url)\n",
    "print(\"User ID: \", user)\n",
    "print(\"Additional Parameters: \", params)\n",
    "print(\"Data File: \", csv_file)\n",
    "print(\"Image Tile Collection: \", dzc_file)\n",
    "print(\"Active Object: \", active_object)\n",
    "print(\"Jupyter Hub URL: \", base_url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Retrieve the survey file for processing\n",
    "\n",
    "As before, Shift-Enter to run this cell and continue to the next one. This merely prepares the data for subsequent processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "LocationValueError",
     "evalue": "No host specified.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mLocationValueError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-cd3249cbbc35>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mhttp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0murllib3\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mPoolManager\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m \u001b[0mr\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mhttp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'GET'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcsv_url\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mpreload_content\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m \u001b[0mpath\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"temp_csvs/\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mcsv_file\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/urllib3/request.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, fields, headers, **urlopen_kw)\u001b[0m\n\u001b[1;32m     64\u001b[0m             return self.request_encode_url(method, url, fields=fields,\n\u001b[1;32m     65\u001b[0m                                            \u001b[0mheaders\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mheaders\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 66\u001b[0;31m                                            **urlopen_kw)\n\u001b[0m\u001b[1;32m     67\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m             return self.request_encode_body(method, url, fields=fields,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/urllib3/request.py\u001b[0m in \u001b[0;36mrequest_encode_url\u001b[0;34m(self, method, url, fields, headers, **urlopen_kw)\u001b[0m\n\u001b[1;32m     85\u001b[0m             \u001b[0murl\u001b[0m \u001b[0;34m+=\u001b[0m \u001b[0;34m'?'\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0murlencode\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfields\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     86\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 87\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murlopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mextra_kw\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     88\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     89\u001b[0m     def request_encode_body(self, method, url, fields=None, headers=None,\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/urllib3/poolmanager.py\u001b[0m in \u001b[0;36murlopen\u001b[0;34m(self, method, url, redirect, **kw)\u001b[0m\n\u001b[1;32m    309\u001b[0m         \"\"\"\n\u001b[1;32m    310\u001b[0m         \u001b[0mu\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparse_url\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 311\u001b[0;31m         \u001b[0mconn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnection_from_host\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mhost\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mport\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mport\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscheme\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mscheme\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    312\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    313\u001b[0m         \u001b[0mkw\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'assert_same_host'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/usr/local/lib/python3.5/dist-packages/urllib3/poolmanager.py\u001b[0m in \u001b[0;36mconnection_from_host\u001b[0;34m(self, host, port, scheme, pool_kwargs)\u001b[0m\n\u001b[1;32m    216\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    217\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mhost\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 218\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mLocationValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"No host specified.\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    219\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    220\u001b[0m         \u001b[0mrequest_context\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_merge_pool_kwargs\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpool_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mLocationValueError\u001b[0m: No host specified."
     ]
    }
   ],
   "source": [
    "import urllib3\n",
    "\n",
    "csv_url = survey_url.split(\"main\")[0] + \"/surveys/\" + csv_file\n",
    "\n",
    "http = urllib3.PoolManager()\n",
    "r = http.request('GET', csv_url, preload_content=False)\n",
    "\n",
    "path = \"temp_csvs/\" + csv_file \n",
    "with open(path, 'wb') as out:\n",
    "    while True:\n",
    "        data = r.read(1024)\n",
    "        if not data:\n",
    "            break\n",
    "        out.write(data)\n",
    "\n",
    "r.release_conn()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 4. Now, select a notebook to do some work\n",
    "\n",
    "Run the following cell to see a drop-down list of available notebooks. Select a notebook, then continue to the next cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "90cc87c43eb24150a303573f30e8dd10",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "interactive(children=(Dropdown(description='notebooks_menu', options=('3. Classify Images', '1. Arithmetic ope…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "from __future__ import print_function\n",
    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
    "import ipywidgets as widgets\n",
    "\n",
    "nb_menu = {\n",
    "    '1. Arithmetic operations': 'arithmetic/SuaveArithmetic.ipynb',\n",
    "    '4. Color Characteristics': 'colors/ColorCharacteristics.ipynb',\n",
    "    '3. Classify Images': 'classify/ImageClassify.ipynb',\n",
    "    '2. Predictive models': 'predict/PredictiveModel.ipynb',\n",
    "    '5. SVM models': 'svm/SVMPredictiveModel.ipynb',\n",
    "}\n",
    "\n",
    "def f(notebooks_menu):\n",
    "    return notebooks_menu\n",
    "# out = interact(f, notebooks_menu=nb_menu);\n",
    "out = interact(f, notebooks_menu=nb_menu.keys());\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Open the selected notebook and pass survey parameters to it\n",
    "\n",
    "Once the URL is for the next notebook is constructed, click that URL to open it\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'base_url' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-5-c977ce1cb4d4>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mchosen_nb_name\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnb_menu\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwidget\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mwebbrowser\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0murl1\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m'{base_url}/operations/{nb_name}?'\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'surveyurl='\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0msurvey_url\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'&'\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'user='\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0muser\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'&'\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'csv='\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mcsv_file\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'&'\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'dzc='\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mdzc_file\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m\"&\"\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;34m'activeobject='\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0mactive_object\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbase_url\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mbase_url\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnb_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mchosen_nb_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      4\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Click the URL to open the selected notebook :'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'base_url' is not defined"
     ]
    }
   ],
   "source": [
    "chosen_nb_name = nb_menu[out.widget.result]\n",
    "import webbrowser\n",
    "url1 = ('{base_url}/operations/{nb_name}?'+'surveyurl='+survey_url+'&'+'user='+user+'&'+'csv='+csv_file+'&'+'dzc='+dzc_file+\"&\"+'activeobject='+active_object).format(base_url=base_url, nb_name=chosen_nb_name)\n",
    "print('Click the URL to open the selected notebook :', url1)\n",
    "\n",
    "#webbrowser.open(url1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}