Lec18x.FoF.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "04e8837a-5fac-4fd1-8d5d-1cbe63410c88",
   "metadata": {},
   "source": [
    "## Friends of Friends\n",
    "\n",
    "This is a second example program in Hadoop! Map/Reduce. The intention of this class is to reason more carefully about how combiners works and the Map/Reduce sorting guarantee. We used to do this as a homework assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de524d19-e5c7-44d4-bf3d-5df7f1ecbb2d",
   "metadata": {},
   "source": [
    "### In Class Exercise: Three’s Company\n",
    "\n",
    "This assignment is a social network version of the triangle counting problem, which has become an archetypal problem in graph analysis. Your program will output all triangles (which is different than a count of triangles) in a single map/reduce program.\n",
    "\n",
    "Identify threesomes of users that are mutual friends in a social network, i.e. 100 is friends with 200, 200 friends with 300 , and 300 friends with 100. The output should enumerate the mutual friends for each user and avoid duplicate entries, i.e. the trio of users 100, 200, and 300 will contribute output:\n",
    "\n",
    "> 100 200 300  \n",
    "> 200 100 300   \n",
    "> 300 100 200\n",
    "\n",
    "_NOTE:_ There is an integer sorting on the last 2 elements of the output line. As, such we would never get the output 100 300 200.\n",
    "\n",
    "This needs to be implemented in the Map/Reduce paradigm, using no auxiliary data structures and no shared data. The only files used are the inputs to the map stage. The solution will demonstrate one potential parallelism tradeoff, it will expand the data sending more data than is intuitively necessary over the network and examining the expanded data in the reducers.\n",
    "\n",
    "The input to the Map/Reduce program is set of friend lists. Each file in the input contains the id of the user followed by a list of her/his friends. All identifiers are integer values separated by spaces and all friends in the list appear on a single line. For example, the file:\n",
    "\n",
    "> 117 2149 84 57 6048\n",
    "\n",
    "is the list of user 117’s friend, consisting of users 2149, 84, 57, and 6048. You may assume that the input has two properties:\n",
    "  - Symmetry: If 100 is a friend of 200 then 200 is a friend of 100\n",
    "  - No Duplicates: Each friends appears in a list at most once.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d395bee-edd4-4adb-9e73-3e4e3ea7eaec",
   "metadata": {},
   "source": [
    "### Activity\n",
    "\n",
    "Fill in the TODOs in `hadoop/fof/FoF.java` to make a working version. You can build this with:\n",
    "```\n",
    "~/hadoop-3.3.4/bin/hadoop com.sun.tools.javac.Main FoF.java\n",
    "jar cf fof.jar FoF*.class\n",
    "~/hadoop-3.3.4/bin/hadoop jar fof.jar FoF ../../data/simple.input /tmp/output\n",
    "```\n",
    "There is sample input in\n",
    "* `data/simple_input` for debugging\n",
    "* `data/fof.input` for a larger dataset\n",
    "\n",
    "This assumes that you have installed haddoop3.3.4 in ~. You must also have a valid `JAVA_PATH` and \n",
    "```\n",
    "export HADOOP_CLASSPATH=${JAVA_HOME}/lib/tools.jar\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9166c462-9cc4-48d0-a5f0-ab171a0972d4",
   "metadata": {},
   "source": [
    "### Questions:\n",
    "\n",
    "1.  What is the potential parallelism? How many mappers does you implementation allow for? Reducers?\n",
    "\n",
    "2. Why must our implementation leave the combiner class undefined?\n",
    "\n",
    "3. Why is it necessary to have all three elements in the reduce key, i.e. why `(100,200,300),0` versus `(100), 200, 300` or `(100, 200), 300`?  The parens indicate the key in this case.\n",
    "\n",
    "4. Is the output is useful order:\n",
    "    1. If there is only one partition?\n",
    "    2. If there are many partitions?\n",
    "    3. How would you reorder the output to make it useful?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4f9ae5a-b23f-4929-a16c-5f0fcfe922cb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}