Lec18b.MRsemantics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MR Semantics\n",
    "\n",
    "Let's start with some definitions:\n",
    "\n",
    "#### Shuffle\n",
    "\n",
    "This is the **routing process** of mapper outputs to reducer inputs. It routes by keys (key based `groupby`) that gives each reducer a iterator on values. \n",
    "\n",
    "<img src=\"images/hadoopsem.png\" width=512 />\n",
    "\n",
    "#### Partition\n",
    "\n",
    "* Partition is the output file of a reducer _process_ (not a single reducer).\n",
    "  * Contains many reducer keys\n",
    " \n",
    "#### Combiner\n",
    "\n",
    "This image is linked from https://data-flair.training/blogs/hadoop-combiner-tutorial/.  Please refer to their page.\n",
    "\n",
    "<img src=\"images/combiner.webp\" width=512 />\n",
    "\n",
    "**Pre-shuffle aggregation**: Combiner is a function that runs on the outputs of the mapper (local data) before the shuffle.\n",
    "\n",
    "* Combiner executes on the mappers $ \\langle key,value \\rangle$ output while in memory at the mapper\n",
    "  * It is possible to write unique combiner and reduce classes\n",
    "  * It is common to use the reducer as a combiner\n",
    "  * Combiner must have algebraic propertics, i.e. `reduce(combine(A),combine(B)) == reduce(A,B)` \n",
    "  * Traditional reduce operators (aggregates and extrema) work in combiners\n",
    "* Combiner in WordCount:\n",
    "  * compute sum output by mapper for each key and send a single aggregated value to reducers\n",
    "* Combiner for Maximum: \n",
    "  * compute maximum value for each key.  \n",
    "  * Reducer computes a maximum of maxima.\n",
    "* Does no work for uniqueness testing etc.\n",
    "\n",
    "\n",
    "### The Map/Reduce Sorting Guarantee\n",
    "\n",
    "* Map: extracts a sorting key from the value\n",
    "$$\n",
    " \\langle key, value \\rangle -> \\langle sort\\_key, output\\_value \\rangle\n",
    "$$\n",
    "* Shuffle does not sort strictly:\n",
    "  * it route's to reducer based on sort key.\n",
    "  * typically hashing maps key to reducer\n",
    "  * Keys are sorted as they are presented to the reducer\n",
    "\n",
    "Here is the guarantee:\n",
    "\n",
    "> We guarantee that within a given partition, the intermediate key/value pairs are processed in increasing key order. This ordering guarantee makes it easy to generate\n",
    "a sorted output file per partition, which is useful when\n",
    "the output file format needs to support efficient random\n",
    "access lookups by key, or users of the output find it convenient to have the data sorted._\n",
    "\n",
    "The ordering **guarantees sort within partitions**\n",
    "* To sort completely:\n",
    "  * All output to a single partition (use one reducer) BUT reduce parallelism\n",
    "  * Customize the shuffle function (quite complex and can introduce skew)\n",
    "  * The default shuffle uses hashing (for load balance)\n",
    "\n",
    "* The Google paper optimizes sort by customizing shuffle so that partitions are ordered, not randomized\n",
    "  * Run a M/R job to learn the key distribution\n",
    "  * Specify a shuffle based on evenly paritioning the key distribution\n",
    "  * This is also how Hadoop!’s sort record worked"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}