Lec19.spark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Spark: Cloud Programming with Memory\n",
    "\n",
    "Lecture derived from: Zaharia et al. Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing. USENIX NSDI, 2012.\n",
    "\n",
    "Spark in a nutshell: Map/Reduce style data paradigm with restrictive functions and **memory to cache data for reuse**\n",
    "\n",
    "* Map/Reduce style programming\n",
    "  * Data-parallel, batch, restrictive model, functional\n",
    "  * Abstractions to leverage distributed memory\n",
    "* New interfaces to in-memory computations\n",
    "  * Fault-tolerant\n",
    "  * Lazy materialization (pipelined evaluation)\n",
    "* **Good support for iterative computations** on in-memory data sets leads to good performance\n",
    "  * 20x over Map/Reduce\n",
    "  * No writing data to file system, loading data from file system, during iteration\n",
    "\n",
    "  \n",
    "__Note__: Update you virtual environment with `conda install -c conda-forge pyspark`.\n",
    "  \n",
    "### RDD: Resilient Distributed Dataset\n",
    "\n",
    "This is the central abstraction of Spark.\n",
    "* **Read-only** partitioned collection of records\n",
    "* Created from:\n",
    "  * Data in stable storage\n",
    "  * Transformations on other RDDs\n",
    "* Unit of parallelism in a data decomposition:\n",
    "  * Automatic parallelization of transformation, such as map, reduce, filter, etc.\n",
    "* RDDs are not data:\n",
    "  * Not materialized.  They are an abstraction.\n",
    "  * Defined by lineage. Set of transformations on a original data set.\n",
    "  \n",
    "A first example:\n",
    "\n",
    "```scala\n",
    "lines = spark.textFile(\"hdfs://...\")\n",
    "errors = lines.filter(_.startsWith(\"ERROR\"))\n",
    "errors.persist()\n",
    "\n",
    "// Return the time fields of errors mentioning \n",
    "// HDFS as an array (assuming time is field #3 \n",
    "// in a tab separated format\n",
    "errors.filter(_contains(\"HDFS\"))\n",
    "      .map(_.split('\\t')(3))\n",
    "      .collect()\n",
    "```\n",
    "\n",
    "* RDDS in this computation\n",
    "  * `lines` is RDD backed by HDFS.\n",
    "  * `errors` is derived from filter.\n",
    "  * next two are implicit (not named variables)\n",
    "* `persist` indicates to store something in memory for reuse. Best-effort basis optimization\n",
    "* `collect` materializes computation to HDFS\n",
    "\n",
    "Associate with each Spark computation is a lineago of RDDS.\n",
    "\n",
    "<img src=\"./images/spark_lineage.png\" width=384 />\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A PySpark Example\n",
    "\n",
    "Trivial example: Spark program with NO data\n",
    "\n",
    "Monte Carlo approximation of $\\Pi$ based on determining whether points are inside or outside a radius 1 circle of area $\\Pi r^2$ inside a square of area $4r^2$.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pyspark import SparkContext\n",
    "sc = SparkContext(\"local\", \"App Name\",)\n",
    "\n",
    "import random\n",
    "num_samples = 10000000\n",
    "def inside(p):     \n",
    "  x, y = random.random(), random.random()\n",
    "  return x*x + y*y < 1\n",
    "\n",
    "# build parallel data structure\n",
    "count = sc.parallelize(range(0, num_samples)).filter(inside).count()\n",
    "pi = 4 * count / num_samples\n",
    "print(pi)\n",
    "sc.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Logistical Regression in Spark\n",
    "\n",
    "```scala\n",
    "val points = spark.textFile(...)\n",
    "                  .map(parsePoint).persist()\n",
    "var = w  // random initial vector\n",
    "for (i <- 1 to ITERATIONS) {\n",
    "    val gradient = points.map{ p => \n",
    "          p.x * (1/(1+exp(-p.y*(w dot p.x)-1)*p.y\n",
    "    }.reduce((a+b) => a+b) \n",
    "    w -= gradient\n",
    "}\n",
    "```\n",
    "\n",
    "* Features:\n",
    "  * Scala closures, functions with free variables\n",
    "  * `points` is a read-only RDD reused in each iteration\n",
    "  * Only w (a scalar) gets updated\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Managing Memory in Spark\n",
    "\n",
    "* `persist()` indicates desire to reuse an RDD, encourages Spark to keep it in memory\n",
    "* Spark breaks data up into: \n",
    "  * RDD: the representation of a logical data set\n",
    "  * sequence: a physical, materialized data set\n",
    "* In Spark-land, RDDs and sequences are differentiated by the concepts of \n",
    "  * **Transformations: RDD->RDD**\n",
    "  * **Actions: RDD->sequence/data**\n",
    "* RDDs define a pipeline of computations from data set (HDFS) to sequence/data\n",
    "* RDDs evaluated lazily as needed to build a sequence\n",
    "  * A sequence computation pulls data through the Spark pipeline\n",
    "* Parallelized constructs in Spark\n",
    "  * **Transformations are lazy whereas actions launch computation**\n",
    "  \n",
    "<img src=\"./images/spark_ops.png\" width=768 />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note on `map` v.s. `flatMap`\n",
    "* `map`: 1 to 1 relation between input and output partitions\n",
    "* `flatmap`: can have arbitrary cardinality. This is the \"Map\" of Google's Map/Reduce"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Map/Reduce in Spark\n",
    "\n",
    "The following steps compute a Map/Reduce\n",
    "```\n",
    "spark.textfile(...).flatMap(...).reduceByKey(...).save()\n",
    "```\n",
    "* Doesn’t use RDD pipelining.  \n",
    "* `flatMap` produces a sequence.\n",
    "* Doesn’t use memory abstraction\n",
    "\n",
    "#### Many Maps\n",
    "\n",
    "* `map()` is one-to-one consistent w/ scala semantics\n",
    "* `flatMap` is many-to-one like Map in M/R\n",
    "* `mapValues` does not transform key (important for partitioning)\n",
    "\n",
    "#### M/R equivalent in PySpark\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                \r"
     ]
    }
   ],
   "source": [
    "from pyspark import SparkContext\n",
    "sc = SparkContext(\"local\", \"App Name\",)\n",
    "\n",
    "text_file = sc.textFile(\"./data/textbible/*\")\n",
    "counts = text_file.flatMap(lambda line: line.split(\" \")) \\\n",
    "             .map(lambda word: (word, 1)) \\\n",
    "             .reduceByKey(lambda a, b: a + b)\n",
    "counts.saveAsTextFile(\"/tmp/sparkoutput2\")\n",
    "\n",
    "sc.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note:\n",
    "* ```line.split(\" \")``` breaks each line into tokens: `flatMap` operation\n",
    "* word -> (word, 1) is 1-to-1 mapping: `map` operation\n",
    "\n",
    "* Map/Reduce guarantee of sorted key within each partition no longer holds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "sc.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Lineage and Recovery\n",
    "\n",
    "* Spark in the presence of failures:\n",
    "  * Identify partitions of data (in an RDD) that have failed\n",
    "  * Recompute the failed partitions using lineage\n",
    "  * Parallelize recomputation (using Spark)\n",
    "  * Easy because all RDDs are immutable\n",
    "* Does not require checkpoint/restart or rollback\n",
    "  * Checkpoint = save required memory (application state) to persistent storage sufficient to restart the computation\n",
    "  * Restart = restart computation from a checkpoint\n",
    "  * Rollback = Undo changes made to memory associated with computations that have failed or did not complete\n",
    "  * All concepts related to managing a writeable memory related to computational progress in the presence of failures.\n",
    "  \n",
    "#### Spark Checkpointing\n",
    "\n",
    "* Spark supports checkpoints as performance optimization\n",
    "  * save intermediate computation state as RDD\n",
    "  * make an RDD persistent to limit recovery work\n",
    "* It is desirable to checkpoint when:\n",
    "  * Lineages become long (in dependencies)\n",
    "  * Dependencies become wide\n",
    "* This is a tradeoff between memory usage and redundancy\n",
    "* Spark’s default is to use the initial data load as the only checkpoint and restart from that checkpoint\n",
    "\n",
    "**PageRank** Checkpoint Example. Scala code and the resulting lineage.\n",
    "```scala\n",
    "val links = spark.textFile(...).map(...).persist()\n",
    "var ranks = // RDD of (URL, rank) pairs\n",
    "for (i <- 1 to ITERATIONS) {\n",
    "    // Build an RDD of (TargetURL, float) pairs\n",
    "    // with contributions sent by each page\n",
    "    val contribs = links.join(ranks).flatMap {\n",
    "        (url, (links, rank)) => \n",
    "            links.map(dest => dest, rank/link.size))\n",
    "    }\n",
    "    // Sum contributions by URL and get new ranks\n",
    "    ranks = contribs.reduceByKey((x,y) => x+y)\n",
    "        .mapValues(sum => a/N + (1-a)*sum)\n",
    "}\n",
    "```\n",
    "<img src=\"images/sparkpr.png\" width=384 />\n",
    "\n",
    "**What RDDs do we want to persist?**\n",
    "\n",
    "Links, since they are read-only, use the same copy every iteration, and reused. Use `persist()` to ensure that links are only read from disk into cache once.\n",
    "\n",
    "Recovering the computation from lineage repeats the entire comptutation history.  One would prefer to checkpoint `ranks` intermittently so that the computation can restart from a recent point.\n",
    "\n",
    "\n",
    "\n",
    "#### Dependencies\n",
    "\n",
    "RDDs exist in partitions in which each partition is (potentially) on a different computer.  So when a node fails, that parition fails.\n",
    "\n",
    "<img src=\"images/sparkdep.png\" width=512 />\n",
    "\n",
    "Different operations have different dependency patterns.\n",
    "\n",
    "* If partition fails in an RDD with a wide-dependency, the entire previous computation needs to be repeated\n",
    "  * Partition after wide dependencies.\n",
    "\n",
    "\n",
    "\n",
    "  \n",
    "  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Conclusions\n",
    "\n",
    "Everything M/R can do.  \n",
    "\n",
    "* With memory abstractions.\n",
    "  * Caching\n",
    "  * Persistence\n",
    "  * Checkpoints\n",
    "* And support for iterative algorithms\n",
    "* And an interactive programming interface\n",
    "* And multi-language support"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}