Lec05.cache_hierarchy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Cache Hierarchy\n",
    "\n",
    "* Memory is an abstraction\n",
    "  * looks to processor like a 1-d adress space of data locations\n",
    "  * uniform access from all cores/processors\n",
    "*  Actually a steep, hieararchy of cache in which different levels have different:\n",
    "  * Performance\n",
    "  * Capacity\n",
    "  * Sharing\n",
    "  \n",
    "<img src=\"https://sites.google.com/site/cachememory2011/_/rsrc/1311628836036/memory-hierarchy/hei.png\" width=512 title=\"Cache Hierarchy\" />\n",
    "\n",
    "* Caches are a place to store a smaller amount of data the is frequently/recently used to make data access faster.\n",
    "  * Processor caches (on chip) cache for memory.  Managed by hardware.\n",
    "  * Memory (DRAM) is a cache for pages from disk.  Managed by a storage system (database, file system).\n",
    "  * Management refers to the process of loading and evicting the contents in response to workload.\n",
    "  \n",
    "### The Hierarchy\n",
    "\n",
    "<img src=\"http://www.imexresearch.com/newsletters/images/201009_SSDImages/20100913_SSD_0000.png\" width=512 title=\"IMEX Data on Latency and Cost\" />\n",
    "\n",
    "<img src=\"https://eda360insider.files.wordpress.com/2012/05/wegener-1.gif?w=1400\" width=512 title=\"Cache latency and granularity\" />\n",
    "\n",
    "### Latency\n",
    "\n",
    "Delays (in clock cycles) to different levels in the cache hierarchy for an i7 (Nehalem)\n",
    " * $1$ cycle to registers (private to each core)\n",
    " * $1$ cycle to L1 (private to each core)\n",
    " * $4$ cycles to L2 (private to each core)\n",
    " * $35$ cycles to L3 (shared by cores)\n",
    " * $145$ cycles to memory (shared by processors)\n",
    " * $10^5$ cycles to NVRAM\n",
    " * $10^7$ cycles to magnetic disk\n",
    "\n",
    "_Data Loading_: New data that has not been used yet must come from SSD or disk.  Can be very slow.\n",
    "\n",
    "_Data Sharing_: When two threads need to share data, they incur the cost of transferring data through the fastest shared cache.\n",
    "  * 2 cores on the same processor take 70 cycles (35 to write to L3 and 35 to read from L3)\n",
    "  * 2 processors take 290 cycles\n",
    "  \n",
    "The following figure is almost right. SMP should really say something like QPI (quickpath interconnect) or somesuch. It's not an SMP (symmetric multiprocessor). However, it is useful to visualize sharing betweeen cores in L2 and processors in L3.\n",
    "\n",
    "<img src=\"https://www.enterpriseai.news/wp-content/uploads/2014/06/shared-memory-cluster-story-1-processor-cluster.jpg\" width=512 title=\"NUMA schematic from EnterpriseAI\" />\n",
    "\n",
    "This sharing result in _interference_ between processes that share data in OpenMP and threads.  This is the major source for lost parallelism in these programming models.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Processor Caching Concepts\n",
    "\n",
    "The memory system should be thought of a a vectorized parallel system.  Whenever you \n",
    "get data, you get many words of data.  To get good memory throughput, you must \n",
    "use all that data.  Most important to understanding cache performance are:\n",
    "* __cache line__: data are moved among levels in the cache one line at a time\n",
    "  * as small as 64 bytes (L1 or L2)\n",
    "  * think of each load as the parallel load of an entire line\n",
    "  * good parallel programs will use 64 or 128 bytes\n",
    "* __unified__: refers to whether or not the cache is shared (among cores or processors)\n",
    "\n",
    "Other concepts that don't matter as much.\n",
    "* __inclusive vs exclusive__: has implications for hardware management policies.  We don't care.\n",
    "  * __inclusive__: data in higher level caches are also in lower level caches\n",
    "  * __exclusive__: data in higher level caches are not in lower level caches\n",
    "* __associativity__: the number of hardware locations that a cache line can go into\n",
    "  * important for HW design.  We typically don't care until we get to CUDA."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### So What\n",
    "\n",
    "* You can’t just access memory\n",
    "  * Different memory access patterns result in 50x performance differences for the same computation\n",
    "* Worry about:\n",
    "  * Parallelism: am I using all the data in a cache line\n",
    "    * To access a single byte, one must load a whole line\n",
    "    * Sequential access to memory is always parallel!\n",
    "  * Sharing/reuse: is my program referencing data in the cache more than once?  At what levels?\n",
    "\n",
    "Good memory access patterns are __aligned, sequential__ and __coalesced__.\n",
    "  * Aligned – access range starts/ends on cache line boundaries\n",
    "  * Sequential – a continuous range of bytes\n",
    "  * Coalesced – combine multiple small accesses into fewer large accesses\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Memory Access Patterns in OpenMP\n",
    "\n",
    "* To be efficient, memory access should be __sequential__ and __aligned__\n",
    "  * Sequential = in memory address order\n",
    "  * Aligned = to physical boundaries in HW. Cache lines, memory pages.\n",
    "  \n",
    "<img src=\"https://cvw.cac.cornell.edu/gpu/images/figure8.png\" width=512 />\n",
    "\n",
    "This is a picture from a Cornell workshop, but it makes the point.\n",
    "\n",
    "* To accomplish this in looping programs\n",
    "  * choose an iteration order that is sequential in memory\n",
    "  * allocate data aligned (or hope it happens because alignment interfaces are not portable)\n",
    "* Nested loops are a good example\n",
    "  * Row versus column order can make a big difference.\n",
    "  * Think of memory as reading a sequential cache line at a time\n",
    "\n",
    "  \n",
    "<img src=\"./images/rowvcol.png\" width=512 title=\"http://akira.ruc.dk/~keld/teaching/IPDC_f10/Slides/pdf4x/4_Performance.4x.pdf\" />\n",
    "\n",
    "* Reading data a row at a time results in sequential access of all elements.  \n",
    "* Reading successive elements in a column results in strided I/O."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LMBench: Understanding Cache Misses\n",
    "\n",
    "LMBench is a suite of performance benchmarking tools written by Carl Staelin and Larry McVoy in 1996. The strided access benchmark still provides the best insight into the structure of cache latencies.  The experiment does the following:\n",
    "\n",
    "> Access a single byte at 128 byte strides (i.e. 0, 128, 256, 384, ...) for an array of a specified size.  Loop over the array multiple times to amortize any initial load costs.\n",
    "\n",
    "Plotting the per access latency versus the array size reveals the structure of the cache. A blog about the Alibaba Cloud shows a simple to interpret chart:\n",
    "\n",
    "<img \n",
    "     src=\"https://yqintl.alicdn.com/83b8b62fadc0921e4f516e161f254173913aa779.png\" width=768 title=\"AliBaba Cloud Memory Performance (from: https://www.alibabacloud.com/blog/understanding-memory-performance_594708)\" />\n",
    "\n",
    "For arrays that fit into:\n",
    "  * L1 cache: the L1 cache that contains the entire array and each byte can be accessed in a single clock cycle\n",
    "  * L2 cache but not L1 cache: every byte access transfers a line from the L2 cache to the L1 cache. (Lines are 128 bytes). Each access occurs at L2 latency\n",
    "  * L3 cache but not L2 cache: every byte access transfers a line from L3->L2->L1.\n",
    "  * Larger than L3: performance increases as a function of the working set size. The operating system manages this cache and has access to predictive prefetching and other optimizations.\n",
    "    \n",
    "__Conclusion__: the exact same code at different sizes can have >20x performance differences.\n",
    "  * you have to understand the cache hierarchy and reason about where your code runs to write fast code.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LMBench and NUMA:\n",
    "\n",
    "<img src=\"https://sites.utexas.edu/jdm4372/files/2012/03/RangerLatencyChart.jpg\" width=768 title=\"Ranger Memory Performance at TACC\" />\n",
    "\n",
    "For reads that are bigger than L3, there is different performance depending upon on which physical memory the array resides. This is a direct measure of the non-uniformity of memory.\n",
    "\n",
    "__Conclusion__: NUMA matters."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}