6 - Cross-Language-Integration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Jupyter and multiple Languages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IPython and cross language integration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The rest of this is mostly a demo on how cross-language integration works in a Jupyter notebook by using the features of the _Reference IPython Kernel implementation_. These features are completely handled in the kernel so need to be reimplemented on a per-kernel basis. Though they **also** work on pure terminal IPython, nbconvert or any other programmatic use of IPython.\n",
    "\n",
    "Most of what you will see here are _just_ thin wrappers around already existing libraries. These libraries (and their respective authors) do all the heavy lifting. I just show how seamless a cross language environment can be from the user point of view. The installation of these library might not be easy either and getting all these language to play together can be complex task. It is though becoming easier and easier.\n",
    "\n",
    "The term _just_ does not imply that the wrappers are simple, or easy to write. It indicate that the wrappers are far from being complete. What is shown here is completely doable using standard Python syntax and bit of manual work. SO what you'll see here is mostly _convenience_. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The good old example of Fibonacci"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understanding the multiple languages themselves is not necessary; most of the code here should self explanatory and straightforward. We'll define many function that compute the nth `Fibonacci` number more or less efficiently. We'll define them either using the classic recursive implementation, or sometime using an unrolled optimized version. As a reminder the Fibonacci sequence is defines a the following:\n",
    "\n",
    "$$ F_n = \\begin{cases} 1 &\\mbox{if } n \\leq 2 \\\\ \n",
    "F_{n-1}+F_{n-2} & \\mbox{otherwise }\\end{cases}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The fact that we calculate the Fibonacci sequence as little importance, except that the value of $F_n$ can grow _really_ fast in $O(e^n)$ if I remember correctly. And the recursive implementation will have a hard time getting beyond (n=100) as the number of call will be greater than $O(e^n)$ as well. Be careful especially if you calculate $F_{F_n}$ or more composition. Remembering that n=5 is stable via $F$ might be useful. \n",
    "\n",
    "Here are the first terms of the Fibonacci sequence:\n",
    "\n",
    " 1. 1\n",
    " 1. 1\n",
    " 1. 1+1 = 2\n",
    " 1. 2+1 = 3\n",
    " 1. 3+2 = 5\n",
    " 1. 5+3 = 8\n",
    " 1. 8+5 = 13\n",
    " ...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Python cross-language integration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define the `fibonacci` function in python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fib(n):\n",
    "    \"\"\"\n",
    "    A simple definition of fibonacci manually unrolled\n",
    "    \"\"\"\n",
    "    if n<2:\n",
    "        return 1\n",
    "    x,y = 1,1\n",
    "    for i in range(n-2):\n",
    "        x,y = y,x+y\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "[fib(i) for i in range(1,10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Store the value from 1 to 30 in `Y`, and graph it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "X = np.arange(1,30)\n",
    "Y = np.array([fib(x) for x in X])\n",
    "import matplotlib.pyplot as plt\n",
    "fig, ax = plt.subplots()\n",
    "ax.scatter(X, Y)\n",
    "ax.set_xlabel('n')\n",
    "ax.set_ylabel('fib(n)')\n",
    "ax.set_title('The Fibonacci sequence grows fast !')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It may not surprise you, but this looks like an exponential, so if we were to look at $log(fib(n))$ × $n$ it would look approximately like a line.:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can try to do a linear regression using this model. R is a language many people use to do statistics. So, let's use R. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's enable integration between Python and R using the [__`RPy2`__](https://rpy2.readthedocs.io/en/version_2.8.x/) python package developed by Laurent Gautier and the rest of the rpy2 team."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Side note, you might need to change the environment variable passed to your kernel for this to work.\n",
    "Here is what I had to do only once.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#!a2km add-env 'python 3' DYLD_FALLBACK_LIBRARY_PATH=$HOME/anaconda/pkgs/icu-54.1-0/lib:/Users/bussonniermatthias/anaconda/pkgs/zlib-1.2.8-3/lib "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import rpy2.rinterface\n",
    "\n",
    "%load_ext rpy2.ipython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Following will \"Send\" the X and Y array to R."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%Rpush Y X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now let's try to fit a linear model ($ln(Y) = A.X + B$) using R. I'm not a R user myself, so don't take this as idiomatic R."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%R\n",
    "my_summary = summary(lm(log(Y)~X))\n",
    "val <- my_summary$coefficients\n",
    "\n",
    "plot(X, log(Y))\n",
    "abline(my_summary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%R\n",
    "my_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good, we have now the some statistics on the fit, which also looks good. __And__ we were able to not only send variable to R, but to plot directly from R !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are happy as $F_n = \\left[\\frac{\\phi^n}{\\sqrt 5}\\right]$, where `[]` is closest integer and $\\phi = \\frac{1+\\sqrt 5}{2}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also look at the variables more carefully"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%R\n",
    "val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or even the following that _looks_ more like python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%R val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can even get the variable back from R as Python objects:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "coefs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "coefs = %Rget val\n",
    "y0,k = coefs[0:2]\n",
    "y0,k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's all from the R part. I hope this shows you some of the power of IPython, both in notebook and command line. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cython"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "IPython used to ship with the Cython magic that is now part of [Cython](http://cython.org/) itself.\n",
    "Cython is a superset of Python that compiles to C and importable from Python. You should be a ble to take your python code as is, type annotate it, and get c-like speed.\n",
    "The same principle applies:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import cython"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext cython"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%%cython\n",
    "\n",
    "def cyfib(int n): # note the `int` here\n",
    "    \"\"\"\n",
    "    A simple definition of fibonacci manually unrolled\n",
    "    \"\"\"\n",
    "    cdef int x,y # and the `cdef int x,y` here\n",
    "    if n < 2:\n",
    "        return 1\n",
    "    x,y = 1,1\n",
    "    for i in range(n-2):\n",
    "        x,y = y,x+y\n",
    "    return y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "[cyfib(x) for x in range(1,10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  benchmark"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%timeit -n100 -r3 fib(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%timeit -n100 -r3 cyfib(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The benchmark result can be astonishing, but keep in mind that the Python and Cython version use manually unrolled loop. Main point being that we reached our goal and used Fortran, Cython, C (and Rust) in the middle of our Python program."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try adding a loop around the `%timeit` magic and see if yu can get a graph of runtime vs fibonaci-number for both functions."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}