GANs_disc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Unsupervised representation learning with generative adversarial networks (GANs)\n",
    "\n",
    "### Outline\n",
    "* What is a GAN?\n",
    "* Why study GANs?\n",
    "* GANs vs Other generative models\n",
    "* How do GANs work?\n",
    "    * GAN framework\n",
    "    * Training process\n",
    "        * Cost functions\n",
    "        * Minimax game\n",
    "* Implementations\n",
    "    * GAN implementation\n",
    "    * DCGAN implementation\n",
    "    * Cycle GAN implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$ % Latex macros\n",
    "\\newcommand{\\mat}[1]{\\begin{pmatrix} #1 \\end{pmatrix}}\n",
    "\\newcommand{\\p}[2]{\\frac{\\partial #1}{\\partial #2}}\n",
    "\\newcommand{\\b}[1]{\\boldsymbol{#1}}\n",
    "\\newcommand{\\c}[1]{\\mathcal{#1}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How do GANs work?\n",
    "### GAN Framework\n",
    "\n",
    "A generative adversarial network consists of two models: the __generator__ and the __discriminator__. \n",
    "    \n",
    "The discriminator is a binary classifier  which identifies whether an input belongs to either of the two classes, _real_ or _fake_. Meanwhile, the generator is trained to fool the discriminator such that its outputs are classified as part of the _real_ data. \n",
    "\n",
    "The generator could be thought of as being like a forger that makes fake art while a discriminator is like a fraud detective that distinguishes between genuine and fake artwork. The goal of the forger is to be able to make art that is as indistinguishable from the real ones, as much as the goal of the generator is to create fake samples that are drawn from the same distribution as the actual data.\n",
    "\n",
    "Hence we end up with two models competing against each other with the generator learning how to generate samples that the discrimniator can no longer detect as being fake. The competition between the two networks drives their learning.\n",
    "\n",
    "<img src=\"figures/GAN_framework.png\" width=\"500\">\n",
    "\n",
    "Formally, the players in the game are represented as two functions: \n",
    "* Discriminator: $D(x, \\b \\theta^{(D)})$ where $\\b x$ are observed variables\n",
    "* Generator: $G(z, \\b \\theta^{(G)})$ where $\\b z$ are latent variables\n",
    "\n",
    "$\\theta^{(i)}$ represent the parameters/weights of the models\n",
    "\n",
    "The discriminator $D(x, \\b \\theta^{(D)})$ and the generator $G(z, \\b \\theta^{(G)})$ optimize cost functions that are dependent on each others' parameters: \n",
    "\n",
    "$J^{(D)}(\\b \\theta^{(D)}, \\b \\theta^{(G)})$ $\\rightarrow$ The discriminator minimizes this cost function while changing only its parameters $\\b \\theta^{(D)}$\n",
    "\n",
    "$J^{(G)}(\\b \\theta^{(D)}, \\b \\theta^{(G)})$ $\\rightarrow$ The generator minimizes this cost function while changing only \n",
    "its parameters $\\b \\theta^{(G)}$\n",
    "\n",
    "This problem is framed more easily as a game rather than an optimization problem, because both players have cost functions that depend on each others' parameters while only having control of their own parameters. In a game theoretic approach, the problem becomes that of finding the Nash equilibria $(\\b \\theta^{(D)}, \\b \\theta^{(G)})$ for which $J^{(D)}$ is a minimum with respect to $\\b \\theta^{(D)}$ and $J^{(G)}$ is a minimum with respect to $\\b \\theta^{(G)}$.\n",
    "\n",
    "In order to train the network, simultaneous stochastic gradient descent (SGD) steps are taken for the generator and the discriminator: one step for $D$ to minimize $J^{(D)}$ and another step for $G$ to minimize $J^{(G)}$\n",
    "\n",
    "### Cost functions\n",
    "\n",
    "$\\textbf {Discriminator cost function}$\n",
    "\n",
    "For most implementations of GANs, the cost function used for the discriminator is the standard binary cross-entropy loss used for binary classifiers with sigmoid output activations. The only difference is that it is trained on both real data from the dataset (labeled as $1$) and fake data from the generator (labeled as $0$). \n",
    "\n",
    "$$J^{(D)}(\\b \\theta^{(D)}, \\b \\theta^{(G)})  = -\\textbf{E}_{x\\sim p_{data}(x)}[\\log{D(x)}] - \\textbf{E}_{z\\sim p_{z}(z)}[\\log{(1-D(G(z)))}]$$\n",
    "\n",
    "In this manner, the model approximates by using supervised learning to estimate a density ratio:\n",
    "$$\\frac{p_{data}(x)}{p_{model}(x)}$$\n",
    "\n",
    "$\\textbf {Generator cost function}$\n",
    "\n",
    "To complete the specification of the game, the generator cost function must be defined. The simplest case is to consider a zero-sum game, i.e., the total cost for all players is always zero:\n",
    "\n",
    "$$J^{(G)} = -J^{(D)}$$\n",
    "\n",
    "Then we can define the game just by specifying the discriminator pay-off:\n",
    "\n",
    "$$V(\\b \\theta^{(D)}, \\b \\theta^{(G)})  = J^{(D)}(\\b \\theta^{(D)}, \\b \\theta^{(G)})$$\n",
    "\n",
    "\n",
    "And we end up with a $\\textbf {minimax}$ game\n",
    "\n",
    "$$\\arg \\min_{\\b \\theta^{(G)}}\\max_{\\b \\theta^{(D)}}V(\\b \\theta^{(G)}, \\b \\theta^{(D)})  = \\textbf{E}_{x\\sim p_{data}(x)}[\\log{D(x)}] + \\textbf{E}_{z\\sim p_{z}(z)}[\\log{(1-D(G(z)))}]$$\n",
    "\n",
    "In practice, the two players are represented as neural networks for differentiability.  \n",
    "\n",
    "\n",
    "The following section describes the first implementation of a GAN using two multilayer perceptrons acting as the adversarial discriminator and generator networks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vanilla Generative Adversarial Network (GAN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Deep convolutional GAN (DCGAN)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the most widely used type of GAN is the deep convolutional GAN (DCGAN) first implemented by Radford et al (). The main feature of the DCGAN is its use of all convolutional layers with batch normalization to stabilize training. The discriminator network is a convolutional neural network classifier with all convolutional layers (CNN), while the generator is a network of transposed convolution blocks. \n",
    "\n",
    "<table align=\"center\">\n",
    "<tr>\n",
    "<td> <img src=\"figures/conv_anim.gif\" style=\"height: 350px; width: 350px\"/> \n",
    "<td> <img src= \"figures/convT_anim.gif\" style=\"height: 350px; width: 350px\"/> \n",
    "</tr>\n",
    "<tr>\n",
    "<th style= \"text-align:center\"> Convolution operation </th >\n",
    "<th style= \"text-align:center\"> Convolution transpose operation</th>  \n",
    "</tr>\n",
    "</table>\n",
    "\n",
    "The key features of a DCGAN are given by Radford et al (2016) as follows :\n",
    "<img src=\"figures/Architecture_guidelines.png\" width=\"700\">\n",
    "\n",
    "\n",
    "Batch normalization is described by the following algorithm:\n",
    "<img src=\"figures/batch_norm.png\" width=\"300\">\n",
    "\n",
    "The architecture used for their generator network is shown below:\n",
    "<img src=\"figures/Generator_network.png\" width=\"700\">\n",
    "<table align=\"center\">\n",
    "<tr>\n",
    "<th style= \"text-align:center\"> Generator network architecture for the DCGAN</th>\n",
    "</tr>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DCGAN PyTorch implementation\n",
    "\n",
    "In our implementation of the DCGAN, we follow the architecture and training method used in Radford et al, 2016. Shown bellow are code snippets for the pytorch implementation of a DCGAN."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import division\n",
    "from IPython.display import clear_output\n",
    "import torch\n",
    "from torch import nn\n",
    "from torch.autograd import Variable\n",
    "from torch.optim import Adam\n",
    "from torchvision import transforms, datasets\n",
    "import torchvision.utils as vutils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Discriminator(torch.nn.Module):\n",
    "    \"\"\"\n",
    "    This discriminator network is based on the original DCGAN paper by Radford et al.\n",
    "    The discriminator is a CNN which takes as input a 3-channel image data (i.e. RGB image)\n",
    "    and outputs a probability,p(real), that the image is from the real dataset.\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        super(Discriminator, self).__init__()\n",
    "        #Conv block 1\n",
    "        self.Conv1 = nn.Sequential(\n",
    "            nn.Conv2d(in_channels=3, out_channels=128, kernel_size=4, stride=2, padding=1, bias=False),\n",
    "            nn.LeakyReLU(0.2, inplace=True)\n",
    "        )\n",
    "        #Conv block 2\n",
    "        self.Conv2 = nn.Sequential(\n",
    "            nn.Conv2d(in_channels=128, out_channels=256, kernel_size=4, stride=2, padding=1, bias=False),\n",
    "            nn.BatchNorm2d(256),\n",
    "            nn.LeakyReLU(0.2, inplace=True)\n",
    "        )\n",
    "        #Conv block 3\n",
    "        self.Conv3 = nn.Sequential(\n",
    "            nn.Conv2d(in_channels=256, out_channels=512, kernel_size=4, stride=2, padding=1, bias=False),\n",
    "            nn.BatchNorm2d(512),\n",
    "            nn.LeakyReLU(0.2, inplace=True)\n",
    "        )\n",
    "        #Conv block 4\n",
    "        self.Conv4 = nn.Sequential(\n",
    "            nn.Conv2d(in_channels=512, out_channels=1024, kernel_size=4,stride=2, padding=1, bias=False),\n",
    "            nn.BatchNorm2d(1024),\n",
    "            nn.LeakyReLU(0.2, inplace=True)\n",
    "        )\n",
    "        self.Out = nn.Sequential(\n",
    "            nn.Linear(1024*4*4, 1),\n",
    "            nn.Sigmoid(),\n",
    "        )\n",
    "\n",
    "    def forward(self, I):\n",
    "        # Convolutional layers\n",
    "        X = self.Conv1(I)\n",
    "        X = self.Conv2(X)\n",
    "        X = self.Conv3(X)\n",
    "        X = self.Conv4(X)\n",
    "        # reshape and apply sigmoid activation\n",
    "        X = X.view(-1, 1024*4*4)\n",
    "        X = self.Out(X)\n",
    "        return X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Generator(torch.nn.Module):\n",
    "    \"\"\"\n",
    "    This generator network is based on the original DCGAN paper by Radford et al.\n",
    "    The generator takes as input a 100-dimensional noise vector (z) and maps it to the data space \n",
    "    (which in this case is the image space) via a series of transposed convolution blocks.\n",
    "    From the input random noise, the generator outputs an image with the same size as the input.    \n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self):\n",
    "        super(Generator, self).__init__()\n",
    "        self.nz = 100\n",
    "        self.linear = torch.nn.Linear(self.nz, 1024*4*4)\n",
    "        \n",
    "        #first transposed convolution block\n",
    "        self.Conv1 = nn.Sequential(\n",
    "            nn.ConvTranspose2d(in_channels=1024, out_channels=512, kernel_size=4,stride=2, padding=1, bias=False),\n",
    "            nn.BatchNorm2d(512),\n",
    "            nn.ReLU(inplace=True))\n",
    "        \n",
    "        #second transposed convolution block\n",
    "        self.Conv2 = nn.Sequential(\n",
    "            nn.ConvTranspose2d(in_channels=512, out_channels=256, kernel_size=4, stride=2, padding=1, bias=False),\n",
    "            nn.BatchNorm2d(256),\n",
    "            nn.ReLU(inplace=True)\n",
    "        )\n",
    "        \n",
    "        #third transposed convolution block\n",
    "        self.Conv3 = nn.Sequential(\n",
    "            nn.ConvTranspose2d(in_channels=256, out_channels=128, kernel_size=4, stride=2, padding=1, bias=False),\n",
    "            nn.BatchNorm2d(128),\n",
    "            nn.ReLU(inplace=True)\n",
    "        )\n",
    "        \n",
    "        #fourth transposed convolution block\n",
    "        self.Conv4 = nn.Sequential(\n",
    "            nn.ConvTranspose2d(in_channels=128, out_channels=3, kernel_size=4, stride=2, padding=1, bias=False)\n",
    "        )\n",
    "        self.out = torch.nn.Tanh()\n",
    "\n",
    "    def forward(self, z):\n",
    "        \"\"\"\n",
    "        Perform forward calculation for generator output, given random noise input z\n",
    "        \"\"\"\n",
    "        # Project and reshape\n",
    "        X = self.linear(z)\n",
    "        X = X.view(X.shape[0], 1024, 4, 4)\n",
    "        # conv blocks\n",
    "        X = self.Conv1(X)\n",
    "        X = self.Conv2(X)\n",
    "        X = self.Conv3(X)\n",
    "        X = self.Conv4(X)\n",
    "        # tanh activation\n",
    "        return self.out(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_discriminator(optimizer, real_data, fake_data):\n",
    "    # Reset gradients\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # 1.1 Train on Real Data\n",
    "    prediction_real = netD(real_data)\n",
    "    # Calculate error and backpropagate\n",
    "    error_real = loss(prediction_real, real_data_target(real_data.size(0)))\n",
    "    error_real.backward()\n",
    "\n",
    "    # 1.2 Train on Fake Data\n",
    "    prediction_fake = netD(fake_data)\n",
    "    # Calculate error and backpropagate\n",
    "    error_fake = loss(prediction_fake, fake_data_target(real_data.size(0)))\n",
    "    error_fake.backward()\n",
    "    \n",
    "    # 1.3 Update weights with gradients\n",
    "    optimizer.step()\n",
    "    \n",
    "    # Return error\n",
    "    return error_real + error_fake, prediction_real, prediction_fake\n",
    "\n",
    "def train_generator(optimizer, fake_data):\n",
    "    # 2. Train Generator\n",
    "    # Reset gradients\n",
    "    optimizer.zero_grad()\n",
    "    # Sample noise and generate fake data\n",
    "    prediction = netD(fake_data)\n",
    "    # Calculate error and backpropagate\n",
    "    error = loss(prediction, real_data_target(prediction.size(0)))\n",
    "    error.backward()\n",
    "    # Update weights with gradients\n",
    "    optimizer.step()\n",
    "    # Return error\n",
    "    return error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weights_init(m):\n",
    "    classname = m.__class__.__name__\n",
    "    if classname.find('Conv') != -1 or classname.find('BatchNorm') != -1:\n",
    "        m.weight.data.normal_(0.00, 0.02)\n",
    "        \n",
    "def noise(s):\n",
    "    \"\"\"\n",
    "    Generate s-dimensional noise vector from random normal distribution with mean zero and std one\n",
    "    \"\"\"\n",
    "    z = Variable(torch.randn(s, 100))\n",
    "    if torch.cuda.is_available(): \n",
    "        return z.cuda()\n",
    "    return z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create network instances\n",
    "netG = Generator()\n",
    "netD = Discriminator()\n",
    "\n",
    "#initialize weights\n",
    "netD.apply(weights_init)\n",
    "netG.apply(weights_init)\n",
    "\n",
    "#use cuda if available\n",
    "if torch.cuda.is_available():\n",
    "    netG.cuda()\n",
    "    netD.cuda()\n",
    "\n",
    "# Set learning rate\n",
    "lr = 0.0002\n",
    "\n",
    "# Number of training epochs\n",
    "num_epochs = 200\n",
    "\n",
    "# setup optimizers\n",
    "optD = Adam(netD.parameters(), lr=lr, betas=(0.5, 0.999))\n",
    "optG = Adam(netG.parameters(), lr=lr, betas=(0.5, 0.999))\n",
    "\n",
    "# fixed test noise\n",
    "test_noise = noise(20)\n",
    "\n",
    "# loss function\n",
    "loss = nn.BCELoss()\n",
    "\n",
    "real_label = 1\n",
    "fake_label = 0\n",
    "\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for epoch in range(num_epochs):\n",
    "    dloss_log = []\n",
    "    gloss_log = []\n",
    "    for i, data in enumerate(data_loader, 0):\n",
    "        real_data = Variable(data[0])        \n",
    "        # 1. Train Discriminator\n",
    "        if torch.cuda.is_available(): real_data = real_data.cuda()\n",
    "        # Generate fake data\n",
    "        fake_data = netG(noise(real_data.size(0))).detach()\n",
    "        # Train D\n",
    "        d_error, d_pred_real, d_pred_fake = train_discriminator(optD, \n",
    "                                                                real_data, fake_data)\n",
    "\n",
    "        # 2. Train Generator\n",
    "        # Generate fake data\n",
    "        fake_data = netG(noise(real_data.size(0)))\n",
    "        # Train G\n",
    "        g_error = train_generator(optG, fake_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results for MNIST\n",
    "<img src=\"figures/MNIST.gif\" width = '800'> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results for Fashion MNIST\n",
    "<img src=\"figures/FASHION.gif\" width = '800'> "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Results for CIFAR10\n",
    "<img src=\"figures/CIFAR10.gif\" width = '800'> "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}