LSTM_based_RNNs_with_TensorFlow_MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# I - LSTM Based RNNs with TensorFlow\n",
    "In this note book we implement a single layer LSTM based Recurrent Neural Network (RNN) classifier, for MNIST database of handwritten digits. For more useful reading on RNN try the following links. In this notebook I try to combine the good parts of all the following links while presenting a structral and classified view on RNN in TensorFlow.\n",
    "1. __[COlah's](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)__ has an amazing tutorial for understanding LSTM.\n",
    "2. __[wildml's](http://www.wildml.com/2016/08/rnns-in-tensorflow-a-practical-guide-and-undocumented-features/)__ post on RNNs in Tensorflow by Denny Britz. He provides an in depth understanding of RNN in TF, while discuss some undocumented issues.\n",
    "2. __[RNN_for_mnist](https://medium.com/the-artificial-impostor/notes-understanding-tensorflow-part-2-f7e5ece849f5)__ and its __[cloab_page](https://colab.research.google.com/drive/18FqI18psdH30WUJ1uPd6zVgK2AwxO_Bj#scrollTo=A-fpqklBefZy)__ by Ceshine Lee.\n",
    "3. __[Morvan_Zhou's](https://github.com/MorvanZhou/Tensorflow-Tutorial/blob/master/tutorial-contents/402_RNN_classification.py)__ RNN implementation.\n",
    "\n",
    "\n",
    "This note book organization is as follows. If you are just interested to see the code and dont care about details, jump to **IV. RNN Implementation and The Code**.\n",
    "1. LSTM Based RNNs with TensorFlow\n",
    "2. Basics of LSTM Based RNN\n",
    "3. LSTM Based RNN in TensorFlow - A Closer Look into Details\n",
    "    1. Static VS Dynamic RNN\n",
    "    2. Important Notes on the shape of the input and outputs for RNN (tf.nn.dynamic_rnn)\n",
    "    3. Using Dynam RNN Requires Caution if we have inputs with different length \n",
    "    4. The Choice of LSTM Cell\n",
    "4. MNIST Dataset Overview\n",
    "5. RNN Implementation and The Code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# II - Basics of LSTM Based RNN\n",
    "1. Long Short Term Memory networks – usually just called “LSTMs” – are a special kind of RNN, capable of learning long-term dependencies.\n",
    "    1. One main novelty of LSTM is that it uses cell state, i.e., $c_t$. The cell state runs through the entire chain and it is very easy for information to flow down the whole chain (time sequences) with minor linear interaction. Given that, The LSTM does have the ability to remove or add information to the cell state, carefully regulated by structures called gates.\n",
    "    2. One popular LSTM variant, introduced by Gers & Schmidhuber (2000), is adding “peephole connections.” This means that we let the gate layers look at the cell state.\n",
    "    3. The next big thing is attention!\n",
    "    4. Remembering information for long periods of time is practically the LSTM's default behavior, not something they struggle to learn!\n",
    "    \n",
    "2. Below is a schematic representation of LSTM based RNN, from __[colah](http://colah.github.io/posts/2015-08-Understanding-LSTMs/)__ and also TensorFlow documentations.\n",
    "    <img src=\"LSTM3-chain.png\" width=\"600\" height=\"400\">\n",
    "Fig.1 - LSTM based RNN representation by Colah (TF team) [credit:Colah] </figcaption>   \n",
    "3. Before delving into more details, lets remember how Andrew Ng was showing an RNN and resolve a possible confusion due to change of names and notaions between Ng and COlah (TF team).\n",
    "\n",
    "<img src=\"AndrewNg_RNN.png\" width=\"600\" height=\"400\">\n",
    "Fig.2 - LSTM based RNN representation by Andrew Ng\n",
    "\n",
    "Ng uses the following relations to describe the RNN:\n",
    "\n",
    "$\\mathbf{a}^{<t>} = g (\\mathbf{W}_{aa} \\; \\mathbf{a}^{<t-1>} + \\mathbf{W}_{ax} \\; \\mathbf{x}^{<t>} + b_a)$  &nbsp; &nbsp;  <font size=\"3\"> Equation. (1) </font>   \n",
    "\n",
    "$\\mathbf{\\hat{y}}^{<t>} = \\sigma (\\mathbf{W}_{ya} \\; \\mathbf{a}^{<t>} + b_y)$  &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp;  <font size=\"3\"> Equation. (2) </font> \n",
    "\n",
    "Note that $\\mathbf{\\hat{y}}^{<t>}$ is the estimated label at time step $t$, and it is calculated by passing the activation at time step t ($\\mathbf{a}^{<t>}$) through a neural net. But the *outputs* in *tf.nn.dynamic_rnn* is in fact $\\mathbf{a}^{<t>}$, and is denoted by $h_t$ in COlah's and TensorFlow documentations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# III - LSTM Based RNN in TensorFlow - A Closer Look into Details\n",
    "In order to implement any RNN in TF, we need to clearly determine two things:\n",
    "1. The RNN itself, which we can either use a static or a dynamic RNN, as shown below.\n",
    "    1. tf.contrib.rnn.static_rnn (or its aliase tf.nn.static_rnn) \n",
    "    2. tf.nn.dynamic_rnn\n",
    "2. The cell that is being used by the RNN, which also can have many different implementations as shown below:\n",
    "    1. tf.nn.rnn_cell.BasicLSTMCell (aliase with 2) (this one is deprecated)\n",
    "    2. tf.contrib.rnn.BasicLSTMCell (aliase with 1) (this one is deprecated)\n",
    "    3. tf.contrib.rnn.LSTMCell (which is an aliase of 4)\n",
    "    4. tf.nn.rnn_cell.LSTMCell (which is an aliase of 3)\n",
    "    5. tf.contrib.rnn.LSTMBlockCell\n",
    "    6. tf.contrib.rnn.LSTMBlockFusedCell\n",
    "    \n",
    "So, the question is which RNN (static or dynamic) and which LSTM cell (6 cases) shall we use? This is discussed belwo."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III.A. Static VS Dynamic RNN\n",
    "Here we explain whether we use static or dynamic RNN. In short, it is better to use dynamic RNN while we should be careful when we have inputs with different length (number of time steps). \n",
    "\n",
    "Static RNN creates an unrolled graph for a fixed RNN length. That means, if you call tf.nn.rnn with inputs having 200 time steps you are creating a static graph with 200 RNN steps. First, graph creation is slow. Second, you’re unable to pass in longer sequences (> 200) than you’ve originally specified. \n",
    "\n",
    "*tf.nn.dynamic_rnn* solves this. It uses a tf.While loop to dynamically construct the graph when it is executed. That means graph creation is faster and you can feed batches of variable size. So dynamic RNN is faster and more flexible.\n",
    "\n",
    "Also, based on __[here](https://medium.com/the-artificial-impostor/notes-understanding-tensorflow-part-2-f7e5ece849f5)__, we can supply the whole batch of input data as a tensor to dynamic_rnn instead of slicing them into a list of tensor (sequences). This is easier to write and read than static_rnn. See the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# input shape: (batch_size, length (num of time steps), ..(the rest of dimenssions depends on the input data)..)\n",
    "# Static RNN is like below\n",
    "x = tf.unstack(x, timesteps, 1) # <==> This requires to manually take care of shape of the input batch\n",
    "lstm_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)\n",
    "\n",
    "# Dynamic RNN is like below\n",
    "outputs, _ = tf.nn.dynamic_rnn(cell=lstm_cell, inputs=x, time_major=False, dtype=tf.float32)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III.B.  Important Notes on the shape of the input and outputs for RNN (*tf.nn.dynamic_rnn*)\n",
    "<div class=\"alert alert-block alert-danger\"> Notes 1, 2, and 3 are extremely useful. </div>\n",
    "\n",
    "### III.B.1. Note 1\n",
    "Considering **outputs, last_states = tf.nn.dynamic_rnn(cell=lstm_cell,inputs=x,dtype=tf.float32)**, it provides two outputs:\n",
    "1. **outputs: referes to the activations at time step t**, which is:\n",
    "   1. **Based on Andrew Ng's notations: $\\mathbf{a}^{<t>}$**\n",
    "   2. **Based on TensorFlow documentation and COlah's notations: $\\mathbf{h}_{t}$**\n",
    "2. **last_states: referes to the cell state at the last time step, i. e.,  $\\mathbf{c}^{<\\text{t-last-step}>}$. But *tf.nn.dynamic_rnn* not only output $\\mathbf{c}^{<\\text{t-last-step}>}$, but also $\\mathbf{h}_{\\text{t-last-step}}$ as a tuple.** So $\\text{last_states}$ becomes $[\\mathbf{c}^{<\\text{t-last-step}>}, \\mathbf{h}_{\\text{t-last-step}}]$. Hence, $\\mathbf{h}_{\\text{t-last-step}}$ = $\\mathbf{a}^{<\\text{t-last-step}>}$ = $\\text{last_states}[1]$ = $\\text{outputs}[:,-1,:]$. Also $\\mathbf{c}^{<\\text{t-last-step}>} = \\text{last_states}[0]$.\n",
    "\n",
    "### III.B.2.  Note 2\n",
    "**If time_major == False (default mode) then:**\n",
    "- **input must be a Tensor of shape: (batch_size, num_time_steps, ...)**\n",
    "- **outputs will be a Tensor shaped: (batch_size, num_time_steps, cell.output_size)<=> (batch_size , num_time_steps , num_hidden_units)**\n",
    "- **state: will be  (2 , batch_size , num_hidden_units)**. It is in fact $\\mathbf{c}^{<\\text{t-last-step}>}$ and $\\mathbf{a}^{<\\text{t-last-step}>}$ (note $\\mathbf{a}^{<\\text{t-last-step}>} = \\mathbf{h}_{\\text{t-last-step}}$). \n",
    "\n",
    "\n",
    "\n",
    "### III.B.3.  Note 3\n",
    "**Note here is a miss-use of names and might cause confusion.** In TensorFlow documentations the outputs refer to the activation functions at different time_steps, i.e., $\\mathbf{a}^{<t>}$s based on Andrew Ng's notaion (or $\\mathbf{h}_t$ in Christopher Olah's notation (fig.1)). While Andrew Ng refers to the output by $\\mathbf{\\hat{y}}^{<t>}$. (Do not confuse them.) Therefore, in TensorFlow *tf.nn.dynamic_rnn(cell=lstm_cell,inputs=x,dtype=tf.float32)* outputs are $\\mathbf{a}^{<\\text{t-last-step}>}$ and will be of dimenssion (batch_size , num_time_steps , num_hidden_units).  Hence if we are interested to the outpt (estimated label at time step t, i.e., $\\mathbf{\\hat{y}}^{<t>}$), we should use Equation 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III.C. Using Dynam RNN Requires Caution if we have inputs with different length (time steps). Then one should also feed in sequence_length\n",
    "**Example)** For exampl let’s say you have a batch of two examples, one is of length 13, and the other of length 20. Each one is a vector of 128 numbers. The length 13 example is 0-padded to length 20. Then your RNN input tensor is of shape [2, 20, 128]. The dynamic_rnn function returns a tuple of (outputs, state), where outputs is a tensor of size [2, 20, ...] with the last dimension being the RNN output at each time step. state is the last state for each example, and it’s a tensor of size [2, ...] where the last dimension also depends on what kind of RNN cell you’re using.\n",
    "\n",
    "In the above example, once you reach time step 13, your first example in the batch is already “done” and you don’t want to perform any additional calculation on it. The second example isn’t and must go through the RNN until step 20. Hence we need to pass something called sequence_length. By passing sequence_length=[13,20] you tell Tensorflow to stop calculations for example 1 at step 13 and simply copy the state from time step 13 to the end. The output will be set to 0 for all time steps past 13. You’ve just saved some computational cost. But more importantly, if you didn’t pass sequence_length you would get incorrect results! Without passing sequence_length, Tensorflow will continue calculating the state until T=20 instead of simply copying the state from T=13. This means you would calculate the state using the padded elements, which is not what you want.\n",
    "\n",
    "\n",
    "For example if we have an input batch X with two inputs and length of first one is 10 and second is 6 as below, then here is how we pass the sequence_length to the tf.nn.dynamic_rnn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_ex = np.random.randn(2, 10, 8) # we create a batch \n",
    "X_ex[1,6:] = 0 # Force the second example to be of length 6\n",
    "X_ex_lengths = [10, 6] # here we create the vector to be passed as sequence_length\n",
    "cell_ex = tf.nn.rnn_cell.LSTMCell(num_units=64, state_is_tuple=True)\n",
    "outputs_ex, last_states_ex = tf.nn.dynamic_rnn(cell=cell,dtype=tf.float64,sequence_length=X_lengths,inputs=X)\n",
    "result_ex = tf.contrib.learn.run_n({\"outputs\": outputs, \"last_states\": last_states},n=1,feed_dict=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Also further discussion on dynamic and static RNN can be found at __[stackoverflow](https://stackoverflow.com/questions/39734146/whats-the-difference-between-tensorflow-dynamic-rnn-and-rnn)__, __[github](https://github.com/tensorflow/tensorflow/issues/3801)__. Also some further discussions on RNN with tensorflow is avaliable at __[wildml](http://www.wildml.com/2016/08/rnns-in-tensorflow-a-practical-guide-and-undocumented-features/)__ and __[Here](https://medium.com/the-artificial-impostor/notes-understanding-tensorflow-part-2-f7e5ece849f5)__."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## III.D. The Choice of LSTM Cell\n",
    "For LSTM cell we intrduced 6 different API implementaions. And there are even more implementations. So which one shall we use? In short, read this __[page](https://www.tensorflow.org/guide/performance/overview#rnn_performance)__.\n",
    "\n",
    "In fact, as the first two implementations are depracted and the third and fourth are aliases, we have 3 major options. \n",
    "1. tf.nn.rnn_cell.LSTMCell  \n",
    "2. tf.contrib.rnn.LSTMBlockCell\n",
    "3. tf.contrib.rnn.LSTMBlockFusedCell\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "tf.contrib contains volatile and experimental code, and at some point it will be removed. While, tf.nn is said to contain wrappers for primitive neural net operations, and hence is more reliable. The final practice will be using tf.keras or tf.eager. So try to learn them. Refer to TF official page and check their explanation on the API for details.\n",
    "</div>\n",
    "\n",
    "In what follows I choose *tf.nn.rnn_cell.LSTMCell*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IV. MNIST Dataset Overview\n",
    "The MNIST database of handwritten digits, available from this __[Page](http://yann.lecun.com/exdb/mnist/)__, has a training set of 60,000 examples, and a test set of 10,000 examples. The digits have been size-normalized and centered in a fixed-size image (28$\\times$28 pixels where each pixel has a value within the interval [0,1]). Let us load the mnist."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting /home/meysam/github/ML-with-TensorFlow/Datasets/mnist/train-images-idx3-ubyte.gz\n",
      "Extracting /home/meysam/github/ML-with-TensorFlow/Datasets/mnist/train-labels-idx1-ubyte.gz\n",
      "Extracting /home/meysam/github/ML-with-TensorFlow/Datasets/mnist/t10k-images-idx3-ubyte.gz\n",
      "Extracting /home/meysam/github/ML-with-TensorFlow/Datasets/mnist/t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline \n",
    "\n",
    "\n",
    "old_v = tf.logging.get_verbosity()\n",
    "tf.logging.set_verbosity(tf.logging.ERROR)\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "mnist = input_data.read_data_sets(\"/home/meysam/github/ML-with-TensorFlow/Datasets/mnist\", one_hot=True)\n",
    "\n",
    "# Load data\n",
    "X_train = mnist.train.images.reshape([-1,28,28])\n",
    "Y_train = mnist.train.labels\n",
    "X_validation = mnist.validation.images.reshape([-1,28,28])\n",
    "Y_validation = mnist.validation.labels\n",
    "X_test = mnist.test.images.reshape([-1,28,28])\n",
    "Y_test = mnist.test.labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Check the dimenssions\n",
      "X_train.shape: (55000, 28, 28)\n",
      "Y_train.shape: (55000, 10)\n",
      "X_validation.shape: (5000, 28, 28)\n",
      "X_validation.shape: (5000, 28, 28)\n",
      "X_test.shape: (10000, 28, 28)\n",
      "Y_test.shape: (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "print('Check the dimenssions')\n",
    "print('X_train.shape:', X_train.shape)\n",
    "print('Y_train.shape:', Y_train.shape)\n",
    "print('X_validation.shape:', X_validation.shape)\n",
    "print('X_validation.shape:', X_validation.shape)\n",
    "print('X_test.shape:', X_test.shape)\n",
    "print('Y_test.shape:', Y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# V. RNN Implementation and The Code\n",
    "In what follows, we implement a single layer RNN network with LSTM cells with the following parameters:\n",
    "- num_hidden $\\leftrightarrow$ is the number of neurons for each neural network within the LSTM cell. Note that LSTM has 4 neural nets within itself. \n",
    "- num_steps $\\leftrightarrow$ is the number of time steps we consider for our RNN\n",
    "- num_input $\\leftrightarrow$ is the dimenssion of the each input, or equivalently, the number of features. \n",
    "\n",
    "As a side practice try to count the number of parameters that we have by (4 ((num_input + num_hidden ) x num_hidden+1)), as we have 4 neural network in one LSTMM cell, each of which has an input size of (num_input + num_hidden) where the former is due to input and the latter is due to the activation comming from previous layer. Also at the hidden layer we have num_hidden neurons and one bias parameter.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph() # in the begining of the code to avoid namespace error\n",
    "num_hidden = 64\n",
    "num_steps = 28  # as we want to read the images column by column\n",
    "num_input = 28 # as one column of image is 28 pixels\n",
    "num_classes = 10\n",
    "num_epochs = 1\n",
    "batch_size = 32\n",
    "num_batches = int(len(Y_train) / batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>NOTE</b>: As discussed in III.B. Tensorflow’s <i>tf.nn.dynamic_rnn</i> function expects a tensor of shape [B, T, ...] as input, where B is the batch size and T is the length in time of each input (e.g. the number of words in a sentence, or num_steps). The last dimensions depend on your data.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = tf.placeholder(tf.float32, [None, num_steps, num_input])\n",
    "Y = tf.placeholder(tf.float32,[None,num_classes])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "def RNN_model(x,num_classes): \n",
    "    # Define an lstm cell with tensorflow\n",
    "    lstm_cell = tf.nn.rnn_cell.LSTMCell(num_units = num_hidden,state_is_tuple=True) #lstm_cell = rnn.BasicLSTMCell(num_hidden, forget_bias=1.0)\n",
    "\n",
    "    # Get lstm cell output\n",
    "    #outputs: is the RNN output tensor. If time_major == False (default), this will be a Tensor shaped: [batch_size, max_time, cell.output_size] = [batch_size, num_time_steps, num_hidden_units]\n",
    "    #last_states:is the final state of RNN.  cell.state_size is an int, this will be shaped [batch_size, cell.state_size]. If it is a TensorShape, this will be shaped [batch_size] + cell.state_size\n",
    "    outputs, last_states = tf.nn.dynamic_rnn(cell=lstm_cell,  # an instance of RNN cell\n",
    "                                             inputs=x,        # The RNN inputs. If time_major == False (default), this must be a Tensor of shape: [batch_size, max_time, ...], or a nested tuple of such elements\n",
    "                                             dtype=tf.float32 # It is NOT optional, if we do not provide \n",
    "                                             # sequence_length = sequence_length # this one is optional (read the note above on sequence_length). When all our input data points have the same number of time steps\n",
    "                                             # time_major = False # It is optional. time_major determines the shape format of the inputs and outputs Tensors. If true, these Tensors must be shaped [max_time, batch_size, depth]. If false, these Tensors must be shaped [batch_size, max_time, depth].\n",
    "                                            )\n",
    "    \n",
    "    # If you do not need batch normalization, comment next line and change the return\n",
    "    batch_normzd = tf.layers.batch_normalization(outputs[:, -1, :])\n",
    "    \n",
    "    # following Andrew Ng's Notations\n",
    "    y_hat = tf.layers.dense(batch_normzd,num_classes, activation=None, kernel_initializer=tf.orthogonal_initializer())\n",
    "    return y_hat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you were interested in a bidirectional RNN with LSTM cells, check this __[page](http://www.wildml.com/2016/08/rnns-in-tensorflow-a-practical-guide-and-undocumented-features/)__. Now let us create the graph,."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " Train Acc:0.1875   Test Acc:0.125    Train Cost:2.2979531288146973   Test Cost:2.3131699562072754\n",
      "\n",
      " Train Acc:0.78125   Test Acc:0.84375    Train Cost:0.845337986946106   Test Cost:0.6402199268341064\n",
      "\n",
      " Train Acc:0.875   Test Acc:0.75    Train Cost:0.3727201223373413   Test Cost:0.6333989500999451\n",
      "\n",
      " Train Acc:0.9375   Test Acc:0.84375    Train Cost:0.23183369636535645   Test Cost:0.2615077495574951\n",
      "\n",
      " Train Acc:0.9375   Test Acc:0.78125    Train Cost:0.2576085925102234   Test Cost:0.544012725353241\n",
      "\n",
      " Train Acc:0.90625   Test Acc:0.90625    Train Cost:0.35494691133499146   Test Cost:0.23916910588741302\n",
      "\n",
      " Train Acc:0.9375   Test Acc:0.96875    Train Cost:0.1679994910955429   Test Cost:0.06558889895677567\n"
     ]
    }
   ],
   "source": [
    "logits = RNN_model(X,num_classes)\n",
    "predictions = tf.nn.softmax(logits)\n",
    "\n",
    "cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits_v2(labels=Y,logits=logits))\n",
    "optimizer = tf.train.AdamOptimizer().minimize(cost)\n",
    "\n",
    "accuracy = tf.reduce_mean( (tf.cast(tf.equal(tf.argmax(predictions,1), tf.argmax(Y,1)),dtype=tf.float32)) )\n",
    "\n",
    "#saver = tf.train.Saver()\n",
    "\n",
    "init = tf.global_variables_initializer()\n",
    "with tf.Session() as sess:\n",
    "    sess.run(init)\n",
    "    for epoch_cntr in range(num_epochs):\n",
    "        for batch_cntr in range(num_batches):\n",
    "            x_train_batch, y_train_batch = mnist.train.next_batch(batch_size)\n",
    "            x_train_batch = x_train_batch.reshape([-1,28,28])\n",
    "            sess.run(optimizer, feed_dict={X:x_train_batch, Y:y_train_batch})\n",
    "            batch_train_cost,batch_train_acc= sess.run([cost,accuracy], feed_dict={X:x_train_batch, Y:y_train_batch})\n",
    "            \n",
    "            if batch_cntr % 250 == 0:\n",
    "                x_test_batch, y_test_batch = mnist.test.next_batch(batch_size)\n",
    "                x_test_batch = x_test_batch.reshape([-1,28,28])\n",
    "                batch_test_cost,batch_test_acc= sess.run([cost,accuracy], feed_dict={X:x_test_batch, Y:y_test_batch})\n",
    "                \n",
    "                print('\\n Train Acc:{}   Test Acc:{}    Train Cost:{}   Test Cost:{}'.format(batch_train_acc, batch_test_acc, batch_train_cost, batch_test_cost))             "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}