notebooks/403-action-recognition-webcam/403-action-recognition-webcam.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "# Human Action Recognition with OpenVINO\n",
    "\n",
    "This notebook demonstrates live human action recognition with OpenVINO. We use the [Action Recognition Models](https://docs.openvino.ai/2020.2/usergroup13.html) from [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo), specifically an [Encoder](https://docs.openvino.ai/2020.2/_models_intel_action_recognition_0001_encoder_description_action_recognition_0001_encoder.html) and a [Decoder](https://docs.openvino.ai/2020.2/_models_intel_action_recognition_0001_decoder_description_action_recognition_0001_decoder.html). Both models create a sequence to sequence (`\"seq2seq\"`)<sup id=\"a1\">[1](#f1)</sup> system to identify the  human activities for [Kinetics-400 dataset](https://deepmind.com/research/open-source/kinetics). The models use the Video Transformer approach with ResNet34 encoder<sup id=\"a2\">[2](#f2)</sup>. In the notebook we show how to create the following pipeline:\n",
    "\n",
    "\n",
    "<img align='center' src=\"https://user-images.githubusercontent.com/10940214/148401661-477aebcd-f2d0-4771-b107-4b37f94d0b1e.jpeg\" alt=\"drawing\" width=\"1000\"/>\n",
    "\n",
    "\n",
    "At the end of this notebook, you will see live inference results from your webcam. You can also upload a video file.\n",
    "\n",
    "NOTE: _To use the webcam, you must run this Jupyter notebook on a computer with a webcam. If you run on a server, the webcam will not work. However, you can still do inference on a video in the final step._\n",
    "\n",
    "> <b id=\"f1\">1</b> seq2seq: Deep learning models that take a sequence of items to the input and output. In this case, input: video frames, output: actions sequence. This `\"seq2seq\"` is composed of an encoder and a decoder. The encoder captures the `\"context\"` of the inputs to be analyzed by the decoder, and finally gets the human action and confidence.[$\\hookleftarrow$](#a1)\n",
    "\n",
    "> <b id=\"f2\">2</b>  [Video Transformer](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)) and [ResNet34](https://www.kaggle.com/pytorch/resnet34). [$\\hookleftarrow$](#a2)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import collections\n",
    "import os\n",
    "import sys\n",
    "import time\n",
    "from typing import Tuple, List\n",
    "\n",
    "import cv2\n",
    "import numpy as np\n",
    "from IPython import display\n",
    "from openvino.runtime import Core\n",
    "from openvino.runtime.ie_api import CompiledModel\n",
    "\n",
    "sys.path.append(\"../utils\")\n",
    "import notebook_utils as utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The models\n",
    "\n",
    "### Download the models\n",
    "\n",
    "We use `omz_downloader`, which is a command-line tool from the `openvino-dev` package. `omz_downloader` automatically creates a directory structure and downloads the selected model.\n",
    "\n",
    "In this case you can use `\"action-recognition-0001\"` as a model name, and the system automatically downloads the two models `\"action-recognition-0001-encoder\"` and `\"action-recognition-0001-decoder\"`\n",
    "\n",
    "> Note: _If you want to download another model, such as `\"driver-action-recognition-adas-0002\"` (`\"driver-action-recognition-adas-0002-encoder\"` + `\"driver-action-recognition-adas-0002-decoder\"`), please change the model name. Using a model outside the list can require different pre- and post-processing._ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Directory where model will be downloaded\n",
    "base_model_dir = \"model\"\n",
    "# Model name as named in Open Model Zoo\n",
    "model_name = \"action-recognition-0001\"\n",
    "# Selected precision (FP32, FP16, FP16-INT8)\n",
    "precision = \"FP16\"\n",
    "model_path_decoder = (\n",
    "    f\"model/intel/{model_name}/{model_name}-decoder/{precision}/{model_name}-decoder.xml\"\n",
    ")\n",
    "model_path_encoder = (\n",
    "    f\"model/intel/{model_name}/{model_name}-encoder/{precision}/{model_name}-encoder.xml\"\n",
    ")\n",
    "if not os.path.exists(model_path_decoder) or not os.path.exists(model_path_encoder):\n",
    "    download_command = f\"omz_downloader \" \\\n",
    "                       f\"--name {model_name} \" \\\n",
    "                       f\"--precision {precision} \" \\\n",
    "                       f\"--output_dir {base_model_dir}\"\n",
    "    ! $download_command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load your labels\n",
    "\n",
    "We are using [Kinetics-400 dataset](https://deepmind.com/research/open-source/kinetics), and also we are providing you with the text file embedded into this notebook. \n",
    "\n",
    "> Note: _If you want to run `\"driver-action-recognition-adas-0002\"` model, change the txt file for `\"driver_actions.txt\"`_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = \"data/kinetics.txt\"\n",
    "\n",
    "with open(labels) as f:\n",
    "    labels = [line.strip() for line in f]\n",
    "\n",
    "print(labels[0:9], np.shape(labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "### Load the models\n",
    "\n",
    "We will load the two models for this particular architecture, Encoder and Decoder. Downloaded models are located in a fixed structure, indicating vendor, model name, and precision.\n",
    "\n",
    " 1. Initialize inference engine (IECore)\n",
    " 2. Read the network from *.bin and *.xml files (weights and architecture)\n",
    " 3. Compile the model for the \"CPU.\"\n",
    " 4. Get input and output names of nodes.\n",
    "\n",
    "Only a few lines of code are required to run the model. Let's see it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Model Initialization function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize inference engine\n",
    "ie_core = Core()\n",
    "\n",
    "\n",
    "def model_init(model_path: str) -> Tuple:\n",
    "    \"\"\"\n",
    "    Read the network and weights from file, load the\n",
    "    model on the CPU and get input and output names of nodes\n",
    "\n",
    "    :param: model: model architecture path *.xml\n",
    "    :retuns:\n",
    "            compiled_model: Compiled model \n",
    "            input_key: Input node for model\n",
    "            output_key: Output node for model\n",
    "    \"\"\"\n",
    "\n",
    "    # Read the network and corresponding weights from file\n",
    "    model = ie_core.read_model(model=model_path)\n",
    "    # compile the model for the CPU (you can use GPU or MYRIAD as well)\n",
    "    compiled_model = ie_core.compile_model(model=model, device_name=\"CPU\")\n",
    "    # Get input and output names of nodes\n",
    "    input_keys = compiled_model.input(0)\n",
    "    output_keys = compiled_model.output(0)\n",
    "    return input_keys, output_keys, compiled_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Initialization for Encoder and Decoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Encoder initialization\n",
    "input_key_en, output_keys_en, compiled_model_en = model_init(model_path_encoder)\n",
    "# Decoder initialization\n",
    "input_key_de, output_keys_de, compiled_model_de = model_init(model_path_decoder)\n",
    "\n",
    "# Get input size - Encoder\n",
    "height_en, width_en = list(input_key_en.shape)[2:]\n",
    "# Get input size - Decoder\n",
    "frames2decode = list(input_key_de.shape)[0:][1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper functions\n",
    "\n",
    "We will use the following helper functions for preprocessing and postprocessing frames:\n",
    "\n",
    "1. Preprocess the input image before running the Encoder model. (`center_crop` and `adaptative_resize`)\n",
    "2. Decode top-3 probabilities into label names. (`decode_output`)\n",
    "3. Draw the Region of Interest (ROI) over the video. (`rec_frame_display`)\n",
    "4. Prepare the frame for displaying label names over the video. (`display_text_fnc`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def center_crop(frame: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Center crop squared the original frame to standardize the input image to the encoder model\n",
    "\n",
    "    :param frame: input frame\n",
    "    :returns: center-crop-squared frame\n",
    "    \"\"\"\n",
    "    img_h, img_w, _ = frame.shape\n",
    "    min_dim = min(img_h, img_w)\n",
    "    start_x = int((img_w - min_dim) / 2.0)\n",
    "    start_y = int((img_h - min_dim) / 2.0)\n",
    "    roi = [start_y, (start_y + min_dim), start_x, (start_x + min_dim)]\n",
    "    return frame[start_y : (start_y + min_dim), start_x : (start_x + min_dim), ...], roi\n",
    "\n",
    "\n",
    "def adaptive_resize(frame: np.ndarray, size: int) -> np.ndarray:\n",
    "    \"\"\"\n",
    "     The frame going to be resized to have a height of size or a width of size\n",
    "\n",
    "    :param frame: input frame\n",
    "    :param size: input size to encoder model\n",
    "    :returns: resized frame, np.array type\n",
    "    \"\"\"\n",
    "    h, w, _ = frame.shape\n",
    "    scale = size / min(h, w)\n",
    "    w_scaled, h_scaled = int(w * scale), int(h * scale)\n",
    "    if w_scaled == w and h_scaled == h:\n",
    "        return frame\n",
    "    return cv2.resize(frame, (w_scaled, h_scaled))\n",
    "\n",
    "\n",
    "def decode_output(probs: np.ndarray, labels: np.ndarray, top_k: int = 3) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Decodes top probabilities into corresponding label names\n",
    "\n",
    "    :param probs: confidence vector for 400 actions\n",
    "    :param labels: list of actions\n",
    "    :param top_k: The k most probable positions in the list of labels\n",
    "    :returns: decoded_labels: The k most probable actions from the labels list\n",
    "              decoded_top_probs: confidence for the k most probable actions\n",
    "    \"\"\"\n",
    "    top_ind = np.argsort(-1 * probs)[:top_k]\n",
    "    out_label = np.array(labels)[top_ind.astype(int)]\n",
    "    decoded_labels = [out_label[0][0], out_label[0][1], out_label[0][2]]\n",
    "    top_probs = np.array(probs)[0][top_ind.astype(int)]\n",
    "    decoded_top_probs = [top_probs[0][0], top_probs[0][1], top_probs[0][2]]\n",
    "    return decoded_labels, decoded_top_probs\n",
    "\n",
    "\n",
    "def rec_frame_display(frame: np.ndarray, roi) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Draw a rec frame over actual frame\n",
    "\n",
    "    :param frame: input frame\n",
    "    :param roi: Region of interest, image section processed by the Encoder\n",
    "    :returns: frame with drawed shape\n",
    "\n",
    "    \"\"\"\n",
    "\n",
    "    cv2.line(frame, (roi[2] + 3, roi[0] + 3), (roi[2] + 3, roi[0] + 100), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[2] + 3, roi[0] + 3), (roi[2] + 100, roi[0] + 3), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[3] - 3, roi[1] - 3), (roi[3] - 3, roi[1] - 100), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[3] - 3, roi[1] - 3), (roi[3] - 100, roi[1] - 3), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[3] - 3, roi[0] + 3), (roi[3] - 3, roi[0] + 100), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[3] - 3, roi[0] + 3), (roi[3] - 100, roi[0] + 3), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[2] + 3, roi[1] - 3), (roi[2] + 3, roi[1] - 100), (0, 200, 0), 2)\n",
    "    cv2.line(frame, (roi[2] + 3, roi[1] - 3), (roi[2] + 100, roi[1] - 3), (0, 200, 0), 2)\n",
    "    # Write ROI over actual frame\n",
    "    FONT_STYLE = cv2.FONT_HERSHEY_SIMPLEX\n",
    "    org = (roi[2] + 3, roi[1] - 3)\n",
    "    org2 = (roi[2] + 2, roi[1] - 2)\n",
    "    FONT_SIZE = 0.5\n",
    "    FONT_COLOR = (0, 200, 0)\n",
    "    FONT_COLOR2 = (0, 0, 0)\n",
    "    cv2.putText(frame, \"ROI\", org2, FONT_STYLE, FONT_SIZE, FONT_COLOR2)\n",
    "    cv2.putText(frame, \"ROI\", org, FONT_STYLE, FONT_SIZE, FONT_COLOR)\n",
    "    return frame\n",
    "\n",
    "\n",
    "def display_text_fnc(frame: np.ndarray, display_text: str, index: int):\n",
    "    \"\"\"\n",
    "    Include text on the analized frame\n",
    "\n",
    "    :param frame: input frame\n",
    "    :param display_text: text to add on the frame\n",
    "    :param index: index line dor adding text\n",
    "\n",
    "    \"\"\"\n",
    "    # Configuration for displaying images with text\n",
    "    FONT_COLOR = (255, 255, 255)\n",
    "    FONT_COLOR2 = (0, 0, 0)\n",
    "    FONT_STYLE = cv2.FONT_HERSHEY_DUPLEX\n",
    "    FONT_SIZE = 0.7\n",
    "    TEXT_VERTICAL_INTERVAL = 25\n",
    "    TEXT_LEFT_MARGIN = 15\n",
    "    # ROI over actual frame\n",
    "    (processed, roi) = center_crop(frame)\n",
    "    # Draw a ROI over actual frame\n",
    "    frame = rec_frame_display(frame, roi)\n",
    "    # Put text over actual frame\n",
    "    text_loc = (TEXT_LEFT_MARGIN, TEXT_VERTICAL_INTERVAL * (index + 1))\n",
    "    text_loc2 = (TEXT_LEFT_MARGIN + 1, TEXT_VERTICAL_INTERVAL * (index + 1) + 1)\n",
    "    cv2.putText(frame, display_text, text_loc2, FONT_STYLE, FONT_SIZE, FONT_COLOR2)\n",
    "    cv2.putText(frame, display_text, text_loc, FONT_STYLE, FONT_SIZE, FONT_COLOR)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### AI Functions\n",
    "\n",
    "Following this pipeline, \n",
    "\n",
    "<img align='center' src=\"https://user-images.githubusercontent.com/10940214/148401661-477aebcd-f2d0-4771-b107-4b37f94d0b1e.jpeg\" alt=\"drawing\" width=\"1000\"/>\n",
    "\n",
    "\n",
    "we will use the next functions to help us to:\n",
    "\n",
    "1. Preprocess frame before running the Encoder. (`preprocessing`)\n",
    "2. Encoder Inference per frame. (`encoder`)\n",
    "3. Decoder inference per set of frames. (`decoder`)\n",
    "4. Normalize the Decoder output to get confidence values per action recognition label. (`softmax`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocessing(frame: np.ndarray, size: int) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Preparing frame before Encoder.\n",
    "    The image should be scaled to its shortest dimension at \"size\"\n",
    "    and cropped, centered, and squared so that both width and\n",
    "    height have lengths \"size\". Frame must be transposed from\n",
    "    Height-Width-Channels (HWC) to Channels-Height-Width (CHW).\n",
    "\n",
    "    :param frame: input frame\n",
    "    :param size: input size to encoder model\n",
    "    :returns: resized and cropped frame\n",
    "    \"\"\"\n",
    "    # Adaptative resize\n",
    "    preprocessed = adaptive_resize(frame, size)\n",
    "    # Center_crop\n",
    "    (preprocessed, roi) = center_crop(preprocessed)\n",
    "    # Transpose frame HWC -> CHW\n",
    "    preprocessed = preprocessed.transpose((2, 0, 1))[None,]  # HWC -> CHW\n",
    "    return preprocessed, roi\n",
    "\n",
    "\n",
    "def encoder(\n",
    "    preprocessed: np.ndarray,\n",
    "    compiled_model: CompiledModel\n",
    ") -> List:\n",
    "    \"\"\"\n",
    "    Encoder Inference per frame. This function calls the network previously\n",
    "    configured for the encoder model (compiled_model), extracts the data\n",
    "    from the output node, and appends it in an array to be used by the decoder.\n",
    "\n",
    "    :param: preprocessed: preprocessing frame\n",
    "    :param: compiled_model: Encoder model network\n",
    "    :returns: encoder_output: embedding layer that is appended with each arriving frame\n",
    "    \"\"\"\n",
    "    output_key_en = compiled_model.output(0)\n",
    "    \n",
    "    # Get results on action-recognition-0001-encoder model\n",
    "    infer_result_encoder = compiled_model([preprocessed])[output_key_en]\n",
    "    return infer_result_encoder\n",
    "\n",
    "\n",
    "def decoder(encoder_output: List, compiled_model_de: CompiledModel) -> List:\n",
    "    \"\"\"\n",
    "    Decoder inference per set of frames. This function concatenates the embedding layer\n",
    "    froms the encoder output, transpose the array to match with the decoder input size.\n",
    "    Calls the network previously configured for the decoder model (compiled_model_de), extracts\n",
    "    the logits and normalize those to get confidence values along specified axis.\n",
    "    Decodes top probabilities into corresponding label names\n",
    "\n",
    "    :param: encoder_output: embedding layer for 16 frames\n",
    "    :param: compiled_model_de: Decoder model network\n",
    "    :returns: decoded_labels: The k most probable actions from the labels list\n",
    "              decoded_top_probs: confidence for the k most probable actions\n",
    "    \"\"\"\n",
    "    # Concatenate sample_duration frames in just one array\n",
    "    decoder_input = np.concatenate(encoder_output, axis=0)\n",
    "    # Organize input shape vector to the Decoder (shape: [1x16x512]]\n",
    "    decoder_input = decoder_input.transpose((2, 0, 1, 3))\n",
    "    decoder_input = np.squeeze(decoder_input, axis=3)\n",
    "    output_key_de = compiled_model_de.output(0)\n",
    "    # Get results on action-recognition-0001-decoder model\n",
    "    result_de = compiled_model_de([decoder_input])[output_key_de]\n",
    "    # Normalize logits to get confidence values along specified axis\n",
    "    probs = softmax(result_de - np.max(result_de))\n",
    "    # Decodes top probabilities into corresponding label names\n",
    "    decoded_labels, decoded_top_probs = decode_output(probs, labels, top_k=3)\n",
    "    return decoded_labels, decoded_top_probs\n",
    "\n",
    "\n",
    "def softmax(x: np.ndarray) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    Normalizes logits to get confidence values along specified axis\n",
    "    x: np.array, axis=None\n",
    "    \"\"\"\n",
    "    exp = np.exp(x)\n",
    "    return exp / np.sum(exp, axis=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    },
    "tags": []
   },
   "source": [
    "### Main Processing Function\n",
    "\n",
    "Run action recognition function will run in different operations, either a webcam or a video file. See the list of procedures below:\n",
    "\n",
    "1. Create a video player to play with target fps (`utils.VideoPlayer`).\n",
    "2. Prepare a set of frames to be encoded-decoded.\n",
    "3. Run AI functions\n",
    "4. Visualize the results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def run_action_recognition(\n",
    "    source: str = \"0\",\n",
    "    flip: bool = True,\n",
    "    use_popup: bool = False,\n",
    "    compiled_model_en: CompiledModel = compiled_model_en,\n",
    "    compiled_model_de: CompiledModel = compiled_model_de,\n",
    "    skip_first_frames: int = 0,\n",
    "):\n",
    "    \"\"\"\n",
    "    Use the \"source\" webcam or video file to run the complete pipeline for action-recognition problem\n",
    "    1. Create a video player to play with target fps\n",
    "    2. Prepare a set of frames to be encoded-decoded\n",
    "    3. Preprocess frame before Encoder\n",
    "    4. Encoder Inference per frame\n",
    "    5. Decoder inference per set of frames\n",
    "    6. Visualize the results\n",
    "\n",
    "    :param: source: webcam \"0\" or video path\n",
    "    :param: flip: to be used by VideoPlayer function for flipping capture image\n",
    "    :param: use_popup: False for showing encoded frames over this notebook, True for creating a popup window.\n",
    "    :param: skip_first_frames: Number of frames to skip at the beginning of the video.\n",
    "    :returns: display video over the notebook or in a popup window\n",
    "\n",
    "    \"\"\"\n",
    "    size = height_en  # Endoder input size - From Cell 5_9\n",
    "    sample_duration = frames2decode  # Decoder input size - From Cell 5_7\n",
    "    # Select frames per second of your source\n",
    "    fps = 30\n",
    "    player = None\n",
    "    try:\n",
    "        # Create a video player\n",
    "        player = utils.VideoPlayer(source, flip=flip, fps=fps, skip_first_frames=skip_first_frames)\n",
    "        # Start capturing\n",
    "        player.start()\n",
    "        if use_popup:\n",
    "            title = \"Press ESC to Exit\"\n",
    "            cv2.namedWindow(title, cv2.WINDOW_GUI_NORMAL | cv2.WINDOW_AUTOSIZE)\n",
    "\n",
    "        processing_times = collections.deque()\n",
    "        processing_time = 0\n",
    "        encoder_output = []\n",
    "        decoded_labels = [0, 0, 0]\n",
    "        decoded_top_probs = [0, 0, 0]\n",
    "        counter = 0\n",
    "        # Create a text template to show inference results over video\n",
    "        text_inference_template = \"Infer Time:{Time:.1f}ms,{fps:.1f}FPS\"\n",
    "        text_template = \"{label},{conf:.2f}%\"\n",
    "\n",
    "        while True:\n",
    "            counter = counter + 1\n",
    "\n",
    "            # read a frame from the video stream\n",
    "            frame = player.next()\n",
    "            if frame is None:\n",
    "                print(\"Source ended\")\n",
    "                break\n",
    "\n",
    "            scale = 1280 / max(frame.shape)\n",
    "\n",
    "            # Adaptative resize for visualization\n",
    "            if scale < 1:\n",
    "                frame = cv2.resize(frame, None, fx=scale, fy=scale, interpolation=cv2.INTER_AREA)\n",
    "\n",
    "            # Select one frame every two for processing through the encoder.\n",
    "            # After 16 frames are processed, the decoder will find the action,\n",
    "            # and the label will be printed over the frames.\n",
    "\n",
    "            if counter % 2 == 0:\n",
    "                # Preprocess frame before Encoder\n",
    "                (preprocessed, _) = preprocessing(frame, size)\n",
    "\n",
    "                # Measure processing time\n",
    "                start_time = time.time()\n",
    "\n",
    "                # Encoder Inference per frame\n",
    "                encoder_output.append(encoder(preprocessed, compiled_model_en))\n",
    "\n",
    "                # Decoder inference per set of frames\n",
    "                # Wait for sample duration to work with decoder model\n",
    "                if len(encoder_output) == sample_duration:\n",
    "                    decoded_labels, decoded_top_probs = decoder(encoder_output, compiled_model_de)\n",
    "                    encoder_output = []\n",
    "\n",
    "                # Inference has finished ... Let's to display results\n",
    "                stop_time = time.time()\n",
    "\n",
    "                # Calculate processing time\n",
    "                processing_times.append(stop_time - start_time)\n",
    "\n",
    "                # Use processing times from last 200 frames\n",
    "                if len(processing_times) > 200:\n",
    "                    processing_times.popleft()\n",
    "\n",
    "                # Mean processing time [ms]\n",
    "                processing_time = np.mean(processing_times) * 1000\n",
    "                fps = 1000 / processing_time\n",
    "\n",
    "            # Visualize the results\n",
    "            for i in range(0, 3):\n",
    "                display_text = text_template.format(\n",
    "                    label=decoded_labels[i],\n",
    "                    conf=decoded_top_probs[i] * 100,\n",
    "                )\n",
    "                display_text_fnc(frame, display_text, i)\n",
    "\n",
    "            display_text = text_inference_template.format(Time=processing_time, fps=fps)\n",
    "            display_text_fnc(frame, display_text, 3)\n",
    "\n",
    "            # Use this workaround you experience flickering\n",
    "            if use_popup:\n",
    "                cv2.imshow(title, frame)\n",
    "                key = cv2.waitKey(1)\n",
    "                # escape = 27\n",
    "                if key == 27:\n",
    "                    break\n",
    "            else:\n",
    "                # Encode numpy array to jpg\n",
    "                _, encoded_img = cv2.imencode(\".jpg\", frame, params=[cv2.IMWRITE_JPEG_QUALITY, 90])\n",
    "                # Create IPython image\n",
    "                i = display.Image(data=encoded_img)\n",
    "                # Display the image in this notebook\n",
    "                display.clear_output(wait=True)\n",
    "                display.display(i)\n",
    "\n",
    "    # ctrl-c\n",
    "    except KeyboardInterrupt:\n",
    "        print(\"Interrupted\")\n",
    "    # Any different error\n",
    "    except RuntimeError as e:\n",
    "        print(e)\n",
    "    finally:\n",
    "        if player is not None:\n",
    "            # stop capturing\n",
    "            player.stop()\n",
    "        if use_popup:\n",
    "            cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Action Recognition on a Video File\n",
    "\n",
    "Now let's see how the model works in a video file. [Any format supported](https://docs.opencv.org/4.5.1/dd/d43/tutorial_py_video_display.html) by OpenCV will work. Anytime you can press the stop button while the video file is running, it will activate the webcam for the next step.\n",
    "\n",
    "> Note: _Sometimes, the video could be cut off if there are corrupted frames. In that case you could convert. If you experience any problems with your video, please use the [HandBrake](https://handbrake.fr/) and select the MPEG format._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [],
    "test_replace": {
     "skip_first_frames=600": "skip_first_frames=18000"
    }
   },
   "outputs": [],
   "source": [
    "video_file = \"https://archive.org/serve/ISSVideoResourceLifeOnStation720p/ISS%20Video%20Resource_LifeOnStation_720p.mp4\"\n",
    "run_action_recognition(source=video_file, flip=False, use_popup=False, skip_first_frames=600)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run Action Recognition using your webcam\n",
    "\n",
    "Now let's try to see ourselves in our webcam.\n",
    "\n",
    "> NOTE: _To use the webcam, you must run this Jupyter notebook on a computer with a webcam. If you run on a server, the webcam will not work. However, you can still do inference on a video in the final step._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "run_action_recognition(source=0, flip=False, use_popup=False, skip_first_frames=0)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Edit Metadata",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}