SubmissionExample.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## End to End Mlperf Submission example\n",
    "\n",
    "This is following the [General MLPerf Submission Rules](https://github.com/mlperf/policies/blob/master/submission_rules.adoc).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get the mlperf source code\n",
    "\n",
    "You run this notebook from the root of the mlperf inference tree that you cloned with\n",
    "```\n",
    "git clone https://github.com/mlperf/inference.git --depth 1\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build loadgen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# build loadgen\n",
    "!pip install pybind11\n",
    "!cd loadgen; CFLAGS=\"-std=c++14 -O3\" python setup.py develop"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!cd vision/classification_and_detection; python setup.py develop"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set Working Directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/gs/inference/v0.5/classification_and_detection\n"
     ]
    }
   ],
   "source": [
    "%cd vision/classification_and_detection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download data\n",
    "\n",
    "We need to download imagenet and/or coco for the benchmark. \n",
    "In our setup we keep our data in /data and can symlink to /data. \n",
    "You might need to change this to the location in your setup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "mkdir data\n",
    "ln -s  /data/imagenet2012 data/\n",
    "ln -s  /data/coco data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "mkdir models\n",
    "\n",
    "# resnet50\n",
    "wget -q https://zenodo.org/record/2535873/files/resnet50_v1.pb -O models/resnet50_v1.pb \n",
    "wget -q https://zenodo.org/record/2592612/files/resnet50_v1.onnx -O models/resnet50_v1.onnx\n",
    "\n",
    "# mobilenet\n",
    "wget -q https://zenodo.org/record/2269307/files/mobilenet_v1_1.0_224.tgz -O models/mobilenet_v1_1.0_224.tgz\n",
    "cd models; tar zxvf mobilenet_v1_1.0_224.tgz\n",
    "wget -q https://zenodo.org/record/3157894/files/mobilenet_v1_1.0_224.onnx -O models/mobilenet_v1_1.0_224.onnx\n",
    "\n",
    "# ssd-mobilenet\n",
    "wget -q http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz -O models/ssd_mobilenet_v1_coco_2018_01_28.tar.gz\n",
    "cd models; tar zxvf ssd_mobilenet_v1_coco_2018_01_28.tar.gz; mv ssd_mobilenet_v1_coco_2018_01_28/frozen_inference_graph.pb ssd_mobilenet_v1_coco_2018_01_28.pb\n",
    "wget -q https://zenodo.org/record/3163026/files/ssd_mobilenet_v1_coco_2018_01_28.onnx -O models/ssd_mobilenet_v1_coco_2018_01_28.onnx \n",
    "\n",
    "# ssd-resnet34\n",
    "wget -q https://zenodo.org/record/3345892/files/tf_ssd_resnet34_22.1.zip -O models/tf_ssd_resnet34_22.1.zip\n",
    "cd models; unzip tf_ssd_resnet34_22.1.zip; mv tf_ssd_resnet34_22.1/resnet34_tf.22.1.pb .\n",
    "wget -q https://zenodo.org/record/3228411/files/resnet34-ssd1200.onnx -O models/resnet34-ssd1200.onnx"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run benchmarks using the reference implementation\n",
    "\n",
    "Lets prepare a submission for mobilenet on a desktop machine with a NVidia gtx-1080 gpu using tensorflow. \n",
    "\n",
    "The following script will run those combinations and prepare a submission directory, following the general submission rules documented [here](https://github.com/mlperf/policies/blob/master/submission_rules.adoc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "import os\n",
    "logger = logging.getLogger()\n",
    "logger.setLevel(logging.CRITICAL)\n",
    "\n",
    "os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3' \n",
    "os.environ['CUDA_VISIBLE_DEVICES'] = \"0\"\n",
    "\n",
    "# final results go here\n",
    "ORG = \"mlperf-org\"\n",
    "DIVISION = \"closed\"\n",
    "SUBMISSION_ROOT = \"/tmp/mlperf-submission\"\n",
    "SUBMISSION_DIR = os.path.join(SUBMISSION_ROOT, DIVISION, ORG)\n",
    "os.environ['SUBMISSION_ROOT'] = SUBMISSION_ROOT\n",
    "os.environ['SUBMISSION_DIR'] = SUBMISSION_DIR\n",
    "os.makedirs(SUBMISSION_DIR, exist_ok=True)\n",
    "os.makedirs(os.path.join(SUBMISSION_DIR, \"measurements\"), exist_ok=True)\n",
    "os.makedirs(os.path.join(SUBMISSION_DIR, \"code\"), exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "# where to find stuff\n",
    "export DATA_ROOT=`pwd`/data\n",
    "export MODEL_DIR=`pwd`/models\n",
    "\n",
    "# options for official runs\n",
    "gopt=\"--max-batchsize 8 --samples-per-query 40 --threads 2 --qps 145\"\n",
    "\n",
    "\n",
    "function one_run {\n",
    "    # args: mode count framework device model ...\n",
    "    scenario=$1; shift\n",
    "    count=$1; shift\n",
    "    framework=$1\n",
    "    device=$2\n",
    "    model=$3\n",
    "    system_id=$framework-$device\n",
    "    echo \"====== $model/$scenario =====\"\n",
    "\n",
    "    case $model in \n",
    "    mobilenet)\n",
    "        cmd=\"tools/accuracy-imagenet.py --imagenet-val-file $DATA_ROOT/imagenet2012/val_map.txt\"\n",
    "        offical_name=\"mobilenet\";;\n",
    "    resnet50) \n",
    "        cmd=\"tools/accuracy-imagenet.py --imagenet-val-file $DATA_ROOT/imagenet2012/val_map.txt\"\n",
    "        offical_name=\"resnet\";;\n",
    "    ssd-mobilenet) \n",
    "        cmd=\"tools/accuracy-coco.py --coco-dir $DATA_ROOT/coco\"\n",
    "        offical_name=\"ssd-small\";;\n",
    "    ssd-resnet34) \n",
    "        cmd=\"tools/accuracy-coco.py --coco-dir $DATA_ROOT/coco\"\n",
    "        offical_name=\"ssd-large\";;\n",
    "    esac\n",
    "    output_dir=$SUBMISSION_DIR/results/$system_id/$offical_name\n",
    "    \n",
    "    # accuracy run\n",
    "    ./run_local.sh $@ --scenario $scenario --accuracy --output $output_dir/$scenario/accuracy\n",
    "    python $cmd --mlperf-accuracy-file $output_dir/$scenario/accuracy/mlperf_log_accuracy.json \\\n",
    "            >  $output_dir/$scenario/accuracy/accuracy.txt\n",
    "    cat $output_dir/$scenario/accuracy/accuracy.txt\n",
    "\n",
    "    # performance run\n",
    "    cnt=0\n",
    "    while [ $cnt -le $count ]; do\n",
    "        let cnt=cnt+1\n",
    "        ./run_local.sh $@ --scenario $scenario --output $output_dir/$scenario/performance/$cnt\n",
    "    done\n",
    "    \n",
    "    # setup the measurements directory\n",
    "    mdir=$SUBMISSION_DIR/measurements/$system_id/$offical_name/$scenario\n",
    "    mkdir -p $mdir\n",
    "    cp ../mlperf.conf $mdir\n",
    "\n",
    "    # reference app uses command line instead of user.conf\n",
    "    echo \"# empty\" > $mdir/user.conf\n",
    "    touch $mdir/README.md\n",
    "    impid=\"reference\"\n",
    "    cat > $mdir/$system_id\"_\"$impid\"_\"$scenario\".json\" <<EOF\n",
    "    {\n",
    "        \"input_data_types\": \"fp32\",\n",
    "        \"retraining\": \"none\",\n",
    "        \"starting_weights_filename\": \"https://zenodo.org/record/2269307/files/mobilenet_v1_1.0_224.tgz\",\n",
    "        \"weight_data_types\": \"fp32\",\n",
    "        \"weight_transformations\": \"none\"\n",
    "    }\n",
    "    EOF\n",
    "}\n",
    "\n",
    "function one_model {\n",
    "    # args: framework device model ...\n",
    "    one_run SingleStream 1 $@ --max-latency 0.0005\n",
    "    one_run Server 5 $@\n",
    "    one_run Offline 1 $@ --qps 1000\n",
    "    one_run MultiStream 1 $@\n",
    "}\n",
    "\n",
    "\n",
    "# run image classifier benchmarks \n",
    "export DATA_DIR=$DATA_ROOT/imagenet2012\n",
    "one_model tf gpu mobilenet $gopt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There might be large trace files in the submission directory - we can delete them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "!find {SUBMISSION_DIR}/ -name mlperf_log_trace.json -delete"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complete submission directory\n",
    "\n",
    "Add the required meta data to the submission.\n",
    "A template for the meta fields can be found [here](https://github.com/mlperf/inference/tree/master/v0.5/tools/submission/meta.json)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "#\n",
    "# setup systems directory\n",
    "#\n",
    "if [ ! -d ${SUBMISSION_DIR}/systems ]; then\n",
    "    mkdir ${SUBMISSION_DIR}/systems\n",
    "fi\n",
    "\n",
    "cat > ${SUBMISSION_DIR}/systems/tf-gpu.json <<EOF\n",
    "{\n",
    "        \"division\": \"closed\",\n",
    "        \"status\": \"available\",\n",
    "        \"submitter\": \"mlperf-org\",\n",
    "        \"system_name\": \"tf-gpu\",\n",
    "        \n",
    "        \"number_of_nodes\": 1,\n",
    "        \"host_memory_capacity\": \"32GB\",\n",
    "        \"host_processor_core_count\": 1,\n",
    "        \"host_processor_frequency\": \"3.50GHz\",\n",
    "        \"host_processor_model_name\": \"Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz\",\n",
    "        \"host_processors_per_node\": 1,\n",
    "        \"host_storage_capacity\": \"512GB\",\n",
    "        \"host_storage_type\": \"SSD\",\n",
    "        \n",
    "        \"accelerator_frequency\": \"-\",\n",
    "        \"accelerator_host_interconnect\": \"-\",\n",
    "        \"accelerator_interconnect\": \"-\",\n",
    "        \"accelerator_interconnect_topology\": \"-\",\n",
    "        \"accelerator_memory_capacity\": \"8GB\",\n",
    "        \"accelerator_memory_configuration\": \"none\",\n",
    "        \"accelerator_model_name\": \"gtx-1080\",\n",
    "        \"accelerator_on-chip_memories\": \"-\",\n",
    "        \"accelerators_per_node\": 1,\n",
    "\n",
    "        \"framework\": \"v1.14.0-rc1-22-gaf24dc9\",\n",
    "        \"operating_system\": \"ubuntu-16.04\",\n",
    "        \"other_software_stack\": \"cuda-10.1\",\n",
    "        \"sw_notes\": \"\"\n",
    "}\n",
    "EOF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "#\n",
    "# setup code directory\n",
    "#\n",
    "dir=${SUBMISSION_DIR}/code/mobilenet/reference\n",
    "mkdir -p $dir\n",
    "echo \"git clone https://github.com/mlperf/inference.git\" > $dir/VERSION.txt\n",
    "git rev-parse HEAD >> $dir/VERSION.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What's in the submission directory now ?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/tmp/mlperf-submission/closed/mlperf-org/systems/tf-gpu.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/Offline/tf-gpu_reference.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/Offline/README.md\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/Offline/mlperf.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/Offline/user.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/Server/tf-gpu_reference.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/Server/README.md\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/Server/mlperf.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/Server/user.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/SingleStream/tf-gpu_reference.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/SingleStream/README.md\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/SingleStream/mlperf.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/SingleStream/user.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/MultiStream/tf-gpu_reference.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/MultiStream/README.md\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/MultiStream/mlperf.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/measurements/tf-gpu/mobilenet/MultiStream/user.conf\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/code/mobilenet/reference/VERSION.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Offline/accuracy/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Offline/accuracy/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Offline/accuracy/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Offline/accuracy/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Offline/accuracy/accuracy.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Offline/performance/run_1/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Offline/performance/run_1/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Offline/performance/run_1/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Offline/performance/run_1/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/accuracy/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/accuracy/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/accuracy/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/accuracy/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/accuracy/accuracy.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_1/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_1/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_1/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_1/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_2/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_2/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_2/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_2/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_3/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_3/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_3/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_3/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_4/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_4/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_4/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_4/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_5/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_5/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_5/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/Server/performance/run_5/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream/accuracy/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream/accuracy/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream/accuracy/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream/accuracy/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream/accuracy/accuracy.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream/performance/run_1/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream/performance/run_1/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream/performance/run_1/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream/performance/run_1/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/accuracy/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/accuracy/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/accuracy/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/accuracy/results.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/accuracy/accuracy.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/performance/run_1/mlperf_log_accuracy.json\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/performance/run_1/mlperf_log_summary.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/performance/run_1/mlperf_log_detail.txt\r\n",
      "/tmp/mlperf-submission/closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream/performance/run_1/results.json\r\n"
     ]
    }
   ],
   "source": [
    "!find {SUBMISSION_ROOT}/ -type f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look at some files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-- SingleStream Accuracy\n",
      "accuracy=71.676%, good=35838, total=50000\n",
      "\n",
      "-- SingleStream Summary\n",
      "================================================\n",
      "MLPerf Results Summary\n",
      "================================================\n",
      "SUT name : PySUT\n",
      "Scenario : Single Stream\n",
      "Mode     : Performance\n",
      "90th percentile latency (ns) : 3405359\n",
      "Result is : VALID\n",
      "  Min duration satisfied : Yes\n",
      "  Min queries satisfied : Yes\n",
      "\n",
      "-- Server Summary\n",
      "================================================\n",
      "MLPerf Results Summary\n",
      "================================================\n",
      "SUT name : PySUT\n",
      "Scenario : Server\n",
      "Mode     : Performance\n",
      "Scheduled samples per second : 145.35\n",
      "Result is : VALID\n",
      "  Performance constraints satisfied : Yes\n",
      "  Min duration satisfied : Yes\n"
     ]
    }
   ],
   "source": [
    "!echo \"-- SingleStream Accuracy\"; head {SUBMISSION_DIR}/results/tf-gpu/mobilenet/SingleStream/accuracy/accuracy.txt\n",
    "!echo \"\\n-- SingleStream Summary\"; head {SUBMISSION_DIR}/results/tf-gpu/mobilenet/SingleStream/performance/run_1/mlperf_log_summary.txt\n",
    "!echo \"\\n-- Server Summary\"; head {SUBMISSION_DIR}/results/tf-gpu/mobilenet/Server/performance/run_1/mlperf_log_summary.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run the submission checker\n",
    "\n",
    "Finally, run the submission checker tool that does some sanity checking on your submission.\n",
    "We run it at the end and attach the output to the submission."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:main:closed/mlperf-org/results/tf-gpu/mobilenet/Offline/performance/run_1/mlperf_log_detail.txt contains errors\r\n",
      "INFO:main:closed/mlperf-org/results/tf-gpu/mobilenet/Offline OK\r\n",
      "INFO:main:closed/mlperf-org/results/tf-gpu/mobilenet/Server OK\r\n",
      "INFO:main:closed/mlperf-org/results/tf-gpu/mobilenet/SingleStream OK\r\n",
      "INFO:main:closed/mlperf-org/results/tf-gpu/mobilenet/MultiStream OK\r\n",
      "INFO:main:closed/mlperf-org/systems/tf-gpu.json OK\r\n",
      "INFO:main:closed/mlperf-org/measurements/tf-gpu/mobilenet/Offline OK\r\n",
      "INFO:main:closed/mlperf-org/measurements/tf-gpu/mobilenet/Server OK\r\n",
      "INFO:main:closed/mlperf-org/measurements/tf-gpu/mobilenet/SingleStream OK\r\n",
      "INFO:main:closed/mlperf-org/measurements/tf-gpu/mobilenet/MultiStream OK\r\n",
      "INFO:main:SUMMARY: submission looks OK\r\n"
     ]
    }
   ],
   "source": [
    "!python ../tools/submission/submission-checker.py --input {SUBMISSION_ROOT} > {SUBMISSION_DIR}/submission-checker.log 2>&1 \n",
    "!cat {SUBMISSION_DIR}/submission-checker.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}