Merge pull request #12 from microsoft/dev/setup-refactor

Refine setup process 2
microsoft · Aug 4, 2021 · 8d6d7f0 · 8d6d7f0
2 parents 50537db + f9237fb
commit 8d6d7f0
Show file tree

Hide file tree

Showing 26 changed files with 760 additions and 364 deletions.
diff --git a/.github/workflows/integration-test.yml b/.github/workflows/integration-test.yml
@@ -36,7 +36,10 @@ jobs:
 
     - name: Install dependencies
     #  if: steps.cache.outputs.cache-hit != 'true'
-      run: pip install -r tests/requirements.txt 
+      run: |
+        pip install tensorflow==1.15.0
+        pip install onnx==1.9.0
+        pip install torch==1.9.0
         
     - name: Install nn-Meter
       run: pip install .

diff --git a/.github/workflows/integration-test_torch.yml b/.github/workflows/integration-test_torch.yml
@@ -0,0 +1,51 @@
+name: Integrated Test for Torch Model
+
+on: [push]
+
+jobs:
+  build-linux:
+    runs-on: ubuntu-latest
+    strategy:
+      max-parallel: 5
+
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Set up Python 3.6
+      uses: actions/setup-python@v2
+      with:
+        python-version: 3.6.10
+
+    # # to cache and restore the download data and python packages.
+    # # still have bug (Cache not found for input keys)
+    # # leave the issue to another PR
+    # - uses: actions/cache@v2
+    #   id: cache
+    #   with:
+    #     path: |
+    #       /home/runner/work/nn-Meter/data/testmodels
+    #       /home/runner/.nn_meter/data
+    #       # /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6
+    #     key: ${{hashFiles('setup.py')}}-${{hashFiles('tests/requirements.txt') }}
+    #     # location remider:
+    #     # pwd: /home/runner/work/nn-Meter/nn-Meter
+    #     # python /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages
+    #     # pythonLocation: /opt/hostedtoolcache/Python/3.6.10/x64
+    #     # LD_LIBRARY_PATH: /opt/hostedtoolcache/Python/3.6.10/x64/lib
+    #     # package path: /opt/hostedtoolcache/Python/3.6.10/x64/lib/python3.6/site-packages
+
+    - name: Install dependencies
+    #  if: steps.cache.outputs.cache-hit != 'true'
+      run: |
+        pip install onnx==1.9.0
+        pip install torch==1.9.0
+        pip install torchvision==0.10.0
+        
+    - name: Install nn-Meter
+      run: pip install .
+
+    - name: Integration test
+      run: python tests/integration_test_torch.py
+
+    - name: Diff result with reference
+      run: diff tests/test_result_torch.txt tests/reference_result_torch.txt
diff --git a/README.md b/README.md
@@ -19,44 +19,39 @@ The current supported hardware and inference frameworks:
 - Those who want to run **hardware-aware NAS with [NNI](https://github.com/microsoft/nni)**.
 - Those who want to **build latency predictors for their own devices**.
 
-## Installation
+# Installation
 
 To install nn-meter, please first install python3. The test environment uses anaconda python 3.6.10. Install the dependencies via:
 `pip3 install -r requirements.txt`
 Please also check the versions of numpy, scikit_learn. The different versions may change the prediction accuracy of kernel predictors.
 
 If you use nn-meter in NNI, make sure NNI version >= 2.4
 
-## Usage
+# Usage
+## Supported input model format
+We have two types of interfaces：
+- command line `nn-meter` after install the package `nn-meter`
+- Python binding provided by the module `nn_meter`
 
-#### Supported input model format
+Here is a summary of supported inputs of the two methods.
 
-|       name       | format |
-| :--------------: | :----: |
-|    Tensorflow    |  .pb  |
-|       Onnx       | .onnx |
-| nnmeter IR graph | .json |
-|   NNI IR graph   | .json |
+|       Name        |                     Command Support                 | Python Binding |
+| :--------------:  | :-------------------------------------------------: | :---: |
+|    Tensorflow     |  Checkpoint file dumped by `tf.saved_model()` and endwith `.pb`  | Checkpoint file dumped by `tf.saved_model` and endwith `.pb` | 
+|       Torch       | Models in `torchvision.models` | Object of `torch.nn.Module` |
+|       Onnx        | Checkpoint file dumped by `onnx.save()` and endwith `.onnx`  | Checkpoint file dumped by `onnx.save()` or model loaded by `onnx.load()` |
+| nn-Meter IR graph | Json file in the format of [nn-Meter IR Graph](./docs/input_models.md#nnmeter-ir-graph)  | `dict` object following the format of [nn-Meter IR Graph](./docs/input_models.md#nnmeter-ir-graph) | 
+|   NNI IR graph    | -  | `dict` object following [NNI Doc](https://nni.readthedocs.io/en/stable/Tutorial/InstallationLinux.html#installation) |
 
-#### To predict a single model: Run nn-Meter demo
 
-After installation, a command named `nn-meter` is enabled. To predict the latency for a CNN model with a predefined predictor, users can run the following commands
+## Command line Support
+### List all predefined predictors
+
+After installation, a command named `nn-meter` is enabled. Users can get all predefined predictors by running
 
 ```bash
 # to list all predefined predictors
 nn-meter --list-predictors 
-
-# for Tensorflow (*.pb) file
-nn-meter --predictor <hardware> --tensorflow <pb-file> 
-
-# for ONNX (*.onnx) file
-nn-meter --predictor <hardware> --onnx <onnx-file>
-
-# for nn-Meter IR (*.json) file
-nn-meter --predictor <hardware> --nn-meter-ir <json-file> 
-
-# for NNI IR (*.json) file
-nn-meter --predictor <hardware> --nni-ir <json-file> 
 ```
 
 nn-Meter currently supports prediction on the following four config:
@@ -66,16 +61,59 @@ nn-Meter currently supports prediction on the following four config:
 |         cortexA76cpu_tflite21         |
 |         adreno640gpu_tflite21         |
 |         adreno630gpu_tflite21         |
-|       myriadvpu_openvino2019r2       |
+|       myriadvpu_openvino2019r2        |
 
 For the input model file, you can find any example provided under the `data/testmodels`
 
-#### Import nn-Meter in your python code
+
+### Predict latency for CNN model
+
+To predict the latency for a CNN model with a predefined predictor in command line, users can run the following commands
+
+```bash
+# for Tensorflow (*.pb) file
+nn-meter --predictor <hardware> [--predictor-version <version>] --tensorflow <pb-file_or_folder> 
+
+# for ONNX (*.onnx) file
+nn-meter --predictor <hardware> [--predictor-version <version>] --onnx <onnx-file_or_folder>
+
+# for torch model from torchvision model zoo (str)
+nn-meter --predictor <hardware> [--predictor-version <version>] --torchvision <model-name> <model-name>... 
+
+# for nn-Meter IR (*.json) file
+nn-meter --predictor <hardware> [--predictor-version <version>] --nn-meter-ir <json-file_or_folder> 
+```
+
+`--predictor-version <version>` arguments is optional. When the predictor version is not specified by users, nn-meter will use the latest verison of the predictor.
+
+nn-Meter can support batch mode prediction. To predict latency for multiple models in the same model type once, user should collect all models in one folder and state the folder after `--[model-type]` liked argument.
+
+ It should also be noted that for PyTorch model, nn-meter can only support existing models in torchvision model zoo. The string followed by `--torchvision` should be exactly one or more string indicating name(s) of some existing torchvision models.
+
+
+### Convert to nn-Meter IR Graph
+
+Furthermore, users may be interested to convert tensorflow pb-file or onnx file to nn-Meter IR graph. Users could convert nn-Meter IR graph and save to `.json` file be running
+
+```bash
+# for Tensorflow (*.pb) file
+nn-meter getir --tensorflow <pb-file> [--output <output-name>]
+
+# for ONNX (*.onnx) file
+nn-meter getir --onnx <onnx-file> [--output <output-name>]
+```
+
+Output name is default to be `/path/to/input/file/<input_file_name>_<model-type>_ir.json` if not specified by users.
+
+
+## Import nn-Meter in your python code: Python binding
+
+After installation, users can import nn-Meter in python code
 
 ```python
 from nn_meter import load_latency_predictor
 
-predictor = load_lat_predictor(config, hardware_name) # case insensitive in backend
+predictor = load_latency_predictor(hardware_name, hardware_predictor_version) # case insensitive in backend
 
 # build your model here
 model = ... # model is instance of torch.nn.Module
@@ -87,9 +125,11 @@ By calling `load_latency_predictor`, user selects the target hardware (`Framewor
 
 Users could view the information all built-in predictors by `list_latency_predictors` or view the config file in `nn_meter/configs/predictors.yaml`.
 
-### Hardware-aware NAS by nn-Meter and NNI
+Users could get a nn-Meter IR graph by applying `model_file_to_graph` and `model_to_graph` by calling the model name or model object and specify the model type. The supporting model types of `model_file_to_graph` include "onnx", "pb", "torch", "nnmeter-ir" and "nni-ir", while the supporting model types of `model_to_graph` include "onnx", "torch", "nnmeter-ir" and "nni-ir".
+
+## Hardware-aware NAS by nn-Meter and NNI
 
-#### Run multi-trial SPOS demo
+### Run multi-trial SPOS demo
 
 Install NNI by following [NNI Doc](https://nni.readthedocs.io/en/stable/Tutorial/InstallationLinux.html#installation).
 
@@ -105,9 +145,9 @@ Then run multi-trail SPOS demo:
 python ${NNI_ROOT}/examples/nas/oneshot/spos/multi_trial.py
 ```
 
-#### How the demo works
+### How the demo works
 
-Refer to [nni doc ](https://nni.readthedocs.io/en/stable/nas.html)for how to perform NAS by NNI.
+Refer to [nni doc](https://nni.readthedocs.io/en/stable/nas.html) for how to perform NAS by NNI.
 
 To support latency-aware NAS, you first need a `Strategy` that supports filtering the models by latency. We provide such a filter named `LatencyFilter` in NNI and initialize a `Random` strategy with the filter:
 
@@ -126,7 +166,7 @@ RetiariiExperiment(base_model, trainer, [], simple_strategy, True, example_input
 
 Here, `parse_shape=True` means extracting shape info from the torch model as it is required by nn-Meter to predict latency. `example_inputs` is required for tracing shape info.
 
-## Contributing
+# Contributing
 
 This project welcomes contributions and suggestions.  Most contributions require you to agree to a
 Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
@@ -140,7 +180,7 @@ This project has adopted the [Microsoft Open Source Code of Conduct](https://ope
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
 contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
 
-## License
+# License
 
 The entire codebase is under [MIT license](https://github.com/microsoft/nn-Meter/blob/main/LICENSE)