Handling the measurements associated with multiple runs #2547

AlexandreEichenberger · 2023-10-03T04:16:32Z

Using the RunONNXModel.py, we can generate several perf results which are listed in the measurement file (or stdout) separated by ==START-REPORT== strings.

ONNX_MLIR_INSTRUMENT_FILE=run.log RunONNXModel.py -m softmax1.mlir -c "-O3 -mcpu=z16 -maccel=NNPA -shapeInformation=0:32x32x64 -profile-ir=ZHigh" -w 2 -n 24

resulting in this

==START-REPORT==
==PERF-REPORT==, onnx.Constant, softmax1.mlir:3, before, 0.000004, 0.000004
==PERF-REPORT==, onnx.Constant, softmax1.mlir:3, after, 0.000001, 0.000005
==PERF-REPORT==, onnx.Max, softmax1.mlir:3, before, 0.000000, 0.000005
==PERF-REPORT==, onnx.Max, softmax1.mlir:3, after, 0.000047, 0.000052
==PERF-REPORT==, zhigh.Stick, softmax1.mlir:3, before, 0.000000, 0.000052
==PERF-REPORT==, zhigh.Stick, softmax1.mlir:3, after, 0.000037, 0.000089
==PERF-REPORT==, zhigh.Softmax, softmax1.mlir:3, before, 0.000001, 0.000090
==PERF-REPORT==, zhigh.Softmax, softmax1.mlir:3, after, 0.000183, 0.000273
==PERF-REPORT==, zhigh.Unstick, softmax1.mlir:3, before, 0.000001, 0.000274
==PERF-REPORT==, zhigh.Unstick, softmax1.mlir:3, after, 0.000015, 0.000289
==START-REPORT==
==PERF-REPORT==, onnx.Constant, softmax1.mlir:3, before, 0.000000, 0.000000
==PERF-REPORT==, onnx.Constant, softmax1.mlir:3, after, 0.000001, 0.000001
==PERF-REPORT==, onnx.Max, softmax1.mlir:3, before, 0.000000, 0.000001
...

You can then feed it to the make-report.py which will remove the top/bottom quartile and then report the average of the remaining one. The same -warmup option is used to discard the initial runs.

  make-report.py -r run24.log  -w 2      
Analyse PERF
Gather stats from 26 measurement sets with 2 warmup; keep inner 12 experiment(s)
Statistic legend:
  op-name: num, average time (s), cumulative time (s), percent of total 

Statistics start, all ops, ordered_by time, tot_time 0.0002203
  zhigh.Softmax, 1, 0.0001777, 0.0001777, 80.7%
  zhigh.Stick, 1, 0.0000148, 0.0000148, 6.7%
  zhigh.Unstick, 1, 0.0000140, 0.0000140, 6.4%
  onnx.Max, 1, 0.0000096, 0.0000096, 4.3%
  onnx.Constant, 1, 0.0000003, 0.0000003, 0.2%
Statistics end, all ops, ordered_by time, tot_time 0.0002203

Signed-off-by: Alexandre Eichenberger <[email protected]>

AlexandreEichenberger · 2023-10-03T13:02:35Z

@tungld trying to average the results of multiple runs. Technical question, is there an easy way to add instruction to be the first and the last when inserting instrumentation so that I may initialize/close a file? Right now I only initialize during the first... which is ok but not great. Thanks

Signed-off-by: Alexandre Eichenberger <[email protected]>

tungld · 2023-10-04T02:43:10Z

@tungld trying to average the results of multiple runs. Technical question, is there an easy way to add instruction to be the first and the last when inserting instrumentation so that I may initialize/close a file? Right now I only initialize during the first... which is ok but not great. Thanks

I don't know how to determine the last instrument point, but we can call a function to close the file at the end of the entry point run_main_graph. In that case, we would open the file with the append mode.

Signed-off-by: Alexandre Eichenberger <[email protected]>

AlexandreEichenberger · 2023-10-05T02:13:43Z

@tungld its ready to be reviewed.

tungld

LGTM!

tungld · 2023-10-05T02:15:11Z

test/mlir/accelerators/nnpa/conversion/instrument/add-onnx-level.mlir

@@ -1,12 +1,14 @@
-// RUN: onnx-mlir --mcpu=z16 --maccel=NNPA  --printIR --EmitZHighIR --instrument-stage=Onnx --instrument-ops=onnx.* --InstrumentBeforeOp --InstrumentAfterOp --InstrumentReportTime -tag="test" %s | FileCheck %s
+// RUN: onnx-mlir --mcpu=z16 --maccel=NNPA  --printIR --EmitZHighIR -profile-ir=Onnx %s | FileCheck %s


Thanks for updating this!

tungld · 2023-10-05T02:16:26Z

utils/RunONNXModel.py

@@ -550,6 +550,9 @@ def warning(msg):
 def data_without_top_bottom_quartile(data, percent):
    data = np.array(sorted(data))
    trim = int(percent * data.size / 100.0)
+    if trim == 0 or data.size - 2 * trim < 1:
+        # Want at least one element, return as is.
+        return data


Thanks! I sometimes got the error here with a small number of iterations, say 3.

So did I :-)

jenkins-droid · 2023-10-05T18:15:54Z

Jenkins Linux s390x Build #12952 [push] Handling the measurement... started at 14:15

jenkins-droid · 2023-10-05T18:15:57Z

Jenkins Linux amd64 Build #12928 [push] Handling the measurement... started at 13:15

jenkins-droid · 2023-10-05T18:15:58Z

Jenkins Linux ppc64le Build #11945 [push] Handling the measurement... started at 14:23

jenkins-droid · 2023-10-05T19:19:54Z

Jenkins Linux amd64 Build #12928 [push] Handling the measurement... passed after 1 hr 3 min

jenkins-droid · 2023-10-05T19:32:05Z

Jenkins Linux s390x Build #12952 [push] Handling the measurement... passed after 1 hr 16 min

jenkins-droid · 2023-10-05T20:03:48Z

Jenkins Linux ppc64le Build #11945 [push] Handling the measurement... passed after 1 hr 47 min

initial code

224d54e

Signed-off-by: Alexandre Eichenberger <[email protected]>

AlexandreEichenberger marked this pull request as draft October 3, 2023 04:16

AlexandreEichenberger added 2 commits October 3, 2023 10:19

works

7d0eb71

Signed-off-by: Alexandre Eichenberger <[email protected]>

look at littests

398e9de

Signed-off-by: Alexandre Eichenberger <[email protected]>

AlexandreEichenberger added 2 commits October 4, 2023 11:14

merge

53bebbd

Signed-off-by: Alexandre Eichenberger <[email protected]>

fixed littests

d49ac9e

Signed-off-by: Alexandre Eichenberger <[email protected]>

AlexandreEichenberger marked this pull request as ready for review October 4, 2023 16:47

format

c3f5cdd

Signed-off-by: Alexandre Eichenberger <[email protected]>

AlexandreEichenberger requested a review from tungld October 4, 2023 17:05

AlexandreEichenberger added 2 commits October 4, 2023 14:39

Merge branch 'main' into multiple-perf-run

36201ab

Merge branch 'main' into multiple-perf-run

c5c1c1a

tungld approved these changes Oct 5, 2023

View reviewed changes

Merge branch 'main' into multiple-perf-run

7e9fa59

AlexandreEichenberger merged commit 7cacebc into onnx:main Oct 5, 2023
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handling the measurements associated with multiple runs #2547

Handling the measurements associated with multiple runs #2547

AlexandreEichenberger commented Oct 3, 2023 •

edited

Loading

AlexandreEichenberger commented Oct 3, 2023

tungld commented Oct 4, 2023

AlexandreEichenberger commented Oct 5, 2023

tungld left a comment

tungld Oct 5, 2023

tungld Oct 5, 2023

AlexandreEichenberger Oct 5, 2023

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

		@@ -1,12 +1,14 @@
		// RUN: onnx-mlir --mcpu=z16 --maccel=NNPA --printIR --EmitZHighIR --instrument-stage=Onnx --instrument-ops=onnx.* --InstrumentBeforeOp --InstrumentAfterOp --InstrumentReportTime -tag="test" %s \| FileCheck %s
		// RUN: onnx-mlir --mcpu=z16 --maccel=NNPA --printIR --EmitZHighIR -profile-ir=Onnx %s \| FileCheck %s

Handling the measurements associated with multiple runs #2547

Handling the measurements associated with multiple runs #2547

Conversation

AlexandreEichenberger commented Oct 3, 2023 • edited Loading

AlexandreEichenberger commented Oct 3, 2023

tungld commented Oct 4, 2023

AlexandreEichenberger commented Oct 5, 2023

tungld left a comment

Choose a reason for hiding this comment

tungld Oct 5, 2023

Choose a reason for hiding this comment

tungld Oct 5, 2023

Choose a reason for hiding this comment

AlexandreEichenberger Oct 5, 2023

Choose a reason for hiding this comment

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

jenkins-droid commented Oct 5, 2023

AlexandreEichenberger commented Oct 3, 2023 •

edited

Loading