Cherry-pick Add a flag to turn on/off the lowering of scalar broadcasting binary ops to NNPA #2782
Conversation
…ops to NNPA (onnx#2778) * Add a flag to turn on/off scalar broadcasting binary op in NNPA Signed-off-by: Tung D. Le <[email protected]> --------- Signed-off-by: Tung D. Le <[email protected]> Co-authored-by: Alexandre Eichenberger <[email protected]> (cherry picked from commit 08d4fed)
cjvolzka
changed the title
There was a problem hiding this comment.
LGTM.
I assume it is to cover a regression. Are there examples where this optimization is worth it? And others where it is not? It feel that it could use a performance model. Note that the DIV op does also work well on CPU with parallel (as there is non-negligible amount of work).
| @@ -55,6 +55,13 @@ llvm::cl::opt<bool> nnpaEnableCompilerStickUnstick( | |||
| "stick/unstick code. Default is false."), | |||
| llvm::cl::init(false), llvm::cl::cat(OnnxMlirOptions)); | |||
|
|
|||
| llvm::cl::opt<bool> nnpaEnableScalarBcastBinary( | |||
| "nnpa-enable-scalar-bcast-binary", | |||
| llvm::cl::desc("Enable the lowering to NNPA the broadcasting binary ops " | |||
There was a problem hiding this comment.
I would suggest a clearer wording:
Enable the lowering to NNPA of binary operations with broadcasting of a scalar operand. Currently only enable ONNXDiv. Default is false.
There was a problem hiding this comment.
Are there examples where this optimization is worth it? And others where it is not? It feel that it could use a performance model.
I was effective to roberta-base, but later a rule to propagate Div to the inputs of MatMul can remove Div.
I don't have a model to which this feature is effective now, but in general, we want to offload a broadcasting div to NNPA only if it is surrounded by NNPA ops.
I reworded the explanation in PR #2780.
Signed-off-by: Tung D. Le [email protected]
Signed-off-by: Tung D. Le [email protected]
Co-authored-by: Alexandre Eichenberger [email protected]
(cherry picked from commit 08d4fed)