Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Perf] Regressions on Multiple platforms, 8/8/2024 11:52:00 PM #106345

Closed
performanceautofiler bot opened this issue Aug 13, 2024 · 6 comments
Closed

[Perf] Regressions on Multiple platforms, 8/8/2024 11:52:00 PM #106345

performanceautofiler bot opened this issue Aug 13, 2024 · 6 comments
Assignees
Labels
arch-x64 area-Meta os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Milestone

Comments

@performanceautofiler
Copy link

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Tests.Perf_Version

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
36.32 ns 40.57 ns 1.12 0.16 False
26.24 ns 28.18 ns 1.07 0.14 False

graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Tests.Perf_Version*'

System.Tests.Perf_Version.TryParse3

ETL Files

Histogram

JIT Disasms

System.Tests.Perf_Version.TryParse2

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Perf_Ascii

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.10 ns 10.19 ns 1.26 0.06 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Perf_Ascii*'

System.Text.Perf_Ascii.Equals_Chars(Size: 128)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.SearchValuesByteTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.04 ns 10.56 ns 1.31 0.16 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.SearchValuesByteTests*'

System.Buffers.Tests.SearchValuesByteTests.IndexOfAnyExcept(Values: "abcdefABCDEF0123456789Ü")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Xml.Tests.Perf_XmlConvert

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
166.98 ns 185.41 ns 1.11 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Xml.Tests.Perf_XmlConvert*'

System.Xml.Tests.Perf_XmlConvert.DateTime_ToString_Local

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in PerfLabTests.LowLevelPerf

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
196.99 μs 218.85 μs 1.11 0.02 False
109.61 μs 131.62 μs 1.20 0.04 False
262.93 μs 302.78 μs 1.15 0.04 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'PerfLabTests.LowLevelPerf*'

PerfLabTests.LowLevelPerf.StaticDelegate

ETL Files

Histogram

JIT Disasms

PerfLabTests.LowLevelPerf.ClassVirtualMethod

ETL Files

Histogram

JIT Disasms

PerfLabTests.LowLevelPerf.InterfaceInterfaceMethodLongHierarchy

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_SubstringFrozenDictionary

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
513.41 ns 566.84 ns 1.10 0.12 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_SubstringFrozenDictionary*'

System.Collections.Perf_SubstringFrozenDictionary.TryGetValue_True_FrozenDictionary(Count: 100)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<String>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
8.75 μs 9.50 μs 1.09 0.02 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet&lt;String&gt;*'

System.Collections.IndexerSet<String>.Dictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.Encodings.Web.Tests.Perf_Encoders

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
48.20 ns 55.08 ns 1.14 0.08 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.Encodings.Web.Tests.Perf_Encoders*'

System.Text.Encodings.Web.Tests.Perf_Encoders.EncodeUtf16(arguments: JavaScript,no escaping required,512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Tests.Perf_BitArray

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
9.98 ns 11.46 ns 1.15 0.28 False
9.99 ns 11.91 ns 1.19 0.30 False
10.13 ns 11.26 ns 1.11 0.33 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Tests.Perf_BitArray*'

System.Collections.Tests.Perf_BitArray.BitArrayXor(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Tests.Perf_BitArray.BitArrayAnd(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Collections.Tests.Perf_BitArray.BitArrayOr(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Memory.Span<Char>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
4.69 ns 5.91 ns 1.26 0.07 False
7.49 ns 9.33 ns 1.25 0.08 False
15.67 ns 22.77 ns 1.45 0.25 False

graph
graph
graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Memory.Span&lt;Char&gt;*'

System.Memory.Span<Char>.BinarySearch(Size: 33)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.BinarySearch(Size: 512)

ETL Files

Histogram

JIT Disasms

System.Memory.Span<Char>.Fill(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Reflection.Invoke

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
28.82 ns 31.84 ns 1.11 0.13 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Reflection.Invoke*'

System.Reflection.Invoke.StaticMethod4_int_string_struct_class

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Buffers.Tests.SearchValuesCharTests

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
6.19 ns 7.37 ns 1.19 0.33 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Buffers.Tests.SearchValuesCharTests*'

System.Buffers.Tests.SearchValuesCharTests.LastIndexOfAnyExcept(Values: "abcdefABCDEF0123456789")

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.TryGetValueFalse<Int32, Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
833.19 ns 1.04 μs 1.25 0.21 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.TryGetValueFalse&lt;Int32, Int32&gt;*'

System.Collections.TryGetValueFalse<Int32, Int32>.ConcurrentDictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.IndexerSet<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
2.49 μs 2.94 μs 1.18 0.10 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.IndexerSet&lt;Int32&gt;*'

System.Collections.IndexerSet<Int32>.Dictionary(Size: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
87.28 μs 105.50 μs 1.21 0.23 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock*'

System.Text.RegularExpressions.Tests.Perf_Regex_Industry_RustLang_Sherlock.Count(Pattern: "The", Options: None)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository


Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Collections.Perf_Frozen<Int16>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
1.40 μs 1.65 μs 1.18 0.25 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Collections.Perf_Frozen&lt;Int16&gt;*'

System.Collections.Perf_Frozen<Int16>.Contains_True(Count: 512)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@performanceautofiler performanceautofiler bot added arch-x64 os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime untriaged New issue has not been triaged by the area owner labels Aug 13, 2024
Copy link
Author

Run Information

Name Value
Architecture x64
OS ubuntu 22.04
Queue TigerUbuntu
Baseline 44b6b2ae056092d7759b068a9661b86627ec1024
Compare 48aa31535f0e5f318f061eb6f8ad4f25611e6133
Diff Diff
Configs CompilationMode:tiered, RunKind:micro

Regressions in System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives<Int32>

Benchmark Baseline Test Test/Base Test Quality Edge Detector Baseline IR Compare IR IR Ratio
135.79 ns 173.59 ns 1.28 0.05 False

graph
Test Report

Repro

General Docs link: https://github.com/dotnet/performance/blob/main/docs/benchmarking-workflow-dotnet-runtime.md

git clone https://github.com/dotnet/performance.git
python3 .\performance\scripts\benchmarks_ci.py -f net8.0 --filter 'System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives&lt;Int32&gt;*'

System.Numerics.Tensors.Tests.Perf_BinaryIntegerTensorPrimitives<Int32>.ShiftLeft(BufferLength: 3079)

ETL Files

Histogram

JIT Disasms

Docs

Profiling workflow for dotnet/runtime repository
Benchmarking workflow for dotnet/runtime repository

@LoopedBard3
Copy link
Member

LoopedBard3 commented Aug 13, 2024

Likely caused by: #106014, potentially due to unintended side effects. Compare range: 105bf4d...48aa315

This is a mix of regressions that appear to potentially just be noise and ones that are clearly not just noise, so keeping them all.

@LoopedBard3 LoopedBard3 changed the title [Perf] Linux/x64: 24 Regressions on 8/8/2024 11:52:00 PM [Perf] Regressions on Multiple platforms, 8/8/2024 11:52:00 PM Aug 13, 2024
@LoopedBard3 LoopedBard3 removed the untriaged New issue has not been triaged by the area owner label Aug 13, 2024
@LoopedBard3 LoopedBard3 transferred this issue from dotnet/perf-autofiling-issues Aug 13, 2024
@dotnet-issue-labeler dotnet-issue-labeler bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Aug 13, 2024
@dotnet-policy-service dotnet-policy-service bot added the untriaged New issue has not been triaged by the area owner label Aug 13, 2024
@teo-tsirpanis teo-tsirpanis removed the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Aug 16, 2024
Copy link
Contributor

Tagging subscribers to this area: @dotnet/area-meta
See info in area-owners.md if you want to be subscribed.

@tarekgh
Copy link
Member

tarekgh commented Aug 16, 2024

Nothing related to metrics in this issue.

@jeffhandley jeffhandley added this to the 9.0.0 milestone Aug 23, 2024
@jeffhandley jeffhandley removed the untriaged New issue has not been triaged by the area owner label Aug 23, 2024
@jeffhandley jeffhandley added tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark labels Sep 17, 2024
@jeffhandley
Copy link
Member

Closing this out as many of the results show a bi-modal or multi-modal history, and the results are otherwise acceptable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
arch-x64 area-Meta os-linux Linux OS (any supported distro) runtime-coreclr specific to the CoreCLR runtime tenet-performance Performance related issue tenet-performance-benchmarks Issue from performance benchmark
Projects
None yet
Development

No branches or pull requests

6 participants