Benchmark fails to run in Safari

[davehunt]

Whilst testing to see which of our performance tests are able to run in Safari on macOS I discovered the following error for this benchmark:

15:04:51 ERROR - raptor-browsertime Error: Browsertime failed to run
15:04:51 INFO - raptor-browsertime Info: [browsertime] Executing script return (function() {
15:04:51 INFO - raptor-browsertime Info: const supported = PerformanceObserver.supportedEntryTypes;
15:04:51 INFO - raptor-browsertime Info: if (!supported || supported.indexOf('layout-shift') === -1) {
15:04:51 INFO - raptor-browsertime Info: return;
15:04:51 INFO - raptor-browsertime Info: }
15:04:51 INFO - raptor-browsertime Info: // See https://web.dev/layout-instability-api
15:04:51 INFO - raptor-browsertime Info: // https://github.com/mmocny/web-vitals/wiki/Snippets-for-LSN-using-PerformanceObserver#max-session-gap1s-limit5s
15:04:51 INFO - raptor-browsertime Info: let max = 0;
15:04:51 INFO - raptor-browsertime Info: let curr = 0;
15:04:51 INFO - raptor-browsertime Info: let firstTs = Number.NEGATIVE_INFINITY;
15:04:51 INFO - raptor-browsertime Info: let prevTs = Number.NEGATIVE_INFINITY;
15:04:51 INFO - raptor-browsertime Info: const observer = new PerformanceObserver(list => {});
15:04:51 INFO - raptor-browsertime Info: observer.observe({ type: 'layout-shift', buffered: true });
15:04:51 INFO - raptor-browsertime Info: const list = observer.takeRecords();
15:04:51 INFO - raptor-browsertime Info: for (let entry of list) {
15:04:51 INFO - raptor-browsertime Info: if (entry.hadRecentInput) {
15:04:51 INFO - raptor-browsertime Info: continue;
15:04:51 INFO - raptor-browsertime Info: }
15:04:51 INFO - raptor-browsertime Info: if (entry.startTime - firstTs > 5000 || entry.startTime - prevTs > 1000) {
15:04:51 INFO - raptor-browsertime Info: firstTs = entry.startTime;
15:04:51 INFO - raptor-browsertime Info: curr = 0;
15:04:51 INFO - raptor-browsertime Info: }
15:04:51 INFO - raptor-browsertime Info: prevTs = entry.startTime;
15:04:51 INFO - raptor-browsertime Info: curr += entry.value;
15:04:51 INFO - raptor-browsertime Info: max = Math.max(max, curr);
15:04:51 INFO - raptor-browsertime Info: }
15:04:51 INFO - raptor-browsertime Info: return max;
15:04:51 INFO - raptor-browsertime Info: })();
15:04:51 INFO - raptor-browsertime Info:
15:04:51 CRITICAL - raptor-browsertime Critical: [browsertime] Failed to execute user script: TimeoutError
15:04:51 ERROR - Traceback (most recent call last):
15:04:51 INFO - File "/Users/dave/workspace/mozilla-unified/testing/raptor/raptor/raptor.py", line 206, in
15:04:51 INFO - main()
15:04:51 INFO - File "/Users/dave/workspace/mozilla-unified/testing/raptor/raptor/raptor.py", line 152, in main
15:04:51 INFO - success = raptor.run_tests(raptor_test_list, raptor_test_names)
15:04:51 INFO - File "/Users/dave/workspace/mozilla-unified/testing/raptor/raptor/perftest.py", line 469, in run_tests
15:04:51 INFO - self.run_test(test, timeout=int(test.get("page_timeout")))
15:04:51 INFO - File "/Users/dave/workspace/mozilla-unified/testing/raptor/raptor/browsertime/base.py", line 803, in run_test
15:04:51 INFO - raise Exception(self.browsertime_failure)
15:04:51 INFO - Exception: [browsertime] Failed to execute user script: TimeoutError
15:04:51 ERROR - Return code: 1

[moztcampbell]

This error is coming out of the 'web-vitals' package used by 'sentry' used by the 'matrix-react-sdk' we use for the benchmark. I'm not sure why this doesn't cause trouble for matrix chat itself in safari?

[92kns]

last week I found that it consistently timed out on running the cpuBenchmark script

e.g. happens just a bit earlier in the verbose error log

19:18:45     INFO -  raptor-browsertime Info: [browsertime] Executing script return (function() {
19:18:45     INFO -  raptor-browsertime Info:     // simple CPU benchmarking
19:18:45     INFO -  raptor-browsertime Info:     // following https://github.com/wikimedia/mediawiki-extensions-NavigationTiming/blob/master/modules/ext.navigationTiming.js
19:18:45     INFO -  raptor-browsertime Info:     // except not in worker to avoid CSP issues.
19:18:45     INFO -  raptor-browsertime Info:     const amount = 100000000;
19:18:45     INFO -  raptor-browsertime Info:     const startTime = performance.now();
19:18:45     INFO -  raptor-browsertime Info:     for ( let i = amount; i > 0; i-- ) {
19:18:45     INFO -  raptor-browsertime Info:         // empty
19:18:45     INFO -  raptor-browsertime Info:     }
19:18:45     INFO -  raptor-browsertime Info:     return Math.round( performance.now() - startTime );
19:18:45     INFO -  raptor-browsertime Info:  })();
19:18:45     INFO -  raptor-browsertime Info: [browsertime] Couldn't execute script named cpuBenchmark error:TimeoutError
19:18:45    ERROR -  raptor-browsertime Error: Browsertime failed to run
19:18:45     INFO -  raptor-browsertime Info: [browsertime] Failed to execute user script: TimeoutError

and then what follows immediately after is the same message in the first comment from Dave

19:18:45    ERROR -  raptor-browsertime Error: Browsertime failed to run
19:18:46     INFO -  raptor-browsertime Info: [browsertime] Executing script return (function() {
19:18:46     INFO -  raptor-browsertime Info:   const supported = PerformanceObserver.supportedEntryTypes;
19:18:46     INFO -  raptor-browsertime Info:   if (!supported || supported.indexOf('layout-shift') === -1) {
19:18:46     INFO -  raptor-browsertime Info:     return;
19:18:46     INFO -  raptor-browsertime Info:   }
19:18:46     INFO -  raptor-browsertime Info:   // See https://web.dev/layout-instability-api
19:18:46     INFO -  raptor-browsertime Info:   // https://github.com/mmocny/web-vitals/wiki/Snippets-for-LSN-using-PerformanceObserver#max-session-gap1s-limit5s
19:18:46     INFO -  raptor-browsertime Info:   let max = 0;
19:18:46     INFO -  raptor-browsertime Info:   let curr = 0;
19:18:46     INFO -  raptor-browsertime Info:   let firstTs = Number.NEGATIVE_INFINITY;
19:18:46     INFO -  raptor-browsertime Info:   let  prevTs = Number.NEGATIVE_INFINITY;
19:18:46     INFO -  raptor-browsertime Info:   const observer = new PerformanceObserver(list => {});
19:18:46     INFO -  raptor-browsertime Info:   observer.observe({ type: 'layout-shift', buffered: true });
19:18:46     INFO -  raptor-browsertime Info:   const list = observer.takeRecords();
19:18:46     INFO -  raptor-browsertime Info:   for (let entry of list) {
19:18:46     INFO -  raptor-browsertime Info:     if (entry.hadRecentInput) {
19:18:46     INFO -  raptor-browsertime Info:       continue;
19:18:46     INFO -  raptor-browsertime Info:     }
19:18:46     INFO -  raptor-browsertime Info:     if (entry.startTime - firstTs > 5000 || entry.startTime - prevTs > 1000) {
19:18:46     INFO -  raptor-browsertime Info:       firstTs = entry.startTime;
19:18:46     INFO -  raptor-browsertime Info:       curr = 0;
19:18:46     INFO -  raptor-browsertime Info:     }
19:18:46     INFO -  raptor-browsertime Info:     prevTs = entry.startTime;
19:18:46     INFO -  raptor-browsertime Info:     curr += entry.value;
19:18:46     INFO -  raptor-browsertime Info:     max = Math.max(max, curr);
19:18:46     INFO -  raptor-browsertime Info:   }
19:18:46     INFO -  raptor-browsertime Info:   return max;
19:18:46     INFO -  raptor-browsertime Info: })();
19:18:46     INFO -  raptor-browsertime Info:
19:18:46 CRITICAL -  raptor-browsertime Critical: [browsertime] Failed to execute user script: TimeoutError

so I tried both:

1. removing cpuBenchmark from the commands during run time
2. keeping it, but lowering the amount variable

And in both cases it seemed to be passing locally, but not on CI unfortunately (it seems to hang indefinitely waiting for data). Not entirely sure why even cpuBenchmark is failing on matrix-react-bench but not other benchmarks running on Safari. Also the cpuBenchmark error might likely be misleading altogether.

anyhow, @moztcampbell, would you know where I could find more information regarding the sentry/web-vitals/mrb-sdk interactions to look into this more?