Multi-threading reporting question

[mike-barber]

I'm curious about how we're reporting the performance of multi-threaded solutions, and I'm sure there are some good reasons for it.

Essentially, there are two options available:

1. Total throughput
2. Per-thread throughput

It looks like we're reporting (2). The issue with this is that it creates the wrong incentive for the solution: use as few threads as possible, usually just two (so that it's still counted as multithreaded), as this will have the smallest impact on the processor.

For me, the interesting measurement would be (1), as this measures the scalability of the solution, even if the problem is admittedly "trivially" parallelisable. In the absence of serious design flaws, my expectation would be that reported performance would go up with more threads, rather than down. We usually expect per-thread performance to decrease: it's just the cost of sub-linear scaling.

(2) could still make sense if it's a requirement for all solutions to use all available (hyper)threads, but this is not the case currently; there's quite a spectrum.

As a side note, I'd love to see the report run on Dave's ThreadRipper at some point. Even under WSL2, it should report some really good numbers, and we're a little Intel-heavy at this point. :) And I'd really love to see how stuff runs on the M1, but concede that it could be quite tricky getting enough of the solutions running on there.

[ItalyToast]

I havent developed any multithreaded solution but my expectation was that is was measuring total throughput.

(2) could still make sense if it's a requirement for all solutions to use all available (hyper)threads, but this is not the case currently; there's quite a spectrum.

I dont think thats the best solution since you might get better performance by running one thread per core even if you got hyperthreading, it should not affect most algorithms and hardware. But it might be worth considering.

As a side note, I'd love to see the report run on Dave's ThreadRipper at some point. Even under WSL2, it should report some really good numbers, and we're a little Intel-heavy at this point. :) And I'd really love to see how stuff runs on the M1, but concede that it could be quite tricky getting enough of the solutions running on there.

M1 would be cool indeed.

[rbergen]

I'm not aiming to start a semantical discussion as such, but we do report both total throughput ("Passes" in the PrimeView results table) and throughput per thread, per second ("Passes/s/t" in the same). The pre-baked filter preset called "multithreaded leaderboard" in the application does indeed rank by the latter. Noting first that that filter preset('s ranking) is "unofficial" because the actual leaderboard only considers single-threaded solutions, we felt that the passes per second per thread metric best represents a language's intrinsic performance, which is what this project is primarily about.

As a side note, I'd love to see the report run on Dave's ThreadRipper at some point. Even under WSL2, it should report some really good numbers, and we're a little Intel-heavy at this point. :) And I'd really love to see how stuff runs on the M1, but concede that it could be quite tricky getting enough of the solutions running on there.

I agree with the reporting being Intel-heavy, but I do question how meaningful results generated on/under WSL2 would be. We know performance is seriously impacted in that setup, both in raw numbers and in terms of consistency across benchmark runs. I'd also love to include M1 benchmark reporting, and would gladly spend the time on fixing solutions to run on that architecture. One issue with both AMD and M1 is that I don't own the equipment nor have the budget to change that. :)

[mike-barber]

I'm not aiming to start a semantical discussion as such, but we do report both total throughput ("Passes" in the PrimeView results table) and throughput per thread, per second ("Passes/s/t" in the same). The pre-baked filter preset called "multithreaded leaderboard" in the application does indeed rank by the latter. Noting first that that filter preset('s ranking) is "unofficial" because the actual leaderboard only considers single-threaded solutions, we felt that the passes per second per thread metric best represents a language's intrinsic performance, which is what this project is primarily about.

That is true :) I guess I was asking more about the default ranking than the reporting. And I'm curious about why you feel that it is the better measure. I'm definitely not asserting that it's incorrect, but I also don't understand it just yet. I'll try to explain what I mean. In my head, we've got the single-thread benchmarks for measuring single-thread performance already as an indication intrinsic performance. And I absolutely agree this is a good measurement, as there's a ton of variation in the way languages handle threads. However, when scaling a solution to multiple threads, I'm usually more worried about the total throughput I'll be able to achieve rather than the inevitable loss of some performance on a individual thread.

Or to put it another way, on my machine, I have 12C / 24T, and there are two ways I could run the multithreaded benchmark:

• use all 24 threads for 120,844 /sec, or 5,035 /thread/sec
• use just 2 threads for 23,427 /sec, or 11,713 /thread/sec

Doing this is kind of weird to me though: 2 threads shouldn't look like it's 2.3X faster than 24 threads. I guess that's the long-winded way of explaining what I mean :)

[rbergen]

It does seem like using only two threads indeed is faster, if you look at the performance per thread, per second. :) That is to say: the language and algorithm squeeze more performance out of the hardware resources (in this case CPU threads) used with two threads than with 24. And I think that also touches on what's behind all this.

In a number of his videos, Dave mentioned work he did on HyperCache for the Amiga. (I think the most elaborate discussion on it is at the beginning of this video, but I may be wrong.) That whole project was aimed at squeezing the most out of the hardware it was developed for, the specifications of which are rather limited by today's standards. As Dave also explains in that video, HyperCache turned out to be very important to his academic and professional career.

Personally, I'm sure that is one of the reasons that Dave really does care about resource usage optimization, which is also why the "1 bit per prime" aspect is part of the leaderboard filter.

Does this mean that currently, it can help an implementation's default multi-threaded ranking to use fewer threads than available? Yes. Can that come across very counter-intuitive? Absolutely.

[mike-barber]

Thanks @rbergen in kind of makes sense: I can see where you're coming from.

• My perspective is along the lines of "given X hardware, what is the maximum number of total passes/sec you can get out of it?"
• Your and Dave's perspective is almost more like "given X job to do, what's the most efficient use of resources?" It's almost as if you had a batch of 1 million sieves to run, and needed to know how efficiently you could do it without interrupting other work on the machine.

And I can definitely understand this perspective, especially given his background working on operating systems. You really do have specific things to do: opening a file, allocating space, drawing a window, paging, etc. And it's certainly the case that you probably wouldn't want to use all available threads for everything :)

[rbergen]

I think that's a good way to summarize/rephrase it, thank you. On the other hand, I can also totally follow your reasoning.

[BradleyChatha]

And I'd really love to see how stuff runs on the M1, but concede that it could be quite tricky getting enough of the solutions running on there.

M1 would be very interesting. D's first solution was apparently ran on an M1 and it's 2x faster than on my i5-7600k, so I'd love to see how fast the other solutions would be.