Join TechEmpower Benchmarks!

[AmirHmZz]

It would be a great opportunity to see uWebaockets in TechEmpower benchmark! IMHO it can leverages uWebaockets community and introduces it to more people.

[ghost]

TechEmpower benchmarks are not benchmarks for web servers. It's essentially a database benchmark, as all their tests involve database lookups.

[szmarczak]

Those are web server benchmarks as well, note the plaintext section.

[ghost]

Yep plaintext benchmarks are the only ones who qualify. But their execution is poor and tainted. So the overall conclusion is: we do our own benchmarks better, as we have done since 2016

[cirospaciari]

@AmirHmZz @szmarczak @alexhultman just for curiosity

The PyPy3 binding is in the same level of Drogon C++ (a little behind), but got better results than Fibers.
I recently did not tested the original implementation of uWebSocket in C++, but is almost for sure at least Drogon performance. Using TechEmpower benchmark uWebSockets C++ is one of the fastest frameworks in the plaintext benchmark, if not the fastest.
When choosing an option to creating bindings to Ruby, Python and Lua i choose uWebSockets against Drogon because its get the best results with autocannon, apache bench, wrk and TechEmpower with plaintext (I did not test with uSockets http_load at the time) an has WS :D.

Today i testing with TechEmpower plaintext:
Drogon is in 1.2 million to 1.3 millions req/s
PyPy3 bindings is in 1.1 to 1.2 million req/s
Fiber is in 1.0 million to 1.1 millions req/s
japronto is only 0.58 million req/s

In my machine at least, none of this passes from 200k req/s with json (with is limited to the json framework), PyPy json implementations is realy good and its got 227k reqs/s more than 10% over Drogon and Fibers with i got up to 197k and 199k respectively, but this is basically an json framework test not an http load test.

I will test the Ruby extension, the Node.js extension and the C++ version today an i will post here the results.
I will test with uSockets too and compare the results.

I think posting in TechEmpower is a matter of marketing to get more users, here uSockets load test are way better for finding performance botternecks than TechEmpower with uses wrk + lua for pipelining.

[AmirHmZz]

I can just say one thing :
HOLLLLLLLY SHIT !

[AmirHmZz]

@cirospaciari I'm really excited to see socketify.py in production but I'm not a good C++ programmer so I cannot help you in C parts :( but I'm working with python for about 8 years so I would be glad to contribute in Python parts anyway :)

[cirospaciari]

and i think we can get better results than this, because the extensions (Ruby and Python) still is not in the production level i want.

Python3 bindings got 1/3 of the results with 400k req/s in plaintext, but its expected because CFFI is way slower than Cython using Python3, the bindings i wrote is PyPy3 focused.

[cirospaciari]

@cirospaciari I'm really excited to see soketify.py in production but I'm not a good C++ programmer so I cannot help you in C parts :( but I'm working with python for about 8 years so I would be glad to contribute in Python parts anyway :)

@AmirHmZz
Your feedbacks will be appreciated, my intension is to release the docs and examples with postgres, mariadb, sqlite etc for the MVP and after this write some tutorials and post on medium, and other medias :)

[cirospaciari]

I think at least people will say WTF is this uWebSockets that tops the charts using C++, Python, Ruby, Lua.

@alexhultman your work is really amazing :)

[AmirHmZz]

@alexhultman would you please reconsider about joining TechEmpower benchmarks :( ?

[ghost]

TechEmpower is not scientific.

You talk about JSON - why? JSON has literally no relation with uWS whatsoever. This is scientifically nonsense, and anyone with the slightest eye for statistics can see that their plaintext test is full of 20-or-so "winners" with statistically the same result. They cap out on network bandwidth and their pipelined tests are unrealistic. I can write a small C server using uSockets and win that list with ease, but it will just be one of the 20-or-so random winners.

Comparing a standards compliant server with URL router and WebSocket support and header getters and setters, with raw benchmark winners that does nothing other than just win with minimal effort says nothing. Putting uWS in that list is just going to put it among the 20 winners. Then some non scientific interpretation of that list will conclude some one winner and everyone will freak out about it even though it does nothing more than win benchmarks.

Therefore it makes no sense to add uWS, but rather to add uSockets with the most minimal hack to get through that test. And that's not something I have time for or interest in.

This project is 7 years old and it is really not in a position where it needs to win benchmarks, that's already established.

[cirospaciari]

Yeap but only that still not enough, a server that can't scale is also bad, an good application needs to do the most efficient use of hardware and be able to scale properly, Fiber and Gorilla in Go scales badly using coroutines/threads, you can have a better result with multi process. Arquiteture, Scaling Strategy, Hardware usage are the bare minimum to an benchmark.
But my point is just Marketing and visibility :)

[ghost]

But by participating, you agree to their crappy terms. It gives them even more credibility, credibility I highly oppose since I consider them incompetent idiots who have no fucking idea about what they are doing.

And one thing (plaintext) leads to the other thing (db tests) and then we're just looking at PostgreSQL wins (which again could be blatantly shown if they just displayed server CPU usage).

Thinking about how incompetent they are, makes me want to vomit.

[cirospaciari]

Is just like cars company make cars to pass NCAP and not really means it's secure. People do TechEmpower for marketing, you can do your owns benchmarks and use theirs publicity to call attention and get theirs credibility down :)
That's why I wrote a lot of benchmarks with much more deeph and meaning. Use them for marketing and show people's how is interested good science and results.
I don't want to call theirs benchmark bad, I want to show why it's bad and what's is relevant results ;) and at the same time getting support from more and more people

[ghost]

I can make you an uSockets server that will win that list easily

[cirospaciari]

I can make you an uSockets server that will win that list easily

For sure, can we open an discussion about benchmarking practices to help me to polish my work?

[ghost]

The more I look at this, the more I confirm my utter lack of respect for these tests:

1. The requirements say "The Plaintext test is an exercise of the request-routing fundamentals only", yet all of the top performers have no routing at all.
2. The requirements say that, the response buffer must not be pre-formatted but must be formatted on a per-request basis, yet, all the top performers ignore this and have pre-formatted buffers.
3. Looking at the source for some of the top performers, it is blatantly clear that they cut corners and cheat in every single possible way imaginable, it's so utterly pointless of a test since every short cut that can be taken has been taken.
4. I mean look at this https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/C%2B%2B/ulib/src/plaintext.uso they don't even parse anything, or route anything or format anything, the validity of these tests is absolute zero, they are entirely shit and ridiculous in every sense.

[ghost]

Let me ask you a simple question; do you think this application code below, would be comparable with the kind of application code that uWS allows? The answer is clear as day; no fucking way in hell these are comparable...

<!--#declaration
static char* ptr;
-->
<!--#header
-->
<!--#code
char* pwbuffer = UClientImage_Base::wbuffer->data();

U_INTERNAL_DUMP("pwbuffer = %#.10S", pwbuffer)

if (u_get_unalignedp64(pwbuffer+36) != U_MULTICHAR_CONSTANT64('x','t','/','p','l','a','i','n'))
   {
   u_put_unalignedp64(pwbuffer,    U_MULTICHAR_CONSTANT64('C','o','n','t','e','n','t','-'));
   u_put_unalignedp64(pwbuffer+8,  U_MULTICHAR_CONSTANT64('L','e','n','g','t','h',':',' '));
   u_put_unalignedp32(pwbuffer+16, U_MULTICHAR_CONSTANT32('1','3','\r','\n'));
   u_put_unalignedp64(pwbuffer+20, U_MULTICHAR_CONSTANT64('C','o','n','t','e','n','t','-'));
   u_put_unalignedp64(pwbuffer+28, U_MULTICHAR_CONSTANT64('T','y','p','e',':',' ','t','e'));
   u_put_unalignedp64(pwbuffer+36, U_MULTICHAR_CONSTANT64('x','t','/','p','l','a','i','n'));
   u_put_unalignedp32(pwbuffer+44, U_MULTICHAR_CONSTANT32('\r','\n','\r','\n'));

   ptr = pwbuffer + U_CONSTANT_SIZE("Content-Length: 13\r\nContent-Type: text/plain\r\n\r\n");
   }

U_INTERNAL_ASSERT_EQUALS(u_get_unalignedp64(UClientImage_Base::wbuffer->data()), U_MULTICHAR_CONSTANT64('C','o','n','t','e','n','t','-'))

u_put_unalignedp64(ptr,   U_MULTICHAR_CONSTANT64('H','e','l','l','o',',',' ','W'));
u_put_unalignedp64(ptr+8, U_MULTICHAR_CONSTANT64('o','r','l','d','!','\0','\0','\0'));

UClientImage_Base::wbuffer->size_adjust_constant(U_CONSTANT_SIZE("Content-Length: 13\r\nContent-Type: text/plain\r\n\r\n") + 13);
-->

[ghost]

Not only is this solution breaking the pre-formatted response requirement, but it is doing so in ways similar to how Pablo Escobar bought the Colombian police - in ways clear as day. Holy shit, to even allow such nonsense is jaw dropping and really shows how completely idiotic TechEmpower tests are

[ghost]

Final conclusion regarding TechEmpower tests:

Instead of blindly falling for the nice colorful graphs and the exciting "gladiator style" competitive tournament reporting, with one single winner, you should stop and use your analytical brain and actually look into what these tests do, and think:

1. TechEmpower is a database test disguised as a web server test.

Almost all tests include some database and all of the 119 top performers use PostgreSQL while most of the bottom performers use MySQL. Any analytical brain would conclude; "If 119 of the top "web servers" use PostgreSQL, then we can group those winners under the category "PostgreSQL users" rather than under their respective web server usage. It becomes obvious that the only common trait between the 119 top performers is..... PostgreSQL, so it makes no scientifical sense to attribute the victory to the web server (which TechEmpower do!). This would also be obvious if they reported CPU-usage per web server, which they don't.

2. The few tests that actually only test the very web server are cheated.

Looking at the kind of solution that wins in plaintext tests, it should be obvious that these are nothing but unsafe corner cutting hacks which blatantly ignore the given rules of the game.

• No URL routing is done whatsoever, in fact, most hacks don't even parse the incoming HTTP but rely on hard-coded memory offsets or any such obviously bullshit solution that no company or app would ever be able to utilize. This while, ironically, the rules of the game state that this test should stress the request router! What router!? There is none.
• They don't even have the Date field, which is the only varying thing. This breaks the rules of the game but nobody cares.
• These tests are incredibly unrealistic, with the most absurd levels of HTTP pipelining in use, a feature that is disabled by every single web browser out there.
• No response formatting, whatsoever, is performed. It is all just a hard-coded pre-formatted static buffer that gets written back. This breaks the rules of the game, but nobody seems to care.
• Most of these "winners" are abandoned hacks nobody uses, written by amateurs who just want to cheat the top. They mean nothing, for any real usage.

3. Any proper study must refrain from drawing significant conclusions based on insignificant data!

This is the thing that drives me mad. Because TechEmpower is a "gladiator tournament" where there must be only and exactly one winner - scientific sacrilege must be performed. This because, the actual data clearly shows that there are about 10 or 20 "draws" in the top, where differences are so small they cannot even be concluded to be actual differences but rather random noise. You cannot, absolutely not, draw significant conclusions based on insignificant data! You must not declare one single "winner" from a spectra of results so close they must be observed with an electron microscope to even spot an edge.

4. Adding yet another such hack to win their benchmark is pointless and childish.

Not only is it pointless, but it also further gives TechEmpower credibility, something they clearly should not be given, seeing how utterly non-scientific their tests are.

Anyone can win such a test! You don't even need to parse HTTP! You can just count the number of CRLFCRLF you get, sending back already pre-formatted and pre-laid out buffers in one single call. Heck, you can even figure out the size of the request they send us, roughly dividing the incoming data by that number, sending back that many responses from an already pre-formatted and pre-replicated long buffer. You can optimize this to almost nothing at all, which is exactly what these "solutions" already have done! None of this is usable in any realistic app.

We could easily add yet another benchmark-winning hack to TechEmpower, easily scoring among the top 5 "winners", but who cares? None of this is realistic in any way and none of those hacks have any real production usage.