[ feature request ] Use rustls

[yonas]

What is the problem your feature solves, or the need it fulfills?

Use rustls instead of OpenSSL.

Describe the solution you'd like

Either fully replace OpenSSl dependency with rustls (my preference) or publish a new feature that would substitute OpenSSL with rustls.

Describe alternatives you've considered

N/A.

[LessThanGreaterThan]

this would also allow KTLS support rustls/rustls#198

[palant]

This is currently rather complicated because Pingora is using OpenSSL APIs directly, almost without any intermediate layer. So doing this would require implementing OpenSSL APIs on top of rustls. My work in #277 at least shows which parts need to be implemented as OpenSSL is rather massive.

[Fexiven]

This is currently rather complicated because Pingora is using OpenSSL APIs directly, almost without any intermediate layer. So doing this would require implementing OpenSSL APIs on top of rustls. My work in #277 at least shows which parts need to be implemented as OpenSSL is rather massive.

Am I missing something here? Wouldn't the rustls implementation make OpenSSL obsolete? Of course Pingora has to re-implement everything to be compatible with rustls. But isn't that the general goal of the request?

[palant]

Wouldn't the rustls implementation make OpenSSL obsolete?

I suspect that Pingora will want to keep OpenSSL and BoringSSL support around rather than throw it all out for rustls. And while it’s possible to use BoringSSL via rustls (though I’m not sure how reliable boring-rustls-provider is), there is currently nothing comparable for OpenSSL.

[cpu]

there is currently nothing comparable for OpenSSL.

There is a compatibility layer that can be used to replace OpenSSL's libssl.so with Rustls subject to many caveats w.r.t supported APIs. However, in general I think that's not a very appealing integration route for green-field software written in Rust. I think abstracting over the choice of TLS implementation and then natively integrating with Rustls' rust API is the best route.

[eaufavor]

The choice of supporting OpenSSL and BoringSSL is for compliance reasons.

Even with rustls got FIPS certifications recently, switching from one crypto to another could cause millions of dollars and years to audit and certify for large organizations. Therefore, OpenSSL/BoringSSL is unlikely to be replaced.

That being said, rustls is a good addition.

[Walker-00]

We know that it's a pain but we really need rustls re-implemented version or optional rustls feature support. We need both speed, security, modern features and rusty.

[hargut]

I would like to thank the Cloudflare team, and especially @johnhurt for their help and support to integrate Rustls into Pingora. My PR is not ideal in several ways and especially in terms of layout/structure and performance. Thank you for addressing these issues by enhancing the PR and for splitting it into smaller PRs to allow reviewing and to maintain the original authors.

After having spent some time with the Rustls Pingora integration and performance comparisons of OpenSSL & Rustls within Pingora I'd like to share some of the results.

As performance related topics are hard to compare as they depend on hardware, supported instruction sets, OS, architecture, compilers and various other influences all the following should be taken just as what it is: a measurement in a single scenario.

The comparisons are based on the branch in PR #336/(90a823a) which will be in some aspects different from the final implementation. The benchmarks have been run using #367/(36274b4) with slight modifications for RSA 4096 (cert/key & timeout setup).
The binaries have been built using the release profile.
To reduce the amount of information the details will only contain HTTP/1.1 parallel results with handshake always (par_http_11_handshake_always) & the same for HTTP/2.0 (par_http_2_handshake_always).

I hope that the chosen method for comparing is suitable and provides a good view on the performance of the different TLS implementations within Pingora. In case there are issues with the selected procedure please let me know.

The comparisions have been created starting with --no-default-features --feature openssl and are compared to --no-default-features --feature rustls.

RSA 4096

Pingora Acceptor

tls_acceptor::tls_acceptor::bench_server par_http_11_handshake_always:(PARALLEL_ACCEPTORS, Version :: HTTP_11) -> pingor...
  Instructions:          4993397711|5021026699      (-0.55027%) [-1.00553x]
  L1 Hits:               6126515475|6149096019      (-0.36722%) [-1.00369x]
  L2 Hits:                  1291208|2423492         (-46.7212%) [-1.87692x]
  RAM Hits:                   49731|80361           (-38.1155%) [-1.61591x]
  Total read+write:      6127856414|6151599872      (-0.38597%) [-1.00387x]
  Estimated Cycles:      6134712100|6164026114      (-0.47557%) [-1.00478x]
tls_acceptor::tls_acceptor::bench_server par_http_2_handshake_always:(PARALLEL_ACCEPTORS, Version :: HTTP_2) -> pingora...
  Instructions:          5002305253|5028849117      (-0.52783%) [-1.00531x]
  L1 Hits:               6140725662|6160340936      (-0.31841%) [-1.00319x]
  L2 Hits:                  1677155|2947655         (-43.1021%) [-1.75753x]
  RAM Hits:                   52752|82524           (-36.0768%) [-1.56438x]
  Total read+write:      6142455569|6163371115      (-0.33935%) [-1.00341x]
  Estimated Cycles:      6150957757|6177967551      (-0.43720%) [-1.00439x]

Pingora Connector

tls_connector::tls_connector::bench_client par_http_11_handshake_always:(PARALLEL_CONNECTORS, false, Version :: HTTP_11, P...
  Instructions:           268281307|321943759       (-16.6683%) [-1.20002x]
  L1 Hits:                359280543|422791666       (-15.0218%) [-1.17677x]
  L2 Hits:                  1188521|2878135         (-58.7052%) [-2.42161x]
  RAM Hits:                   28323|46866           (-39.5660%) [-1.65470x]
  Total read+write:       360497387|425716667       (-15.3199%) [-1.18091x]
  Estimated Cycles:       366214453|438822651       (-16.5461%) [-1.19827x]
tls_connector::tls_connector::bench_client par_http_2_handshake_always:(PARALLEL_CONNECTORS, false, Version :: HTTP_2, PA...
  Instructions:           277597705|329954629       (-15.8679%) [-1.18861x]
  L1 Hits:                373917825|434292560       (-13.9019%) [-1.16147x]
  L2 Hits:                  1675678|3492317         (-52.0182%) [-2.08412x]
  RAM Hits:                   31378|50708           (-38.1202%) [-1.61604x]
  Total read+write:       375624881|437835585       (-14.2087%) [-1.16562x]
  Estimated Cycles:       383394445|453528925       (-15.4642%) [-1.18293x]

EC prime256v1

Pingora Acceptor

tls_acceptor::tls_acceptor::bench_server par_http_11_handshake_always:(PARALLEL_ACCEPTORS, Version :: HTTP_11) -> pingor...
  Instructions:           107482769|212122985       (-49.3300%) [-1.97355x]
  L1 Hits:                136014380|281281221       (-51.6447%) [-2.06803x]
  L2 Hits:                  1443878|2572731         (-43.8776%) [-1.78182x]
  RAM Hits:                   48543|79576           (-38.9979%) [-1.63929x]
  Total read+write:       137506801|283933528       (-51.5708%) [-2.06487x]
  Estimated Cycles:       144932775|296930036       (-51.1896%) [-2.04874x]
tls_acceptor::tls_acceptor::bench_server par_http_2_handshake_always:(PARALLEL_ACCEPTORS, Version :: HTTP_2) -> pingora...
  Instructions:           115976716|220320734       (-47.3601%) [-1.89970x]
  L1 Hits:                149532727|292910436       (-48.9493%) [-1.95884x]
  L2 Hits:                  1828453|3096953         (-40.9596%) [-1.69376x]
  RAM Hits:                   50906|81505           (-37.5425%) [-1.60109x]
  Total read+write:       151412086|296088894       (-48.8626%) [-1.95552x]
  Estimated Cycles:       160456702|311247876       (-48.4473%) [-1.93976x]

Pingora Connector

tls_connector::tls_connector::bench_client par_http_11_handshake_always:(PARALLEL_CONNECTORS, false, Version :: HTTP_11, P...
  Instructions:           118198059|299077607       (-60.4791%) [-2.53031x]
  L1 Hits:                149451893|395892683       (-62.2494%) [-2.64896x]
  L2 Hits:                  1195115|3111570         (-61.5913%) [-2.60357x]
  RAM Hits:                   30373|49284           (-38.3715%) [-1.62263x]
  Total read+write:       150677381|399053537       (-62.2413%) [-2.64840x]
  Estimated Cycles:       156490523|413175473       (-62.1249%) [-2.64026x]
tls_connector::tls_connector::bench_client par_http_2_handshake_always:(PARALLEL_CONNECTORS, false, Version :: HTTP_2, PA...
  Instructions:           127484051|307137592       (-58.4929%) [-2.40922x]
  L1 Hits:                164039310|407482688       (-59.7432%) [-2.48406x]
  L2 Hits:                  1684366|3727606         (-54.8137%) [-2.21306x]
  RAM Hits:                   33441|53125           (-37.0522%) [-1.58862x]
  Total read+write:       165757117|411263419       (-59.6956%) [-2.48112x]
  Estimated Cycles:       173631575|427980093       (-59.4300%) [-2.46487x]

System Details

~$ cat /proc/cpuinfo | grep flags | uniq
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap
~$ uname -a
Linux 6.10.5-100.fc39.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Aug 14 15:49:25 UTC 2024 x86_64 GNU/Linux
~$ valgrind --version
valgrind-3.22.0
:~$ rustc --version
rustc 1.72.1 (d5c2e9c34 2023-09-13)

Have a great time, and many thanks for reading till the end. 😃

Kind regards,
Harald