Fix memory leak when using OpenSSL and threads

[ashman-p]

'run_tests' memory leak tests fails when build options enables OQS_USE_PTHREADS and OQS_USE_OPENSSL.

valgrind tests/test_sig SPHINCS+-SHA2-256s-simple
==158054== Memcheck, a memory error detector
==158054== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==158054== Using Valgrind-3.15.0 and LibVEX; rerun with -h for copyright info
==158054== Command: tests/test_sig SPHINCS+-SHA2-256s-simple
==158054== 
Testing signature algorithms using liboqs version 0.10.2-dev
Configuration info
==================
Target platform:  x86_64-Linux-5.4.0-126-generic
Compiler:         gcc (9.4.0)
Compile options:  [-Wa,--noexecstack;-O3;-fomit-frame-pointer;-fdata-sections;-ffunction-sections;-Wl,--gc-sections;-Wbad-function-cast]
OQS version:      0.10.2-dev
Git commit:       f47817d8cb0cd621dada5276f30f99934f3132a9
OpenSSL enabled:  Yes (OpenSSL 3.3.3.8.3.0-dev )
AES:              NI
SHA-2:            OpenSSL
SHA-3:            OpenSSL
OQS build flags:  OQS_DIST_BUILD OQS_LIBJADE_BUILD OQS_OPT_TARGET=generic CMAKE_BUILD_TYPE=Release 
CPU exts active:  AES AVX AVX2 BMI1 BMI2 PCLMULQDQ POPCNT SSE SSE2 SSE3
================================================================================
Sample computation for signature SPHINCS+-SHA2-256s-simple
================================================================================
verification passes as expected
==158054== 
==158054== HEAP SUMMARY:
==158054==     in use at exit: 240 bytes in 1 blocks
==158054==   total heap usage: 7,654 allocs, 7,653 frees, 844,712 bytes allocated
==158054== 
==158054== LEAK SUMMARY:
**==158054==    definitely lost: 240 bytes in 1 blocks**
==158054==    indirectly lost: 0 bytes in 0 blocks
==158054==      possibly lost: 0 bytes in 0 blocks
==158054==    still reachable: 0 bytes in 0 blocks
==158054==         suppressed: 0 bytes in 0 blocks
==158054== Rerun with --leak-check=full to see details of leaked memory
==158054== 
==158054== For lists of detected and suppressed errors, rerun with: -s
==158054== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

Fixes #1941.

• Does this PR change the input/output behaviour of a cryptographic algorithm (i.e., does it change known answer test values)? (If so, a version bump will be required from x.y.z to x.(y+1).0.)
• Does this PR change the list of algorithms available -- either adding, removing, or renaming? Does this PR otherwise change an API? (If so, PRs in fully supported downstream projects dependent on these, i.e., oqs-provider will also need to be ready for review and merge by the time this is merged.)

[dstebila]

This is present in the 0.11.0 release, I guess? Is it critical to fix?

Do we have a CI job that tests this configuration?

[SWilson4]

This is present in the 0.11.0 release, I guess? Is it critical to fix?

Do we have a CI job that tests this configuration?

For what it's worth, OQS_USE_PTHREADS is not intended to be exposed to the user as a config variable; it's set automatically based on the availability of pthreads and not documented in CONFIGURE.md. I would guess that all of our Linux and macOS CI systems have pthreads, so we are probably not running any tests without pthreads enabled—which makes me wonder why we haven't previously detected this leak. I'm hoping that maybe this leak only arises when OQS_USE_PTHREADS is set manually instead of being autoset by CMake.

I'm not able to reproduce the leak myself—@ashman-p could you let me know what Linux environment you're running so I can test various configs out in Docker images?

[ashman-p]

I'm not able to reproduce the leak myself—@ashman-p could you let me know what Linux environment you're running so I can test various configs out in Docker images?

@SWilson4 , Thanks for checking on this issue.
I just have a generic linux setup.
Linux nashley-dev 5.4.0-126-generic #142-Ubuntu SMP Fri Aug 26 12:12:57 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

liboqs Build

cmake -DBUILD_SHARED_LIBS=ON -DOQS_USE_OPENSSL=ON -DCMAKE_BUILD_TYPE=Debug -GNinja ..
-- The C compiler identification is GNU 9.4.0
-- The ASM compiler identification is GNU
-- Found assembler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Performing Test CC_SUPPORTS_WA_NOEXECSTACK
-- Performing Test CC_SUPPORTS_WA_NOEXECSTACK - Success
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Algorithms filtered for KEM_ml_kem_512;KEM_ml_kem_768;KEM_ml_kem_1024;KEM_ml_kem_512_ipd;KEM_ml_kem_768_ipd;KEM_ml_kem_1024_ipd;SIG_ml_dsa_44;SIG_ml_dsa_65;SIG_ml_dsa_87;SIG_sphincs_sha2_128f_simple;SIG_sphincs_sha2_128s_simple;SIG_sphincs_sha2_256f_simple;SIG_sphincs_sha2_256s_simple;SIG_sphincs_shake_128f_simple;SIG_sphincs_shake_128s_simple;SIG_sphincs_shake_256f_simple
-- Found OpenSSL: /home/ubuntu/openssl3.3/ossl-install/lib/libcrypto.so (Required is at least version "1.1.1")
-- Looking for aligned_alloc
-- Looking for aligned_alloc - found
-- Looking for posix_memalign
-- Looking for posix_memalign - found
-- Looking for memalign
-- Looking for memalign - found
-- Looking for explicit_bzero
-- Looking for explicit_bzero - found
-- Looking for explicit_memset
-- Looking for explicit_memset - not found
-- Looking for memset_s
-- Looking for memset_s - not found
-- Found Doxygen: /usr/bin/doxygen (found version "1.8.17") found components: doxygen dot
-- Configuring done
-- Generating done

[ashman-p]

@SWilson4, the problem occurs when use of OQS_USE_SHA3_OPENSSL=ON and threading is active. The only significance of the use of threads is that the problem went away when threading was disabled.

[SWilson4]

OK, I've managed to reproduce the leak, but only when building against OpenSSL >= 3.3.2. In particular, the leak does not occur when building against OpenSSL 3.3.1 with the same configuration.

It seems to me that this might actually be an OpenSSL bug rather than a liboqs bug, especially as the fix proposed here should (?) be unnecessary: per the OpenSSL docs:

As of version 1.1.0 OpenSSL will automatically allocate all resources that it needs so no explicit initialisation is required. Similarly it will also automatically deinitialise as required.

Looping in @baentsch @romen @beldmit @levitte: any knowledge of a regression going from 3.3.1 to 3.3.2?

I will try to isolate the exact commit which introduces the leak.

[beldmit]

At first glance nothing suspicious

[SWilson4]

I will try to isolate the exact commit which introduces the leak.

It seems that this leak was introduced in openssl/openssl@83efcfd. When building against the parent of that commit (openssl/openssl@a13df68) the leak does not occur.

[SWilson4]

^ fyi @beldmit

[beldmit]

@nhorman could you please take a look?

[nhorman]

FWIW, i don't see anything expressly wrong with the changes, its fine to call OPENSSL_init_crypto in your application rather than having the library do it as needed, but I'm having a hard time understanding:

1. Where exactly the leak is
2. How manually calling OpenSSL_init_crypto fixes it

I'll try reproduce here, but since you already have it set up, can you re-run valgrind with --leak-check=full to show the stack trace of where the leaked memory is being allocated?

[SWilson4]

Sure, here's the valgrind output:

==462229== 
==462229== HEAP SUMMARY:
==462229==     in use at exit: 240 bytes in 1 blocks
==462229==   total heap usage: 6,841 allocs, 6,840 frees, 845,217 bytes allocated
==462229== 
==462229== 240 bytes in 1 blocks are definitely lost in loss record 1 of 1
==462229==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==462229==    by 0x4AEC3C3: CRYPTO_malloc (mem.c:202)
==462229==    by 0x4AEC44D: CRYPTO_zalloc (mem.c:222)
==462229==    by 0x4B06246: ossl_rcu_read_lock (threads_pthread.c:410)
==462229==    by 0x49D7FB8: module_find (conf_mod.c:403)
==462229==    by 0x49D7A9D: module_run (conf_mod.c:259)
==462229==    by 0x49D782F: CONF_modules_load (conf_mod.c:166)
==462229==    by 0x49D796A: CONF_modules_load_file_ex (conf_mod.c:215)
==462229==    by 0x49D8CB9: ossl_config_int (conf_sap.c:68)
==462229==    by 0x4AEAF13: ossl_init_config (init.c:258)
==462229==    by 0x4AEAEF5: ossl_init_config_ossl_ (init.c:256)
==462229==    by 0x4F31EC2: __pthread_once_slow (pthread_once.c:116)
==462229== 
==462229== LEAK SUMMARY:
==462229==    definitely lost: 240 bytes in 1 blocks
==462229==    indirectly lost: 0 bytes in 0 blocks
==462229==      possibly lost: 0 bytes in 0 blocks
==462229==    still reachable: 0 bytes in 0 blocks
==462229==         suppressed: 0 bytes in 0 blocks
==462229== 
==462229== For lists of detected and suppressed errors, rerun with: -s
==462229== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)

OpenSSL is built with

cd ~/openssl
git checkout 83efcfdfa1
./config --debug --prefix=`pwd`/../.localopenssl_83efcfdfa1_debug && make -j 4 && make install_sw install_ssldirs

liboqs is built with

cd ~/liboqs
git checkout 7f4c89b2
mkdir build && cd build
cmake -GNinja -DOPENSSL_ROOT_DIR=../.localopenssl_83efcfdfa1_debug -DOQS_MINIMAL_BUILD="SIG_ml_dsa_65" -DOQS_USE_SHA3_OPENSSL=ON -DCMAKE_BUILD_TYPE=Debug ..
ninja

The triggering command is

valgrind --leak-check=full ~/liboqs/build/tests/test_sig ML-DSA-65

[nhorman]

Thank you. Thats interesting, I just tried to recreate with the head of the master branch, and didn't encounter the leak. Given that you reported the issue was introduced with commit 83efcfd (which is a backport of the original commit to the 3.3 branch, that makes me think we fixed something in master that we didn't backport. Let me see if I can find it

[SWilson4]

Thank you. Thats interesting, I just tried to recreate with the head of the master branch, and didn't encounter the leak. Given that you reported the issue was introduced with commit 83efcfd (which is a backport of the original commit to the 3.3 branch, that makes me think we fixed something in master that we didn't backport. Let me see if I can find it

Weird—I'm also getting the leak on the latest master (openssl/openssl@c262cc0).

[nhorman]

hmm, what version of valgrind are you using? I'm on valgrind-3.23.0

[nhorman]

also, what does your openssl.conf look like? Its possible I'm not loading any modules, and as such not triggering the allocation thats leaking

[nhorman]

In fact, I'm sure its my config, I just ran it through gdb and never hit a breakpoint in ossl_rcu_read_lock

[nhorman]

There we go, if I load the oqs provider (duh, should have thought of that), I see the leak. Now to figure out why

[ashman-p]

There we go, if I load the oqs provider (duh, should have thought of that), I see the leak. Now to figure out why

@nhorman, now getting some time to revisit this today. I missed the significance of your statement here.
Could we be tracking two different issues? My test setup does not involve oqsprovider. I just have openssl 3.3 and liboqs. Run the test and ...

Anyhow, I will apply your patch and report back.

[nhorman]

@ashman-p I'm not sure, but I'm sufficiently unfamiliar with oqs to fully understand how you would use both together without making use of the liboqs provider. It's certainly possible that there are two issues here, but the fact that the errors.are identical makes me biased towards thinking the problems are simmilar if not identical

[baentsch]

sufficiently unfamiliar with oqs to fully understand how you would use both together without making use of the liboqs provider

Thanks again for taking the time to look into this @nhorman . liboqs is a crypto library making available for external consumption via a common API various PQ algorithms; some of these PQ algorithms need "common" things like PRNG and SHA3. liboqs therefore has a build option to utilize these common things from OpenSSL's libcrypto. That is what @ashman-p has been testing. liboqs also can be built with "its own" (really, from other upstream sources) common code, offering users an option to build liboqs without OpenSSL dependencies. In that configuration, the problems in this issue don't appear, hinting at least at a wrong interaction pattern between liboqs and libcrypto.

The oqsprovider in turn is a shared library pretty much devoid of crypto functionality (except when implementing hybrid PQ) primarily making available PQ crypto implemented in liboqs to the OpenSSL EVP API via the openssl provider API.

|------------------------------|
|     libssl, openssl apps     |
|------------------------------|
|          EVP API             |
|------------------------------|
|         oqsprovider          |
|------------------------------|

|------------------------------|
|           liboqs             |
|------------------------------|
| PQ alg 1 || .... || PQ alg n |
|------------------------------|
|    openssl    ||   external  |
|   libcrypto   ||  sha3/prng  |
|------------------------------|

[nhorman]

ok, thank you for that.

thats interesting, so liboqs is one of those interesting libraries for which applications may use the provider interface to access oqs, and liboqs in turn loads symbols from openssl to do its work.

The implication here is that we may have two dependencies on openssl (listed via DT_NEEDED entries in the app and the liboqs provider), and those instances may point to different versions of libcrypto. That gets tricky.

to make this a bit more concrete, and pull it back to the issue at hand, what config file does the liboqs instance of libcrypto load? Can you post it here? Dependent on how its built and initalized, it may be that the liboqs instance of libcrypto is loading a "vestigual" provider in this test environment, which is never used, but still triggers the allocation that we are failing to free (which would still point to the thread_stop solution we have above as the 'correct' solution to the problem.

[baentsch]

those instances may point to different versions of libcrypto. That gets tricky.

That absolutely is a problem (known to anyone integrating the whole stack and possibly utilizing distros -- which is not many people :)

But it is not at play here as the issue occurs in "standalone" liboqs; no oqsprovider involved.

The config (build) and config file question for a reproducer I cannot answer: @ashman-p @SWilson4 ?

[nhorman]

@baentsch thats not entirely true, and was the point I was getting at. Even if the top level application doesn't use openssl, and the oqsprovider to access the liboqs library, there is still a path to load the provider. If:

1. The libcrypto.so library gets loaded by liboqs (as your diagram illustrates)
2. The libcrypto.so library is not initalized with the OPENSSL_INIT_NO_LOAD_CONFIG flag (likely if you use auto initalization)
3. The default config file for openssl (or whatever file OPENSSL_CONF points to) contains a section pointing to the oqs provider

Then, when all of those conditions are true, libcrypto will still load the oqs provider, even though the library doesn't use it, which will send you down the path here, and you'll get the leak

So in short, if you have a config file that references the oqs provider (or any external provider, like fips), then this issue will be the result, even if the application being run doesn't actually make use of it

@ashman-p can you identify the openssl.cnf file that you are using for the libcrypto instance that liboqs uses and post it here? That would clarify things greatly.

[baentsch]

there is still a path to load the provider.

Fully agree, @nhorman . It was my strong assumption that such circular setup has been excluded in testing as both @ashman-p and @SWilson4 know about the dependencies (and #1942 (comment) seems to show a standalone build -- and #1942 (comment) absolutely guarantees one).

But you're right in checking this explicitly by asking for the config -- particularly considering the problem never materialized in CI. Also helpful may simply be the output of openssl list -providers in a failing setup.

[ashman-p]

@ashman-p can you identify the openssl.cnf file that you are using for the libcrypto instance that liboqs uses and post it here? That would clarify things greatly.

In my setup it was the vanilla openssl.cnf with the default provider.
apps/openssl list -providers
Providers:
default
name: OpenSSL Default Provider
version: 3.3.3
status: active

[nhorman]

That would do it. the default provider gets loaded via:
module_run->do_load_builtin_modules->OPENSSL_load_builtin_modules->ossl_provider_add_conf_module->CONF_module_add->module_add

which then allocates the lock in question thats leaking

I avoided that initially as I didn't have the default provider activated in my config. Once I loaded the oqs provider (which could have been any provider, it was just the one I chose, as I was working with oqs), the problem began to appear

[SWilson4]

Here's the configuration in my environment. (Added the .txt extension for GitHub upload only.)

openssl.cnf.txt

[nhorman]

yeah, that looks to be the same situation

[ashman-p]

Ok, thank you @nhorman. So, my next question is ... base my tests with "OPENSSL_thread_stop()".
This only worked when called, as Neil originally prescribed. i.e from the thread-based application. Calling it from any centrally used function like oqs_ossl_destroy() did not work. Presumable because the thread is already suspended or ended by that time?

Question: We have 2 work-around options... OPENSSL_thread_stop() and OPENSSL_init_crypto().
Is there any reason not to go with the 'OPENSSL_init_crypto' option? It is simpler to implement rather than changing all the tests apps. Also, this path means we would also need to instruct application writers to also make this call.

[SWilson4]

OK, I made an interesting observation: a similar leak also occurs when liboqs is built with -DOQS_USE_SHA3_OPENSSL=OFF if the algorithm being tested uses one of the other underlying primitives. For example, the leak occurs when using SPHINCS+-SHA2-128f-simple, which calls the OpenSSL SHA2 code.

So, to address Norm's question above:

Thats why I asked the question about differences between these 2 options. It seems to me that the problem would be present regardless.
fetch_ossl_objects(void) in common/ossl_helpers.c calls EVP_MD_fetch() for sha2 and sha3. But, I am not sure where sha3 is used.

It looks like the leak does indeed occur for SHA2 as well. It's just triggered when the hashing code is actually used. (Note that ML-DSA and basically all of the PQ algorithms make extensive use of SHA3.)

[baentsch]

Calling it from any centrally used function like oqs_ossl_destroy() did work.

I suppose this should read "...did NOT work.", right @ashman-p ?

[baentsch]

Is there any reason not to go with the 'OPENSSL_init_crypto' option?

It's easier indeed -- but it'd really be good to understand why it is working. Without this understanding this may haunt us in the future again, say if some side effect in openssl initialization making it work right now changes....

The "thread_stop" solution is clearly understandable and documented, would require 3 code changes in the test programs and one documentation change pointing to OpenSSLs guidance: Would that be too undesirable, @ashman-p ?

[nhorman]

+1 to @baentsch comment. I think the argument to go with the thread_stop approach is because it works, and seems to be fairly well understood at this point. I still don't really see how the OPENSSL_init_crypto approach solves the issue (despite the fact that it does seem to do so). Thats not to say you can't go with that issue, but if I were making the decision, I would at least want to better understand how it solves the problem before electing to go with it.

[baentsch]

It looks like the leak does indeed occur for SHA2 as well.

And does it disappear likewise if OPENSSL_thread_stop is added to the corresponding test, @SWilson4 ?

[ashman-p]

Calling it from any centrally used function like oqs_ossl_destroy() did work.

I suppose this should read "...did NOT work.", right @ashman-p ?

That is correct. It did NOT work. Sorry for the omission.

[ashman-p]

Is there any reason not to go with the 'OPENSSL_init_crypto' option?

It's easier indeed -- but it'd really be good to understand why it is working. Without this understanding this may haunt us in the future again, say if some side effect in openssl initialization making it work right now changes....

The "thread_stop" solution is clearly understandable and documented, would require 3 code changes in the test programs and one documentation change pointing to OpenSSLs guidance: Would that be too undesirable, @ashman-p ?

Yeah, i hear you. I am fine with us doing that. I wondered is @nhorman had any idea why the init calls seem to affect things? The other concern i have is for pre-existing applications. Seems possible that there would be some fallout?

[nhorman]

I'm actually particularly mystified about that. There should be no reason explicitly calling init from the application does anything special. The only thing I can think is that, in doing so, the dynamic loader subtly changes the resolution order of the libraries, so you get the version linked to the application, rather than the one linked to liboqs. If their built with different configs (one with no-atexit, one with atexit, or something like that) you might see this behavior. you might be able to observe that behavior using LD_DEBUG if that were the case

[SWilson4]

The "thread_stop" solution is clearly understandable and documented, would require 3 code changes in the test programs and one documentation change pointing to OpenSSLs guidance: Would that be too undesirable, @ashman-p ?

My preference would be to go with the "thread_stop" solution, for the reasons above. Also, if the leak only occurs because of a flaw in our test programs and not because of a flaw in the library itself, then that's great news, and I would endorse patching it only there.

I do wonder, though, if we ought to protect the user a little more—it seems wrong somehow that somebody who wants to use liboqs in a multithreaded application is required to

1. be aware that we're using OpenSSL under the hood (what if they get liboqs via their package manager and are unaware of the build command used?)
2. call the appropriate OpenSSL (not liboqs) cleanup function in order to avoid a memory leak.

The only way I see to avoid this pain for the user is to add an analogous well-documented OQS_thread_stop function—this could be implemented using our usual #ifdef OQS_USE_OPENSSL approach. Thoughts on this?

It looks like the leak does indeed occur for SHA2 as well.

And does it disappear likewise if OPENSSL_thread_stop is added to the corresponding test, @SWilson4 ?

Yes, it does.

[baentsch]

The only way I see to avoid this pain for the user is to add an analogous well-documented OQS_thread_stop function—this could be implemented using our usual #ifdef OQS_USE_OPENSSL approach. Thoughts on this?

Sounds (more) sensible (than pointing to openssl) -- but doesn't alleviate users of the need to call this at the right time.

Yes, it does.

Thanks for checking/confirming, @SWilson4 !

The other concern i have is for pre-existing applications. Seems possible that there would be some fallout?

I don't see which: The problem also is "pre-existing" -- so either it didn't bother folks (so a fix with a new API they call (or don't :) doesn't change anything) or they already properly cleaned up "behind themselves" as per OpenSSL guidance and only the OQS tests didn't do so far (again, not making a difference for other pre-existing apps).

[ashman-p]

@SWilson4, I can complete the code changes if you haven't already done them. But can you work out the documentation?

[SWilson4]

@SWilson4, I can complete the code changes if you haven't already done them. But can you work out the documentation?

Sounds like a plan, thanks @ashman-p.