[release/8.0-staging] Disable implicit rejection for RSA PKCS#1 v1.5

[github-actions]

Backport of #95157 to release/8.0-staging

/cc @vcsjones @bartonjs

Customer Impact

OpenSSL introduced a behavior change in EVP_PKEY_decrypt where RSA decryption with PKCS#1 v1.5 padding no longer produces an error with invalid padding. Instead, it returns “random” data.

Customers using affected versions of OpenSSL will no longer get consistent and documented behavior from RSA.Decrypt.

This change is reacting to a breaking behavioral change in a dependency, OpenSSL, so that RSA.Decrypt continues to function as it is documented to, and to be consistent with Windows and other platforms.

Testing

This behavior change was identified with a test in #95115. Additional tests were added.

Risk

Low. This is reacting to a change in OpenSSL by settings a particular flag. Existing tests cover RSA.Decrypt’s behavior.

[ghost]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Backport of #95157 to release/8.0-staging

/cc @vcsjones

Customer Impact

Testing

Risk

IMPORTANT: If this backport is for a servicing release, please verify that:

• The PR target branch is release/X.0-staging, not release/X.0.

• If the change touches code that ships in a NuGet package, you have added the necessary package authoring and gotten it explicitly reviewed.

Author:	github-actions[bot]
Assignees:	-
Labels:	area-System.Security
Milestone:	-

[neverpanic]

Just to make sure that this doesn't end up being merged just because nobody saw the original comment: This re-enables a Bleichenbacher timing oracle attack against PKCS#1v1.5 decryption. See https://people.redhat.com/~hkario/marvin/ for details and #95157 (comment) for a comment by the researcher who published the vulnerability and proposed the change in OpenSSL.

I don't think this should be merged.

[carlossanlop]

^ There's some feedback to address. @dotnet/area-system-security PTAL.

Friendly reminder that Tuesday 16th 4pm is the Code Complete deadline for the February Release. Please merge before that date and time to ensure this fix gets included in that Release.

[GrabYourPitchforks]

The .NET release management and security teams discussed this both amongst ourselves and with representatives from Red Hat. (Thank you again Red Hat for being gracious with your time!)

The conclusion that .NET drew is that we believe it is appropriate to proceed with the downlevel backport of this change. This decision is based specifically on .NET customer scenarios, our commitment to compatibility in servicing, and what we believe an appropriate default security stance is for the product.

Our conversation with Red Hat identified future opportunities for improvement, including potential API changes, static code analysis rules, and perhaps even behavioral changes. These discussions can continue throughout net9 development, but any proposals would only affect the behavior of net9 and beyond.

The .NET team owns this decision, and we speak only for our own framework. We don't speak for other languages and frameworks, who are of course free to pursue a different strategy in regard to the reported issue.