6.3.10 timeout issues #1361
Comments
|
I've turned on trace logging for our test instance and will post the logs once we have them. This might give more insight into why these endpoints are timing out |
|
We are running to same situation on all of our environments. We had this issue when we used pre-release version as well as public released version. Rebooting seems to resolve the issue, but issue seems to come back after few days and seems consistent. What is worth noting is that we rollbacked version to 6.3.8 and issue was gone. What I have noticed is that issue starts occurring mostly during UTC mid-night on lower environment as well as production as far as I can tell. I hope this might help investigate. |
|
Thanks for this report. We're investigating. My initial suspicion is that this is related to 6.3.9's update of our dependencies on ASP.NET framework packages. In that version, we updated framework packages from version 6.0.0 to version 6.0.26. This updates our transitive dependency on the System.IdentityModel.Tokens.Jwt and Microsoft.IdentityModel.JsonWebTokens packages past versions that have a known Denial of Service vulnerability. @sanket-mistry-jm it would be extremely helpful if you could try to reproduce the issue in your environment with the 6.3.9 build of IdentityServer. In the meantime, since this is preventing application of the hotfix, my recommendation is to apply the workarounds that we describe in the advisory - basically your UI code should not rely on the interaction service to determine if urls are local. You should use the IsLocalUrl api from ASP.NET instead. |
|
Hi @josephdecock, thanks for looking into this. Is it likely that the same issue effects 7.X.X versions? We are planning to upgrade to dotnet8 and v7 in the very near future. |
|
@josephdecock, I will see what I can do and report back here. |
|
FYI, we ran into the same issue again on Production. Again, issue started at almost UTC 12:00 today. |
|
Our test instance started timing out again today, here are the logs.... Logs from a request that times out Logs from a successful request for comparison. |
|
Just wanted to update everyone that we are continuing our investigation. We don't have a fix yet unfortunately, but we'll keep you all updated as we go. |
|
The picture that I'm getting is that
Has anyone in this thread seen it happen at other times of day? |
@josephdecock , for us prettymuch all endpoints start delaying by x seconds eventually going to x minutes. in our case, /connect/authorize works fine and show login UI but post login action it times-out when the issue occurs. I am thinking may be something related to how tokens are signed, or keys are generated/regenerated? We were able to capture .NET Profiler Trace, and we see this.... |
|
Hey @josephdecock, we're also experiencing this issue since around two weeks now. But we are using different versions as reported.
Besides different versions and times, the issue is exactly the same as reported. Only a restart helps.
|
|
If anyone would be able to capture Otel Traces of the timeout that could help us understand the root cause. https://docs.duendesoftware.com/identityserver/v7/diagnostics/otel/traces/ |
|
We're currently using the Elastic APM agent to collect Traces. Unfortunately, it seems that we dont get any logs when the issue comes up. |
|
@Stumm304 Thank you for getting back with that information. We have similar reports from other customers. May I ask what log levels you have enabled? |
|
It happened in our dev environment today at around 11:45 UTC+2. |
|
Hi, we're running Duende 7.0.4 and .NET 8 in our Prod environment and we believe this is happening for us also. @AndersAbel could you please advise on which Otel traces you're interested in capturing, is it all those referenced in https://docs.duendesoftware.com/identityserver/v7/diagnostics/otel/traces/ ? Do we have an update on this issue from your investigations? |
Yes, that's what we're looking for.
Unfortunately, we don't yet have a fix. We're continuing to work on the problem though. @techyian can you give us more history please? Was a previous deployment working? |
Can anyone advise on this please? with dotnet6 approaching EOL, we will need to upgrade to dotnet 8. Another thing of note is, most versions have now been marked as deprecated on nuget because of the vulnerability. We treat warnings as errors in our build pipeline so would need to disable this to be able to use a previously stable version.
@josephdecock you mentioned an update of dependencies as a possible cause, is there anyway to confirm this? |
|
We are seeing this issue on |
|
Hi everyone, I wanted to give you all an update to let you know that we're working hard on this issue, and it is in fact priority 1 for us now. If anyone has an environment where the issue is occurring that they are willing to show to the Duende engineering team, I would be very interested in a troubleshooting call with you. Please email me ([email protected]) if you're able to do so. Other things that would be helpful include:
|
|
@philipwindsora55 We do have reports of this issue affecting both the 6.3.x and 7.0.x release branches. We're still investigating a number of possibilities of what the cause of the issue might be, but one of the few things that did change recently in both of those release branches is that we updated our dependency on the Microsoft.IdentityModel libraries. Are you able to show a before and after of those dependencies? So, first, on the commit that you deployed to your dev and test environements that caused the issue, run |
|
@keithlfs Do you use Linux or Windows as the hosting OS in Azure? We have reports of this happening on Linux hosts, is there anyone affected that hosts on Windows? |
|
@josephdecock I'm happy to setup a troubleshooting call with you, i'll send you an email. Another point to note is, we have up-time monitors in place which keeps the app alive 24/7. As for configuration.... we have deployment slots so need to store keys in a location both instances can access: Our Identity configuration is: @AndersAbel We are hosting on Windows/IIS in Azure |
@josephdecock , Here is the output requested. ` v6.3.10 - Where issue occurs
`v6.2.3 - Where issue did not occur.
Information about usage of Automatic signing keys, including Q: How many are in the store? Q: Which store implementation are you using (Duende.IdentityServer.EntityFramework or something custom)? Let me know if this helps. |
|
We are seeing this issue using Windows hosting in Azure App Services.
Anecdotally, our tester thinks it occurs when we login with the same account simultaneously (multiple sessions) but this could be pure conjecture/coincidence.
…________________________________
From: Anders Abel ***@***.***>
Sent: Thursday, August 29, 2024 7:16:29 PM
To: DuendeSoftware/Support ***@***.***>
Cc: Keith Lawrence ***@***.***>; Mention ***@***.***>
Subject: Re: [DuendeSoftware/Support] 6.3.10 timeout issues (Issue #1361)
@keithlfs<https://github.com/keithlfs> Do you use Linux or Windows as the hosting OS in Azure?
We have reports of this happening on Linux hosts, is there anyone affected that hosts on Windows?
—
Reply to this email directly, view it on GitHub<#1361 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ARLCX6OW7OTDRGWAKO7VCH3ZT5QP3AVCNFSM6AAAAABMCIOKMWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJYGU2TCMBZGY>.
You are receiving this because you were mentioned.Message ID: ***@***.***>
Disclaimer : Fundsmith LLP is the manager of the Fundsmith Equity Fund, the Fundsmith Sustainable Equity Fund and Smithson Investment Trust PLC. Fundsmith LLP is authorised and regulated by the Financial Conduct Authority, FCA reference number 523102. The contents of this email do not constitute or form part of any offer or invitation to subscribe for, or any solicitation of any such offer to subscribe for, any securities in Fundsmith Equity Fund, Fundsmith Sustainable Equity Fund or Smithson Investment Trust PLC. Further information in relation to Fundsmith Equity Fund and the Fundsmith Sustainable Equity Fund can be found at www.fundsmith.co.uk and in relation to Smithson Investment Trust plc at www.smithson.co.uk. Past performance is not necessarily a guide to future performance. The value of investments and the income from them may fall as well as rise and be affected by changes in exchange rates, and you may not get back the amount of your original investment. Fundsmith LLP does not offer investment advice or make any recommendations regarding the suitability of its products. Any financial promotion contained in this email is intended for UK residents only and is communicated by Fundsmith LLP. Fundsmith LLP is a limited liability partnership incorporated in England with registered number OC354233 and registered office address at 33 Cavendish Square, London, W1G 0PW. This email and any attachments are confidential and intended solely for the addressee and may also be privileged or exempt from disclosure under applicable law. If you are not the addressee, or have received this email in error, please notify the sender immediately, delete it from your system and do not copy, disclose, disseminate, forward or otherwise act upon any part of this email or its attachments. Any email which you send to, or receive from Fundsmith LLP will be stored on its systems, and may from time to time be monitored for internal business compliance purposes. In addition, this email, and any email that you send to, or receive from Fundsmith LLP may contain personal data. Further information in relation to how Fundsmith LLP might use your data can be found at www.fundsmith.co.uk/privacy-policy. The integrity and security of this message cannot be guaranteed and it may be subject to interception, loss, delivery delay, alteration, corruption and unauthorised amendment, for which Fundsmith LLP accepts no liability. We do not accept liability for any viruses which may be transmitted by or with this e-mail message. Recipients are expected to take their own steps to ensure that e-mail messages are checked for, and free from, viruses.
|
|
@josephdecock please can we have an update on this issue. |
|
I wanted to check if anyone who has reported was able to bypass it or are we still seeing the same issue? For us, it is still consistent. |
|
@AndersAbel @josephdecock please may we have an update? This is a critical issue and we're stuck between reverting to a known insecure version or having arbitrary application crashes in production. It would be great to have an understanding of what is being done by Duende to investigate and rectify this issue. |
|
Hi Keith, I am really sorry for the radio silence. The engineering team is working on it, but it is very hard to consistently reproduce. Are you saying that by reverting back to an earlier version (which exact version) the problem goes away? thanks! |
|
Understood Dominick, thanks for the update.
I haven’t (yet) tried to roll back, but previously we were on 6.3.8 and didn’t experience this issue.
|
|
In case it helps others: I was seeing the issue every 2-3 days, but it has not recurred in the past two weeks. My environment: IdentityServer 7.06, Azure SQL Server, ASP.NET Core 8 hosted in Azure container app. I made the following three changes more-or-less simultaneously, and haven't seen a failure since:
|
|
Echoing @DanBlumenfeld, we haven't seen a timeout in a few weeks which is strange considering how frequently it was occurring. Could this be a hosting provider issue, are we all hosted in Azure in this thread? |
|
For us it is a different story....We rollbacked to v6.3.8 and we are seeing the same issue on that version as well. So We are unsure what is happening. |
|
We are investigating if the hang is related to data protection and specifically the Azure data protection services. For those affected, do you use Azure.Extensions.AspNetCore.DataProtection.Blobs? In that case, what version? |
|
Azure.Extensions.AspNetCore.DataProtection.Keys: 1.2.3 |
|
|
Hello Everyone. We found the issue is related to specific version of Do not forget, you may have another package that has dependency to this one (Our case) Force update |
|
Thank you @OmidID for sharing that information. The information you linked to is consistent with the version numbers reported above. @keithlfs @mcolebiltd @DanBlumenfeld Could you please try updating |
Updated and am deploying now. In my case, I've not seen the issue recur since the changes I mentioned above, so hopefully this makes no difference :-) |
|
We have not experienced a timeout issue for a good few weeks now, maybe it was an Azure infrastructure issue? I've updated the references to our Azure packages too as suggested above and again, no issues so far. I'll continue to monitor for another few weeks and update. |
|
@philipwindsora55 Thanks for reporting back. So far everyone that has reported this issues have been running on Azure, or at least using Azure services. At this point we strongly suspect that this is/was an Azure issue - either with the Azure services or the Azure SDK (most likely the Azure.Core library). |
Which version of Duende IdentityServer are you using?
6.3.10
Which version of .NET are you using?
6.0
Describe the bug
We patched IdentityServer as per the CVE. We've gone from 6.3.5 to 6.3.10.
I'm not saying for sure that this patch has started causing issues but they only started happening since I applied the patch so I'm reaching out to see if anyone else has experienced issues.
Our UI is working fine, we can login and edit details however, the following endpoints are timing out:
/.well-known/openid-configuration
/.well-known/openid-configuration/jwks
The timeouts have occurred across our dev and test instances the past few days. I've just rebooted out test instance and .well-known/openid-configuration is loading again.
There are no logs showing in Application Insights, I'm also in the process of raising a ticket with azure support.
The text was updated successfully, but these errors were encountered: