After migration to .NET 8(Isolated) from .NET 6(In-Proc), Azure Isolated function (queue triggered) throwing timeout issue. #2510

abkdas123wk · 2024-07-08T06:34:32Z

The Azure function (queue triggered) is hosted in AKS cluster (Linux Debian). We are migrating from .NET 6 to .NET 8, in this process we migrated this in-proc azure function to isolated function. After migration we started receiving timeout exception whenever we submit 20+ messages to the queue quickly. In .NET 6 (In-Proc), we never faced this issue. This queue triggered isolated function makes some http calls, DB calls, does some processing and generates pdf.
These are the settings and packages in .csproj file:
PropertyGroup
TargetFramework net8.0
AzureFunctionsVersion v4
LangVersion 12.0
OutputType Exe
PropertyGroup
ItemGroup
PackageReference Include="Azure.Identity" Version="1.11.2"
PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0"
PackageReference Include="Microsoft.Azure.Core.NewtonsoftJson" Version="2.0.0"
PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0"
PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.3-preview1"
PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0"
PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Blobs" Version="6.4.0"
PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0"
PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1"
PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.4.0"
PackageReference Include="Microsoft.Extensions.Http.Polly" Version="8.0.4"
PackageReference Include="Microsoft.Data.SqlClient" Version="5.2.0"
PackageReference Include="Microsoft.Extensions.Logging" Version="8.0.0"
PackageReference Include="Microsoft.CodeAnalysis.Common" Version="4.9.2"
PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.9.2"
PackageReference Include="NuGet.Packaging" Version="6.9.1"
PackageReference Include="RazorEngineCore" Version="2024.4.1"
PackageReference Include="System.Security.Cryptography.Xml" Version="8.0.0"
PackageReference Include="Microsoft.EntityFrameworkCore" Version="8.0.4"
PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.4"
ItemGroup

Program.cs
public class Program
{
public IConfiguration Configuration { get; private set; }
static async Task Main(string[] args)
{
await CreateHostBuilder(args).Build().RunAsync();
}

public static IHostBuilder CreateHostBuilder(string[] args) =>
    Host.CreateDefaultBuilder(args)
        .ConfigureFunctionsWorkerDefaults(builder =>
        {
            WorkerConfigurationExtensions.UseNewtonsoftJson(builder);
        }, options =>
        {
            options.EnableUserCodeException = true;
        })
        .ConfigureAppConfiguration((hostingContext, config) =>
        {
            var envName = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT") ?? "Production";
            var basePath = Directory.GetCurrentDirectory();
            config.SetBasePath(basePath);
            config.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
                                    .AddJsonFile($"appsettings.{envName}.json", true, true)
                                    .AddJsonFile($"appsettings.{Environment.GetEnvironmentVariable("AxcessEnv")}.json", true, true)
                                    .AddEnvironmentVariables()
                                    .TryAddAzureKeyVault();

            config.Build();
        })
        .ConfigureServices((hostContext, services) =>
        {
            services.Configure<WorkerOptions>(opt =>
            {
                opt.EnableUserCodeException = true;
            });
            services.AddHttpClient();
            services.AddApplicationInsightsTelemetryWorkerService();
            services.ConfigureFunctionsApplicationInsights();
            IConfiguration configuration = hostContext.Configuration;
            services.AddSingleton(configuration);
            var startup = new Startup(configuration);
            startup.ConfigureServices(services);
            services.AddLogging();
        });

}

[Function("Test")]
public async Task Test([QueueTrigger("test", Connection = "TestConnectionString")] Custom custom)
{
try
{
await CreatePdf(custom);
}
catch (Exception ex)
{
await OnFailure(custom, ex);
}
}

The text was updated successfully, but these errors were encountered:

abkdas123wk · 2024-07-08T07:16:55Z

abkdas123wk · 2024-07-08T07:18:15Z

Additional Details from Application Insights:
Microsoft.Azure.WebJobs.Host.FunctionTimeoutException:
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryHandleTimeoutAsync d__35.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:663)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ InvokeWithTimeoutAsync d__33.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:571)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ ExecuteWithWatchersAsync d__32.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:527)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ ExecuteWithLoggingAsync d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:306)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ ExecuteWithLoggingAsync d__26.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:352)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=6.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryExecuteAsync d__18.MoveNext (Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35: D:\a_work\1\s\src\Microsoft.Azure.WebJobs.Host\Executors\FunctionExecutor.cs:108)

Exception Properties
Show less
Event time 7/4/2024, 4:11:36.1631266 PM (Local time)
Message Timeout value of 00:10:00 was exceeded by function: Functions.Test
Exception type Microsoft.Azure.WebJobs.Host.FunctionTimeoutException
Failed method Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryHandleTimeoutAsync d__35.MoveNext
Severity level Error
Telemetry type exception
Operation name Test
Device type PC
Client IP address 0.0.0.0
Role instance pod name
SDK version azurefunctions: 4.34.2.2
Sample rate 1
Problem Id Microsoft.Azure.WebJobs.Host.FunctionTimeoutException at Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+ TryHandleTimeoutAsync d__35.MoveNext
Assembly Microsoft.Azure.WebJobs.Host, Version=3.0.41.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35

bhagyshricompany · 2024-07-09T10:01:39Z

Thanks for reporting .PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" pls update this as updated one. 1.3.2

abkdas123wk · 2024-07-11T11:28:32Z

After updating "Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" from Version="1.2.1" to 1.3.2, still the issue persists. We are still getting timeout exception.

bhagyshricompany · 2024-07-16T06:28:00Z

please use all updated sdk.

yaseen22 · 2024-07-16T06:49:59Z

We have the same issue with Http-trigger Azure function, after migrating from .NET6 to .NET8 Isolated mode, the azure function starts having timeout exception.

And we're using the latest updated packages

abkdas123wk · 2024-07-25T16:28:27Z

We have updated all packages, still we are getting the timeout exception.

bhagyshricompany · 2024-07-30T14:22:19Z

will check please share the updated list of pkg with version and memory you used.Thanks

abkdas123wk · 2024-08-09T09:23:27Z

Below are the list of updated packages:
PackageReference Include="Azure.Identity" Version="1.11.2"
PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0"
PackageReference Include="Microsoft.Azure.Core.NewtonsoftJson" Version="2.0.0"
PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0"
PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.3"
PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0"
PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Blobs" Version="6.4.0"
PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.2.0"
PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.3.2"
PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.4.0"
PackageReference Include="Microsoft.Extensions.Http.Polly" Version="8.0.4"
PackageReference Include="Microsoft.Data.SqlClient" Version="5.2.0"
PackageReference Include="Microsoft.Extensions.Logging" Version="8.0.0"
PackageReference Include="Microsoft.CodeAnalysis.Common" Version="4.9.2"
PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="4.9.2"
PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1"
PackageReference Include="NuGet.Packaging" Version="6.9.1"
PackageReference Include="RazorEngineCore" Version="2024.4.1"
PackageReference Include="System.Security.Cryptography.Xml" Version="8.0.0"
PackageReference Include="Microsoft.EntityFrameworkCore" Version="8.0.4"
PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="8.0.4"

BGbakker · 2024-08-12T12:52:46Z

Hi,
We are experiencing the same issue. After the update of our functions from .net6 in-process to .Net8 isolated, the functions starter throwing timeouts.
We are running 4 functions in a P3V3 app service plan. 2 are time triggered, 2 are event hub triggered. All are affected. The timeout seems to occur when the message ingress is higher than normal.

b-enigma-con · 2024-08-19T07:28:25Z

We have updated all packages, still we are getting the timeout exception.

How does your memory consumption look like?

yaseen22 · 2024-08-19T11:15:52Z

For the Azure functions we didn't check our memory consumption, so I can't tell.

But for our applications, yes after we migrated to .NET 8 we had huge memory consumption that caused us issues.
Check this thread, it's helpful.
dotnet/runtime#95922

What worked for us was moving our Linux distribution image of our containers to be Alpine instead of Debian.

anupamk1506 · 2024-08-20T16:55:18Z

@yaseen22 , Can you please help me with the Alpine base image that you have used for isolated function with .Net 8 ?

yaseen22 · 2024-08-21T11:18:52Z

@yaseen22 , Can you please help me with the Alpine base image that you have used for isolated function with .Net 8 ?

For Azure functions I didn't use the Alpine image, I'm not sure if this is even possible, For Azure function I uses the .NET environment.

The Alpine base image is used for my application at docker image

abkdas123wk · 2024-08-21T11:32:41Z

Hi @yaseen22, can you share the docker file you have used for deployment of azure function?
We have used the base image 'FROM mcr.microsoft.com/azure-functions/dotnet-isolated:4-dotnet-isolated8.0' for our azure function.

MathieuMarchant · 2024-09-05T21:26:11Z

Also running into Timeout exceptions here. Found that this is actually caused by enabling the user code exception, as it is documented in the guide here at the bottom of the logging section:

https://learn.microsoft.com/en-us/azure/azure-functions/dotnet-isolated-process-guide?tabs=windows#logging

.ConfigureFunctionsWorkerDefaults(_ => {}, options =>
    {
        // Makes that exceptions are not wrapped in RpcExceptions
        options.EnableUserCodeException = true;
    })

When commenting out this code, everything works as expected.

But the question is then how do we make sure the exceptions are not getting wrapped?

u4lklusek · 2024-10-16T09:15:57Z

Is there any update around this topic? I have same issue

btw. I do not have the following code piece in my code

.ConfigureFunctionsWorkerDefaults(_ => {}, options => { // Makes that exceptions are not wrapped in RpcExceptions options.EnableUserCodeException = true; })

abkdas123wk added the bug label Jul 8, 2024

bhagyshricompany self-assigned this Jul 9, 2024

bhagyshricompany added Needs: Author feedback and removed bug labels Jul 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

After migration to .NET 8(Isolated) from .NET 6(In-Proc), Azure Isolated function (queue triggered) throwing timeout issue. #2510

After migration to .NET 8(Isolated) from .NET 6(In-Proc), Azure Isolated function (queue triggered) throwing timeout issue. #2510

abkdas123wk commented Jul 8, 2024 •

edited

Loading

abkdas123wk commented Jul 8, 2024

abkdas123wk commented Jul 8, 2024 •

edited

Loading

bhagyshricompany commented Jul 9, 2024 •

edited

Loading

abkdas123wk commented Jul 11, 2024

bhagyshricompany commented Jul 16, 2024

yaseen22 commented Jul 16, 2024

abkdas123wk commented Jul 25, 2024

bhagyshricompany commented Jul 30, 2024 •

edited

Loading

abkdas123wk commented Aug 9, 2024

BGbakker commented Aug 12, 2024

b-enigma-con commented Aug 19, 2024

yaseen22 commented Aug 19, 2024

anupamk1506 commented Aug 20, 2024

yaseen22 commented Aug 21, 2024

abkdas123wk commented Aug 21, 2024

MathieuMarchant commented Sep 5, 2024

u4lklusek commented Oct 16, 2024

After migration to .NET 8(Isolated) from .NET 6(In-Proc), Azure Isolated function (queue triggered) throwing timeout issue. #2510

After migration to .NET 8(Isolated) from .NET 6(In-Proc), Azure Isolated function (queue triggered) throwing timeout issue. #2510

Comments

abkdas123wk commented Jul 8, 2024 • edited Loading

abkdas123wk commented Jul 8, 2024

abkdas123wk commented Jul 8, 2024 • edited Loading

bhagyshricompany commented Jul 9, 2024 • edited Loading

abkdas123wk commented Jul 11, 2024

bhagyshricompany commented Jul 16, 2024

yaseen22 commented Jul 16, 2024

abkdas123wk commented Jul 25, 2024

bhagyshricompany commented Jul 30, 2024 • edited Loading

abkdas123wk commented Aug 9, 2024

BGbakker commented Aug 12, 2024

b-enigma-con commented Aug 19, 2024

yaseen22 commented Aug 19, 2024

anupamk1506 commented Aug 20, 2024

yaseen22 commented Aug 21, 2024

abkdas123wk commented Aug 21, 2024

MathieuMarchant commented Sep 5, 2024

u4lklusek commented Oct 16, 2024

abkdas123wk commented Jul 8, 2024 •

edited

Loading

abkdas123wk commented Jul 8, 2024 •

edited

Loading

bhagyshricompany commented Jul 9, 2024 •

edited

Loading

bhagyshricompany commented Jul 30, 2024 •

edited

Loading