High memory consumption of .Net 6 app on Linux

[denislohachev1991]

I encountered a very serious problem when switching from Windows to Linux (Ubuntu). The problem is very high memory consumption. I first noticed an increase in memory consumption on Windows when moving from .Net Framework to .Net Core 5. But then this was not much of a problem and was acceptable since the increase was not large. The .Net Framework application consumed approximately 150-200 megabytes and the .Net Core 5 application consumed approximately 250-350 megabytes. But when I switched to Linux, the consumption increased many times over. To make the transition, I switched to .Net 6. After running the application on Linux, I noticed that memory grows over time and does not decrease; with long-term operation of the application, memory consumption can reach 6-8 gigabytes. After doing a memory dump, I noticed that almost 90% was taken up by unmanagement memory. There are about 50 instances of this application running on my server, and they quickly consume all the memory, which leads to application crashes. I have no idea why this is happening, it seems that Garbage collection is not freeing the reserved memory. For information, on Windows under IIS the application worked in 32-bit on Linux in 64-bit.

I tried changing Garbage collection settings such as (DOTNET_GCConserveMemory and DOTNET_gcServer), but it didn't help.

My application is essentially a CMS hosted using nginx. Basically this is a site using liquid (https://mtirion.medium.com/using-liquid-for-text-base-templates-with-net-80ae503fa635) to parse content. It also runs several hosted services that periodically run tasks such as retrieving data from the database to send email if any, checking for changes in data in a file. Even when no requests are sent to my site (for example at night), memory usage does not decrease. I have done a few things like getting memory dump, dotnet-gcdump, dottrace and dotnetdump but I am not very knowledgeable about it and can provide as per need.

[ghost]

Tagging subscribers to this area: @dotnet/gc
See info in area-owners.md if you want to be subscribed.

Issue Details

---

I encountered a very serious problem when switching from Windows to Linux (Ubuntu). The problem is very high memory consumption. I first noticed an increase in memory consumption on Windows when moving from .Net Framework to .Net Core 5. But then this was not much of a problem and was acceptable since the increase was not large. The .Net Framework application consumed approximately 150-200 megabytes and the .Net Core 5 application consumed approximately 250-350 megabytes. But when I switched to Linux, the consumption increased many times over. To make the transition, I switched to .Net 6. After running the application on Linux, I noticed that memory grows over time and does not decrease; with long-term operation of the application, memory consumption can reach 6-8 gigabytes. After doing a memory dump, I noticed that almost 90% was taken up by unmanagement memory. There are about 50 instances of this application running on my server, and they quickly consume all the memory, which leads to application crashes. I have no idea why this is happening, it seems that Garbage collection is not freeing the reserved memory. For information, on Windows under IIS the application worked in 32-bit on Linux in 64-bit.

I tried changing Garbage collection settings such as (DOTNET_GCConserveMemory and DOTNET_gcServer), but it didn't help.

My application is essentially a CMS hosted using nginx. Basically this is a site using liquid (https://mtirion.medium.com/using-liquid-for-text-base-templates-with-net-80ae503fa635) to parse content. It also runs several hosted services that periodically run tasks such as retrieving data from the database to send email if any, checking for changes in data in a file. Even when no requests are sent to my site (for example at night), memory usage does not decrease. I have done a few things like getting memory dump, dotnet-gcdump, dottrace and dotnetdump but I am not very knowledgeable about it and can provide as per need.

Author:	denislohachev1991
Assignees:	-
Labels:	area-GC-coreclr, untriaged, needs-area-label
Milestone:	-

[danmoseley]

almost 90% was taken up by unmanagement memory

Do you have an idea what is allocating that? Are you loading any native libraries?

You mention the GC - do you have reason to believe the GC is involved? Eg you are expecting something to be finalized and it's not. Generally unmanaged heap would not be a GC issue.

[denislohachev1991]

I have no idea what fills the memory. But I don’t understand why this only happens on Linux; it works fine on Windows. The difference is that on Windows the application runs under .Net 5, and on Windows under .Net 6.

[janvorli]

@denislohachev1991 please see my comment here on a similar issue: #95922 (comment).
Also, if you would be able to run your application under the heaptrack tool on Linux (just by running heaptrack /path/to/your/app so that it collects everything from the start), it could help pinpoint where the native memory is coming from. Ideally, before running it, download symbols for all the libXXXX.so files in the .NET runtime so that the symbol files are placed next to the relative .so files. The dotnet-symbol tool can do that - just run it on the .so files and it will place the symbol files next to them. Please see https://github.com/dotnet/diagnostics/blob/main/documentation/debugging-coredump.md on how to install the dotnet-symbol tool.
After collecting the heaptrack trace, you can run heaptrack_print -l 1 -a 0 -T 0 -p 0 -f heaptrack_log_file to dump the leaks it has found including the call stacks of the allocations.

[denislohachev1991]

@janvorli I’m not very good at this, in the comment to which you refer, I learned that I can share the dump file with you so that you can conduct your investigation. I created this dump file while the application was running and it was using a lot of memory. If I understand you correctly, tell me how I can give you the dump file.

[janvorli]

@denislohachev1991 you can place it to your onedrive or google drive or something like that and then send me a private email with details on how to access it. My email address my github name @ microsoft.com.

[beckjin]

I seem to have encountered a similar problem, have you resolved it?

[mangod9]

Is this on .NET 6 only, or does the same issue repro with 8?

[denislohachev1991]

@mangod9 After moving to .Net 8, the problem has decreased a little compared to .Net 6. But it still consumes much more memory than on a Windows server.

[petersinsky]

@janvorli I have encountered a similar problem in our microservice. On Windows, the app works fine, but on Linux, the memory usage keeps increasing. I gathered some dumps using heaptrack and dotnet-dump and sent them to your MS email. Please let me know if you could take a look. Thank you very much.