Mini-dump thread stack frames are missing

[iliamosko]

After upgrading our dump parsing application to CLRMD v2.0, and attempting to parse a mini-dump, we are seeing missing stack frames for some threads. The mini-dump was taken from a process that is running on Ubuntu-20.04 and is also being parsed on the same system.

Here are some screenshots of a thread and its stack frame traced using Dotnet-dump analyze, a sample clrstack class file provided in the CLRMD repository and our custom tracing application running on CLRMD v1.1.

Any idea as to what is causing this issue?

Dotnet-dump analyze:

CLRMD v2.0 sample dump parser:
https://github.com/microsoft/clrmd/blob/124b189a2519c4f13610c6bf94e516243647af2e/src/Samples/ClrStack/ClrStack.cs
(*** The only modification done to this class was that I removed the parsing of DumpStackObjects so that there are only thread callstacks in the output.)

Application running CLRMD v1.1:

[leculver]

This is typically due to not being able to properly find PE images for the files in question. We have a System.Diagnostics.Trace.WriteLine call in ClrMD (both versions I think) which prints out messages, for example: https://github.com/microsoft/clrmd/blob/master/src/Microsoft.Diagnostics.Runtime/src/Implementation/SymbolServerLocator.cs#L239

clrmd/src/Microsoft.Diagnostics.Runtime/src/Implementation/SymbolServerLocator.cs

Line 271 in 4051dff

System.Diagnostics.Trace.WriteLine(msg);

You can see these in Visual Studio's debugger events, or you can connect a standard listener to Diagnostics.Trace to write out to console. Can you collect that data and post it here? That should explain what's going on.

[iliamosko]

I've ran the parser with a listener connected, but I received no Trace messages from the SymbolServerLocator. I've also tried to load the binaries before enumerating through the threads.

        foreach (var module in dataTarget.EnumerateModules())
        {
            Console.WriteLine($"Finding binary: {module.FileName}");
            var binaryLocation = dataTarget.BinaryLocator.FindBinaryAsync(module.FileName, module.IndexTimeStamp, module.IndexFileSize, true);
            Console.WriteLine($"FindBinary output: {binaryLocation.Result}");
        }

Each module that was ran through the BinaryLocator returned the module location so I dont think that there are files missing(?).

I've also noticed that the ability to set a symbol path to the SymbolLocator has been removed, instead it is checking for environment variables set beforehand. This could be the cause of the problem if there are any binaries that are not found and are needed to be downloaded from the server, but this doesn't hold since CLRMD 2.0 parses our full-dumps without a problem.