New Relic Initialization Failure in ASP.NET Core 8 Self-Contained Single-File Deployment

[itsbalamurali]

Description
We are encountering an issue when running an ASP.NET Core 8 application in self-contained and single-file mode. The application fails to initialize properly due to missing New Relic environment variables and DLL. The logs show the following errors:

# Build command
dotnet publish "$project" -c Release -o "./publish" -p:PublishSingleFile=true --self-contained -r linux-x64

#Environment Variables
ASPNETCORE_ENVIRONMENT: Production
CORECLR_ENABLE_PROFILING: 1
CORECLR_PROFILER: '{36032161-FFC0-4B61-B559-F6C5D41BAE5A}'
CORECLR_NEWRELIC_HOME: ./newrelic
CORECLR_PROFILER_PATH: ./newrelic/libNewRelicProfiler.so
NEW_RELIC_LICENSE_KEY: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
NEW_RELIC_APP_NAME: Web Portal
NEW_RELIC_REGION: eu
NEW_RELIC_DISTRIBUTED_TRACING_ENABLED: true

[Info ] 2024-06-12 21:04:36 Logger initialized.
[Info ] 2024-06-12 21:04:36 Initializing ThreadProfiler
[Info ] 2024-06-12 21:04:36 ICorProfilerInfo10 available
[Error] 2024-06-12 21:04:36 Unable to find CORECLR_NEWRELIC_HOME or NEWRELIC_INSTALL_PATH environment variables.  Aborting instrumentation.
[Error] 2024-06-12 21:04:36 The New Relic Agent DLL is missing.

Expected Behavior
The application should initialize without errors, with the New Relic agent properly configured and able to perform instrumentation.

Troubleshooting or NR Diag results
We have verified that the necessary environment variables (CORECLR_NEWRELIC_HOME or NEWRELIC_INSTALL_PATH) are not set, which is causing the issue. The New Relic agent DLL is also not included in the single-file bundle.

Steps to Reproduce

1. Create an ASP.NET Core 8 application.
2. Configure the application to run in self-contained and single-file mode.
3. Deploy and run the application.
4. Observe the logs for initialization errors related to New Relic instrumentation.

Your Environment

• ASP.NET Core 8
• Self-contained deployment
• Single-file mode
• New Relic agent version: 10.25.1 (Nuget)
• OS: Linux

Additional context
This issue might be related to how environment variables are handled in single-file applications, and whether the New Relic agent is included correctly in the bundle. Additional guidance on configuring New Relic for self-contained single-file deployments would be helpful.

[workato-integration]

https://new-relic.atlassian.net/browse/NR-281040

[itsbalamurali]

This is what happens when application is not published using single file mode.

NewRelic.Profiler.9993.log
[Info ] 2024-06-14 05:42:51 Logger initialized.
[Info ] 2024-06-14 05:42:51 Initializing ThreadProfiler
[Info ] 2024-06-14 05:42:51 ICorProfilerInfo10 available
[Info ] 2024-06-14 05:42:51 Loading instrumentation from ./newrelic/extensions
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Info ] 2024-06-14 05:42:51 Identified 244 Instrumentation points (not ignored) in .xml files
[Info ] 2024-06-14 05:42:51 Command line: /data01/domeenid/AppApi/AppApi
[Info ] 2024-06-14 05:42:51 Process . with parent process . is not IIS.
[Info ] 2024-06-14 05:42:51 Profiler initialized
[Info ] 2024-06-14 05:42:52 Unable to find Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker for rejit. HR:-2146234064

[tippmar-nr]

@itsbalamurali Thanks for your report.

After a bit of investigation, it looks like the single-file deploy bundles all NewRelic.*.dll files, which is what is causing the problem. The structure of the newrelic folder and subfolders (as shown in the IDE when you add the NewRelic.Agent Nuget package) must be preserved in order for the .NET agent to work properly.

It also appears that the ability to exclude files from the bundle doesn't work well for entire Nuget packages. You might be able to craft a workaround based on the custom build steps mentioned here dotnet/sdk#25411 (comment).

This specific use case isn't something the .NET agent can support natively -- it isn't a problem in the agent but an issue with the implementation of single-file deploy.

[itsbalamurali]

@tippmar-nr This is what happens when application is published in unbundled self-contained mode.

NewRelic.Profiler.log

[Info ] 2024-06-14 05:42:51 Logger initialized.
[Info ] 2024-06-14 05:42:51 Initializing ThreadProfiler
[Info ] 2024-06-14 05:42:51 ICorProfilerInfo10 available
[Info ] 2024-06-14 05:42:51 Loading instrumentation from ./newrelic/extensions
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Warn ] 2024-06-14 05:42:51 extension node not found in instrumentation file. Please validate your instrumentation files against extensions/extension.xsd or contact New Relic support.
[Info ] 2024-06-14 05:42:51 Identified 244 Instrumentation points (not ignored) in .xml files
[Info ] 2024-06-14 05:42:51 Command line: /data01/AppApi/AppApi
[Info ] 2024-06-14 05:42:51 Process . with parent process . is not IIS.
[Info ] 2024-06-14 05:42:51 Profiler initialized
[Info ] 2024-06-14 05:42:52 Unable to find Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker for rejit. HR:-2146234064

OS: Linux

[tippmar-nr]

@itsbalamurali I've been unable to reproduce that result locally on a Ubuntu distro with the NewRelic .NET agent installed via Nuget.

What distro and processor architecture are you running on?

Can you verify the content of the *.xml files in the newrelic/extensions folder after publishing? They should all begin with

<?xml version="1.0" encoding="utf-8"?>
<!--
Copyright 2020 New Relic Corporation. All rights reserved.
SPDX-License-Identifier: Apache-2.0
-->
<extension xmlns="urn:newrelic-extension">