Tips on using the RequestId from API GW as the TraceIdentifier?

[brendonparker]

It seems that API GW creates a RequestId, which is then used by X-Ray to correlate logs. However, the RequestId (which I believe is the TraceIdentifier on the dotnet side) in the logs seems to get generated separately by dotnet. Is there a way to make these line up? To make the dotnet TraceIdentifier match the RequestId? Ideally when looking at logs for a trace, I see more than the plain text logs like START RequestId and END RequestId.

I'm using Serilog with json formatting, if that matters.

I'm not sure to open this as an issue/feature-request, or if there is some other resolution.

EDIT: I've tried manipulating the the Amazon.Lambda.AspNetCoreServer.Hosting library to set the TraceIdentifier using the IHttpRequestIdentifierFeature. At first I thought that solved it, but now it appears there are two id's in play... I'm getting a different id from the APIGatewayProxyRequest.RequestContext.RequestId and what is shown in the START RequestId and End RequestId log messages surrounding my application logs. 🤔

EDIT: I also tried setting the LAMBDA_NET_SERIALIZER_DEBUG environment variable. Which also confirms that the RequestId on the proxy request is different than the request id that is logged just before it. 🤔 🤔

EDIT EDIT: I think I understand why the RequestId is different. It appears that the initial request comes into the lambda, which then bootstraps the aspnet host context, which then handles the request, creating a new RequestId. If I use ILambdaContext.AwsRequestId, then the RequestId "sticks" to whatever it was on the initial request. I'm wondering if chasing the use of the RequestId is a bad path, and if I should instead be using the TraceId?

EDIT EDIT EDIT: So, disregard part of above. Using the ILambdaContext.AwsRequestId does work as expected. However, in the AWS Console, when viewing an x-ray trace, it doesn't pick up the log messages. Apparently it is looking for an @requestId property... If you click the "View in CloudWatch Log Insights" button, it shows the query it is using:

fields @log, @timestamp, @message
| filter @requestId = "8b47febb-1c3a-4fb9-a52a-4a90d1ff1291" or @message like "1-625ddb6e-7a76d2656609a42008a77ab6"
| sort @timestamp, @message desc

The first is the ILambdaContext.AwsRequestId, the second is the X Ray Trace Id. The X Ray Trace Id seems more flexible, as it looks for that string anywhere in the message, where the @requestId is very specific to that property. My problem is the serilog formatter logs the object like:

{
    "@timestamp": "2022-04-18T21:43:10.2386435+00:00",
    "level": "Information",
    "messageTemplate": "",
    "message": "",
    "fields": {
        "EventId": {
            "Id": 1,
            "Name": "ObjectResultExecuting"
        },
        "SourceContext": "Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor",
        "ActionId": "a5661366-bf15-4639-8425-359ea48c01b0",
        "ActionName": "App.Web.Controllers.Controller.Get (App)",
        "RequestId": "Root=1-625ddb6e-7a76d2656609a42008a77ab6",
        "RequestPath": "/prod/test"
    }
}

So I suppose I want one of the following options...
Option A: To use the X-Ray Trace Id as the aspnet TraceIdentifier
Option B: A way to control the serialization of that log message, such the ILambdaContext.AwsRequestId value goes to @requestId instead of fields.RequestId
Option C: A way to customize the query that the X Ray trace logs is using in the AWS console.

I'm wondering if this should become a feature request issue? I see a few desirable features:

1. Add TraceId to Amazon.Lambda.Core.ILambdaContext. Looks like there is already a TODO there: "If/When Amazon.Lambda.Core is major versioned, add this to ILambdaContext."
2. Enhance Amazon.Lambda.AspNetCoreServer.Internal.InvokeFeatures to also implement: Microsoft.AspNetCore.Http.Features.IHttpRequestIdentifierFeature. I'm not sure what value is most correct to use here. Either the ILambdaContext.AwsRequestId or the newly added ILambdaContext.TraceId from above. Perhaps is configurable?

[ashishdhingra]

@brendonparker Thanks for raising the feature request #1147 and starting the discussion. The said feature request needs to be reviewed by team. I'm unsure if there is currently a way to change this mapping.

[ashishdhingra]

@brendonparker BTW, did you look at AWS X-Ray SDK for .NET, which is maintained by X-Ray team?

[normj]

Thanks for the great feature request. I like the idea of having the HttpContext.TraceIdentifier match the X-Ray trace ID. I think we should aim to do that.

As far as customization goes in the older style of writing ASP.NET Core Lambda functions you used to be able to override methods from a base class to customize the marshalling between API Gateway and ASP.NET Core. That ability got lost in the new approach using the AddAWSLambdaHosting extension method to initialize Lambda. My plan is to allow Action callbacks to be added during the AddAWSLambdaHosting method to provide back the old customization features.

[normj]

Closing this QA and will continue the conversation on actual feature request feature.

#1148