Error: Unable to assume role

[tarunm97]

Hi,

I have followed the instructions as mentioned in the doc and created a role having the [AdministratorAccess] policy attached and I am using the eks to achieve this. already the oidc provider is added in the iam and then I have created the trusted-relationship as well.

Plugin.yaml file

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: aws-request-signing-example
plugin: aws-request-signing
config:
  aws_assume_role_arn: arn:aws:iam::********229913:role/test-role-tarun-kong-plugin
  aws_assume_role_name: test-role-tarun-kong-plugin
  aws_region: ap-south-1
  aws_service: s3
  override_target_port: 443
  override_target_protocol: https

and when I am using it I am getting the following error in the kong logs

2023/09/20 17:15:33 [error] 1110#0: *881 [kong] handler.lua:70 [aws-request-signing] failed to get from node cache: {"error":"handshake failed","message":"Unable to assume role [arn:aws:iam::******29913:role/test-role-tarun-kong-plugin]"}, client: 10.160.14.203, server: kong, request: "GET /index.html HTTP/1.1", host: "tarun.beta.*********.com"
2023/09/20 17:15:33 [error] 1110#0: *881 [kong] init.lua:290 [aws-request-signing] /opt/kong/plugins/aws-request-signing/handler.lua:136: code must be a number, client: 10.160.14.203, server: kong, request: "GET /index.html HTTP/1.1", host: "tarun.beta.************.com"
10.160.14.203 - - [20/Sep/2023:17:15:33 +0000] "GET /index.html HTTP/1.1" 500 42 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
2023/09/20 17:15:34 [error] 1110#0: *881 peer closed connection in SSL handshake, client: 10.160.14.203, server: kong, request: "GET /index.html HTTP/1.1", host: "tarun.beta.************.com"
2023/09/20 17:15:34 [error] 1110#0: *881 [kong] handler.lua:70 [aws-request-signing] failed to get from node cache: {"error":"handshake failed","message":"Unable to assume role [arn:aws:iam::********:role/test-role-tarun-kong-plugin]"}, client: 10.160.14.203, server: kong, request: "GET /index.html HTTP/1.1", host: "tarun.beta.*********.com"
2023/09/20 17:15:34 [error] 1110#0: *881 [kong] init.lua:290 [aws-request-signing] /opt/kong/plugins/aws-request-signing/handler.lua:136: code must be a number, client: 10.160.14.203, server: kong, request: "GET /index.html HTTP/1.1", host: "tarun.beta.***********.com"
10.160.14.203 - - [20/Sep/2023:17:15:34 +0000] "GET /index.html HTTP/1.1" 500 42 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/117.0.0.0 Safari/537.36"
127.0.0.1 - - [20/Sep/2023:17:15:34 +0000] "GET /status HTTP/2.0" 200 1218 "-" "Go-http-client/2.0"

Not sure is there anything else I need to configure....

[tarunm97]

Above one is fixed, it is happening because call to sts is not allowed in the service-mesh, I have done that, now facing this error

proxy 2023/09/20 18:08:16 [error] 1109#0: *961 [kong] handler.lua:70 [aws-request-signing] failed to get from node cache: {"sts_status":400,"message":"Unable to assume role [arn:aws:iam::500*******:role/test-role-tarun-kong-plugin]","sts_body":{"RequestId":"d11fcfe0-2640-4b46-b654-bd38b13e95df","Error":{"Message":"1 validation error detected: Value at 'webIdentityToken' failed to satisfy constraint: Member must not be null","Type":"Sender","Code":"ValidationError"}}}, client: 10.160.15.180, server: kong, request: "GET /index.html HTTP/2.0", host: "tarun.beta.**********.com"

Can you help @DanielRailean

[DanielRailean]

Hey there, the error looks a lot like you are not sending the token to the sts, the info you provided is unfortunately not enough for me to debug anything.

I would at least need the request/response timeline (in text ex. from insomnia) for a start.

[tarunm97]

Basically what I have done, created the iam role in with the oidc provider and permissions as mentioned above, then just created the kong-plugin and added the same in the kong-helm-chart as well, so it will be get mounted inside the pods at the path /opt/kong/, where it will be get detected, all the setup looks good, because when I am requesting the call, it is going to the plugin code and then it is showing the error above, not sure what is breaking, I am doing this for the S3.,

Request I am doing:-

curl https://tarun.prod.######.com which will call the s3

Routing Flow:-

tarun.prod.#####.com[Maps to Route53]-----> alb ----> kong pods----> ingress for Tarun.prod.####.com---->service with external-name [s3-vpc-endpoint.prod.*********.com],

what is it this s3-vpc-endpoint ???, you can find in the link attached at the below..

This setup is working fine when the bucket is not using the kms as encryption, because AWS SIGN v4 is not required if bucket is using the SSE-S3 default s3 encryption, but when you choose the SSE-KMS, it requires aws sign v4, if I have enabled the ss3-kms and going without this plugin getting the issue, check this sample image, this is the exact same error i am getting.

For More-Info about the setup you can check this link:- https://aws.amazon.com/blogs/networking-and-content-delivery/hosting-internal-https-static-websites-with-alb-s3-and-privatelink/

[tarunm97]

@DanielRailean, do you got any chance to check this ?

[DanielRailean]

Hey there. Sorry for the late reply.

Maybe I'm wrong, but I feel like your question is a bit out of scope for this plugin as it mostly touches your AWS infrastructure rather than the plugin functionality.

In the end, it's just as simple as :

1. Get aws-sts temp. credentials using a token. (code)
2. Sign the request using the credentials above. (code)

Step one can be manually replicated using the following curl.

curl --request GET \
  --url 'https://sts.amazonaws.com/?Action=AssumeRoleWithWebIdentity&DurationSeconds=3600&RoleArn=your-role-arn&RoleSessionName=your-role-name&Version=2011-06-15&WebIdentityToken=your-token' \
  --header 'Accept: application/json'

Step two can be manually replicated by using https://datafetcher.com/aws-signature-version-4-calculator or any other sigv4 creator, ex. the one in the insomnia HTTP client.

Let me know if you can manually perform the operations and the request is successful, then we can investigate further issues within one of the 2 steps above, otherwise, it's most likely some problems with the underlying AWS infrastructure.

[garciasdos]

Hey @tarunm97, did you happen to solve this? I am facing a similar error but unsure how to solve it. Thanks!

[DanielRailean]

@tarunm97 I will assume your issue is solved and thus will close the issue in 1 week.

[rubenbaraut]

@tarunm97 same problem here, and this is the message received if we enable the return_aws_sts_error:

{
    "stsResponse": {
        "Error": {
            "Code": "ValidationError",
            "Message": "1 validation error detected: Value at 'webIdentityToken' failed to satisfy constraint: Member must not be null",
            "Type": "Sender"
        },
        "RequestId": "XXXXX"
    },
    "message": "Unable to assume role [arn:aws:iam::XXXXX]"
}

[DanielRailean]

Hey there. Sorry for the late reply.

Maybe I'm wrong, but I feel like your question is a bit out of scope for this plugin as it mostly touches your AWS infrastructure rather than the plugin functionality.

In the end, it's just as simple as :

1. Get aws-sts temp. credentials using a token. (code)
2. Sign the request using the credentials above. (code)

Step one can be manually replicated using the following curl.

curl --request GET \
  --url 'https://sts.amazonaws.com/?Action=AssumeRoleWithWebIdentity&DurationSeconds=3600&RoleArn=your-role-arn&RoleSessionName=your-role-name&Version=2011-06-15&WebIdentityToken=your-token' \
  --header 'Accept: application/json'

Step two can be manually replicated by using https://datafetcher.com/aws-signature-version-4-calculator or any other sigv4 creator, ex. the one in the insomnia HTTP client.

Let me know if you can manually perform the operations and the request is successful, then we can investigate further issues within one of the 2 steps above, otherwise, it's most likely some problems with the underlying AWS infrastructure.

hey @rubenbaraut , could you perform the steps from my reply above and get back with the results ?

[DanielRailean]

closing this as mentioned in #35 (comment)