Permenant diff

[aeweda]

What happened?

i'm facing an issue that's driving me a bit insane, everytime i do a preview or an update all my ecs clusters are marked as changed, specifically task definitions & container definition

i had this issue before but it was contained to container env vars and i was gonna fix it by implementing a deep sort however now it's global

the problem is persistent in ci,local

Example

- Command          : [
-     [0]: "sleep"
-     [1]: "60"
]
- Cpu              : 0
- DependsOn        : [
-     [0]: {
        - Condition    : "START"
        - ContainerName: "gateway"
    }
]
- Environment      : []
- Essential        : false
- Image            : "alpine:latest"
- LogConfiguration : {
    - LogDriver    : "awslogs"
    - Options      : {
        - awslogs-group        : "rx-b6b2f05"
        - awslogs-region       : "us-east-2"
        - awslogs-stream-prefix: "x"
    }
    - SecretOptions: <null>
}
- MemoryReservation: 65
- MountPoints      : []
- Name             : "x"
- PortMappings     : []
- SystemControls   : []
- VolumesFrom      : []
+ command          : [
+     [0]: "sleep"
+     [1]: "60"
]
+ cpu              : 0
+ dependsOn        : [
+     [0]: {
        + condition    : "START"
        + containerName: "gateway"
    }
]
+ environment      : []
+ essential        : false
+ image            : "alpine:latest"
+ logConfiguration : {
    + logDriver: "awslogs"
    + options  : {
        + awslogs-group        : "rx-b6b2f05"
        + awslogs-region       : "us-east-2"
        + awslogs-stream-prefix: "x"
    }
}
+ memoryReservation: 65
+ mountPoints      : []
+ name             : "x"
+ portMappings     : []
+ volumesFrom      : []

Output of pulumi about

CLI          
Version      3.134.1
Go Version   go1.23.1
Go Compiler  gc

Plugins
KIND      NAME     VERSION
resource  aws      6.52.0
resource  awsx     2.16.0
resource  command  1.0.1
resource  docker   4.5.6
resource  docker   3.6.1
language  nodejs   unknown

Host     
OS       ubuntu
Version  22.04
Arch     x86_64

This project is written in nodejs: executable='/home/x/nvm/versions/node/v20.12.2/bin/node' version='v20.12.2'

Additional context

i checked on AWS itself and the task definition is stored as camelCase as it should be, idk why pulumi is comparing with a downstream of PascalCase

also i tried switching to pulumi-aws instead of awsx yet the same problem persisted, i should also mention this problem did not happen 2 days ago

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

I'm sorry this is not working as expected! There are some subtleties like pulumi/pulumi-aws#4336 where we're normalization logic is not quite complete, we've been working with the upstream provider (Terraform AWS provider) to get this to work better. Unfortunately to make progress on this our team needs a very specific self-contained repro. If you could provide this it would be greatly appreciated!

[aeweda]

Hi, sorry this took a while i had some things i needed to finish and i worked on this during the weekend

repro-repo

i tried to mimic my current deployment without a ton of other extras but i was still able to reproduce it, details are in the readme

@t0yv0 let me know if i can be of further hellp

[t0yv0]

Thank you!

[flostadler]

Hey @aeweda, I wasn't able to trigger a perma-diff with the repro you provided. Is there any additional steps/setups needed to trigger this?

[aeweda]

@flostadler i just re tried it with a fresh stack / clone of the repo and it's still consistent, here's what i did

1. clone repo
2. pnpm install
3. pulumi stack create org/bug-repro
4. pulumi stack select org/bug-repro
5. pulumi up -y
6. change any true to false in clusters.ts, like Line 11
7. pulumi up -y
8. after step 7 you can see that the inbound diff is pascalcase while the outbound is camelcase
   

not sure why you can't reproduce it but it's pretty consistent, as i said this is the pascalcase vs camelcase issue.

the perma diff is i'm assuming is related to the fact i have 60 different clusters in my workflow and that could be a factor, but if that pascal/camel issue was fixed i could focus on reproducing the other one as i believe they're somewhat separate

just to clarify something as well, i said perma diff because on my repo it's always showing the pascal vs camel as changes even without changing anything unlike the bug-repro repo

[aeweda]

@flostadler i've also been doing some digging and the cloud formation schema linked on aws docs shows the json/yaml config in pascalcase, could it be a recent change or something that broke this?

[flostadler]

Ah ok @aeweda, yes I can reproduce the pascal case vs camel case issue now but not the perma-diff.
Let's tackle the two separately.

Cloudformation uses pascal case (generally, not only for this resource), but we're integrating with the AWS APIs directly. RegisterTaskDefinition shows the props in camel case, so that should be correct.
What's interesting is that this behavior doesn't trigger on a slimmed down example, so it seems like a specific setting is tripping up the normalization logic.
I'll dig deeper to find the root cause for that.

I understand that it's hard to further debug the perma diff on your end with 60 different clusters. Let me know if you have any observations or a hunch what could trigger it, otherwise I'll first focus on getting the pascal case vs camel case diff issue resolved.