Turn on test sharding for k8s and docker-build #1058
Conversation
| run: 'for f in *.tar.gz; do tar -zxf "$f" -C "${f%.tar.gz}"; done', | ||
| "working-directory": "${{ github.workspace}}/sdk", |
There was a problem hiding this comment.
This relaxes the logic to unpack any SDK tarballs that happen to be present, not just a language-specific tarball.
| if (sharded) { | ||
| return ["nodejs", "go", "dotnet", "python", "java"].map((language) => { | ||
| return { | ||
| name: `Download ${language} SDK`, | ||
| uses: action.downloadArtifact, | ||
| with: { | ||
| name: `${language}-sdk.tar.gz`, | ||
| path: "${{ github.workspace}}/sdk/", | ||
| }, | ||
| }; | ||
| }); | ||
| } |
There was a problem hiding this comment.
Each shard now needs every SDK.
| @@ -314,7 +314,7 @@ export function InstallPythonDeps(): Step { | |||
| export function InstallSDKDeps(): Step { | |||
| return { | |||
| name: "Install dependencies", | |||
| run: "make install_${{ matrix.language}}_sdk", | |||
| run: "make install_sdks", | |||
There was a problem hiding this comment.
I think this is supported by our native providers, but I could also add a conditional here to be safe.
There was a problem hiding this comment.
I don't think neither pulumi-kubernetes nor pulumi-docker-build have a install_sdks entrypoint currently?
| @@ -314,7 +314,7 @@ export function InstallPythonDeps(): Step { | |||
| export function InstallSDKDeps(): Step { | |||
| return { | |||
| name: "Install dependencies", | |||
| run: "make install_${{ matrix.language}}_sdk", | |||
| run: "make install_sdks", | |||
There was a problem hiding this comment.
I don't think neither pulumi-kubernetes nor pulumi-docker-build have a install_sdks entrypoint currently?
| name: "Run tests", | ||
| run: testCmd, | ||
| }; | ||
| const shardCmd = `echo go test -v -count=1 -cover -timeout 2h -parallel 4${shortMode} "$(go run github.com/blampe/shard@b251cf8da6a83022628496125f285095772ebe86 --total \${{ strategy.job-total }} --index \${{ strategy.job-index }})" > test-command`; |
There was a problem hiding this comment.
Do we plan on movinggithub.com/blampe/shard to a Pulumi controlled repo?
There was a problem hiding this comment.
Perhaps this should be an inline script so it's versioned per-provider via ci-mgmt? If not, then we should pin it.
This turns on test sharding for the docker-build and p-k providers.
The actual logic for how we discover tests and assign them is here, along with some test cases if there are correctness concerns.
Results for p-k:
Total billable time increased by 35m, which roughly lines up with the fixed setup time we use on the 5 additional shards.
In terms of onboarding, if a provider wants to use this they should:
make install_sdkstarget exists, because each shard now needs all the SDKs.go run github.com/blampe/shard@latest --total 1 --index 0will emit a warning for any tests with name collisions. Due to the way we're invokinggo test -run (names...) packages..., it's possible for a test to get re-run on multiple shards if its name isn't unique, but this isn't the end of the world.Refs #676