Pushing to GitHub fails with BLOB_UPLOAD_UNKNOWN

[mihalyr]

Environment:

• Jib version: 3.4.2
• Build tool: gradle
• OS: Linux - Fedora 40

Description of the issue:

When using Jib Gradle plugin to push a container image to GitHub Container Registry (ghcr.io), it fails frequently in the middle of the operation with the following error:

 > com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<hash> but failed because: other: blob upload unknown to registry

The stacktrace contains the following error message:

Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
PUT https://ghcr.io/v2/<org>/<repo>/blobs/upload/<uuid>?digest=sha256:<hash>
{"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}

	at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)
	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:349)
	... 176 more

So far every time when I manually re-run the failed job (GitHub Actions workflow), it succeeds on retry without problems.

Expected behavior:

Running Jib to upload containers to GitHub registry without errors and manual retries.

Steps to reproduce:

1. Create a GitHub Action workflow that uses gradle jib plugin to push to the GitHub container registry in the same repository (jib.to.image=ghcr.io/org/repo, jib.to.auth.username=github.actor, jib.to.auth.password=github.token, jib.container.labels=org.opencontainers.image.source=github.server_url/github.repository)
2. Run the workflow

Log output:

Build failure logs

Executing tasks:
[======================        ] 72.7% complete
> building images to registry


Executing tasks:
[======================        ] 72.7% complete
> checking existence of manifest for sha256:74e...


Executing tasks:
[=========================     ] 81.8% complete
> building images to registry


Executing tasks:
[=========================     ] 81.8% complete
> scheduling pushing manifests


Executing tasks:
[===========================   ] 90.9% complete
> building images to registry


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':server:jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<hash> but failed because: other: blob upload unknown to registry

* Try:


A new version of jib-gradle-plugin (3.4.3) is available (currently using 3.4.2). Update your build configuration to use the latest features and fixes!
https://github.com/GoogleContainerTools/jib/blob/master/jib-gradle-plugin/CHANGELOG.md

Please see https://github.com/GoogleContainerTools/jib/blob/master/docs/privacy.md for info on disabling this update check.

Executing tasks:
[===========================   ] 90.9% complete
> building images to registry


> Task :server:jib FAILED
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':server:jib'.
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:130)
	at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:127)
	at org.gradle.composite.internal.DefaultBuildController.doRun(DefaultBuildController.java:181)
	at org.gradle.composite.internal.DefaultBuildController.access$000(DefaultBuildController.java:50)
	at org.gradle.composite.internal.DefaultBuildController$BuildOpRunnable.lambda$run$0(DefaultBuildController.java:198)
	at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:85)
	at org.gradle.composite.internal.DefaultBuildController$BuildOpRunnable.run(DefaultBuildController.java:198)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
Caused by: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<hash> but failed because: other: blob upload unknown to registry
	at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild(JibBuildRunner.java:280)
	at com.google.cloud.tools.jib.gradle.BuildImageTask.buildImage(BuildImageTask.java:122)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
	... 145 more
Caused by: com.google.cloud.tools.jib.registry.RegistryErrorException: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<hash> but failed because: other: blob upload unknown to registry
	at com.google.cloud.tools.jib.registry.RegistryErrorExceptionBuilder.build(RegistryErrorExceptionBuilder.java:101)
	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.newRegistryErrorException(RegistryEndpointCaller.java:212)
	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:153)
	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:114)
	at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint(RegistryClient.java:623)
	at com.google.cloud.tools.jib.registry.RegistryClient.pushBlob(RegistryClient.java:564)
	at com.google.cloud.tools.jib.builder.steps.PushBlobStep.call(PushBlobStep.java:89)
	at com.google.cloud.tools.jib.builder.steps.PushLayerStep.call(PushLayerStep.java:92)
	at com.google.cloud.tools.jib.builder.steps.PushLayerStep.call(PushLayerStep.java:33)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:79)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.scheduleCallables(StepsRunner.java:648)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$pushApplicationLayers$13(StepsRunner.java:507)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:79)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.pushApplicationLayers(StepsRunner.java:505)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$run$2(StepsRunner.java:218)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.run(StepsRunner.java:218)
	at com.google.cloud.tools.jib.api.Containerizer.run(Containerizer.java:406)
	at com.google.cloud.tools.jib.api.JibContainerBuilder.containerize(JibContainerBuilder.java:610)
	at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild(JibBuildRunner.java:229)
	... 148 more
Caused by: com.google.cloud.tools.jib.http.ResponseException: 404 Not Found
PUT https://ghcr.io/v2/<org>/<repo>/blobs/upload/<uploadid>?digest=sha256:<hash>
{"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}

	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:355)
	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:266)
	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:138)
	... 174 more
Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
PUT https://ghcr.io/v2/<org>/<repo>/blobs/upload/<uploadid>?digest=sha256:<hash>
{"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}

	at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)
	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:349)
	... 176 more


BUILD FAILED in 2m 50s
22 actionable tasks: 15 executed, 7 from cache
Error: Process completed with exit code 1.

[MrPowerGamerBR]

I can confirm the same issue, it is very annoying. There are other users talking about the same issue on #2372, and @KRK1ST said that he talked with GitHub Support and they said that GitHub belives that it is a issue in Jib.

"one theory being that Jib may be retrying an upload before the previous upload has actually failed. This could also explain the inconsistency of failure you are seeing, as different uploads would take different amounts of time, and thus some would fit within the retry window (thus not failing), while a few might exceed the retry window and thus fail."

I have tried running it with -Djib.serialize=true but it doesn't fix the issue. What I'm doing right now is running all jib tasks in different build steps with -Djib.serialize=true, it doesn't 100% fix the issue, but it does make it less common.

[martintomac]

It happens with maven also. We have this issue for long time now over many versions.

Error:  Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default-cli) on project download-gateway: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<sha> but failed because: other: blob upload unknown to registry: 404 Not Found
Error:  PUT https://ghcr.io/v2/<org>/<repo>/blobs/upload/<uuid>?digest=sha256:<sha>
Error: [ERROR] {"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}

Some of our builds push multiple images sequentially so this fails our build every other or third time. We tried -Djib.serialize=true but it doesn't fix the issue.