launchpad builder won't clone one of three identical repos

[popey]

Bug Description

I am making a prototype of three snaps of syft, grype, and grant. They all use the same yaml, and come from the same org on GitHub. Syft and Grant clone and start building fine. Grype is unable to be cloned.

To Reproduce

Use the yaml below, start a remote build. It will fail.

part yaml

name: grype
base: core22
version: 'v0.79.3'
summary: Vulnerability scanner
description: |
  CLI vulnerability scanner for container images and filesystems

architectures:
  - build-on: amd64
  - build-on: arm64
  - build-on: armhf
  - build-on: s390x
  - build-on: ppc64el

grade: stable
confinement: classic

parts:
  grype:
    plugin: go
    source: https://github.com/anchore/grype.git
    source-type: git
    source-tag: 'v0.79.3'
    build-snaps:
      - go

apps:
  grype:
    command: bin/grype

Relevant log output

Running pull phase...
Initializing parts lifecycle
Installing build-packages
Installing build-snaps
Pulling grype
[19/Jul/2024:14:13:48 +0000] "CONNECT github.com:443 HTTP/1.1" 200 5117550 "-" "git/2.34.1"
Failed to pull source: command ['git', 'clone', '--recursive', '--branch', 'v0.79.3', 'https://github.com/anchore/grype.git', '/build/snapcraft-grype-4b4eef8795d90369d120d00672377f51/parts/grype/src'] exited with code 1.
Make sure sources are correctly specified.
Full execution log: '/root/.local/state/snapcraft/log/snapcraft-20240719-141338.341400.log'
Build failed
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 323, in run
    self.pull()
  File "/usr/lib/python3/dist-packages/lpbuildd/target/build_snap.py", line 261, in pull
    self.run_build_command(
  File "/usr/lib/python3/dist-packages/lpbuildd/target/operation.py", line 62, in run_build_command
    return self.backend.run(args, cwd=cwd, env=full_env, **kwargs)
  File "/usr/lib/python3/dist-packages/lpbuildd/target/lxd.py", line 718, in run
    subprocess.check_call(cmd, **kwargs)
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['lxc', 'exec', 'lp-jammy-ppc64el', '--env', 'LANG=C.UTF-8', '--env', 'SHELL=/bin/sh', '--env', 'http_proxy=http://10.10.10.1:8222/', '--env', 'https_proxy=http://10.10.10.1:8222/', '--env', 'GIT_PROXY_COMMAND=/usr/local/bin/lpbuildd-git-proxy', '--env', 'SNAPPY_STORE_NO_CDN=1', '--env', 'SNAPCRAFT_LOCAL_SOURCES=1', '--env', 'SNAPCRAFT_SETUP_CORE=1', '--env', 'SNAPCRAFT_BUILD_INFO=1', '--env', 'SNAPCRAFT_IMAGE_INFO={"build-request-id": "lp-90687766", "build-request-timestamp": "2024-07-19T14:10:46Z", "build_url": "https://launchpad.net/~popey/popey-craft-remote-build/+snap/snapcraft-grype-4b4eef8795d90369d120d00672377f51/+build/2546651"}', '--env', 'SNAPCRAFT_BUILD_ENVIRONMENT=host', '--', '/bin/sh', '-c', 'cd /build/snapcraft-grype-4b4eef8795d90369d120d00672377f51 && linux64 snapcraft pull']' returned non-zero exit status 1.
Revoking proxy token...
RUN: /usr/share/launchpad-buildd/bin/in-target scan-for-processes --backend=lxd --series=jammy --arch=ppc64el SNAPBUILD-2546651
Scanning for processes to kill in build SNAPBUILD-2546651

[popey]

Also fails if you don't specify the tag:

Pulling grype
[19/Jul/2024:16:35:02 +0000] "CONNECT github.com:443 HTTP/1.1" 200 5117245 "-" "git/2.34.1"
Failed to pull source: command ['git', 'clone', '--recursive', 'https://github.com/anchore/grype.git', '/build/snapcraft-grype-b106cde28bf55aa27c4a803831e37e81/parts/grype/src'] exited with code 1.
Make sure sources are correctly specified.

[popey]

Works if I pull the tarball, as you might expect...

parts:
  grype:
    plugin: go
    source: https://github.com/anchore/grype/archive/refs/tags/v0.79.3.tar.gz
    build-snaps:
      - go

Log:

Pulling grype
[19/Jul/2024:16:44:53 +0000] "CONNECT github.com:443 HTTP/1.0" 200 7334 "-" "-"
[19/Jul/2024:16:44:53 +0000] "CONNECT codeload.github.com:443 HTTP/1.0" 200 1189336 "-" "-"
Running build phase...
Initializing parts lifecycle
Installing build-packages
Installing build-snaps
Skipping pull for grype (already ran)
Building grype
:: + go mod download all
[19/Jul/2024:16:44:58 +0000] "CONNECT proxy.golang.org:443 HTTP/1.1" 200 11103 "-" "Go-http-client/1.1"
[19/Jul/2024:16:44:58 +0000] "CONNECT proxy.golang.org:443 HTTP/1.1" 200 11103 "-" "Go-http-client/1.1"
[19/Jul/2024:16:44:58 +0000] "CONNECT proxy.golang.org:443 HTTP/1.1" 200 11105 "-" "Go-http-client/1.1"
:: + go install -p 4 ./...
[19/Jul/2024:16:48:07 +0000] "CONNECT proxy.golang.org:443 HTTP/1.1" 200 226596747 "-" "Go-http-client/1.1"
[19/Jul/2024:16:48:07 +0000] "CONNECT sum.golang.org:443 HTTP/1.1" 200 3422023 "-" "Go-http-client/1.1"
[19/Jul/2024:16:48:07 +0000] "CONNECT storage.googleapis.com:443 HTTP/1.1" 200 431837387 "-" "Go-http-client/1.1"
Staging grype
Priming grype
Extracting and updating metadata...
Copying snap assets...
Generating snap metadata...
Generated snap metadata
'--enable-manifest' is deprecated, and will be removed in core24.
Generating snap manifest...
Generated snap manifest
Reading snap metadata...
Running linters...
Running linter: classic
Running linter: library
Lint OK:
- classic: Snap confinement is set to classic.
Lint warnings:
- classic: bin/generate: ELF interpreter should be set to '/snap/core22/current/lib64/ld-linux-x86-64.so.2'. (https://snapcraft.io/docs/linters-classic)
- classic: bin/generate: ELF rpath should be set to '/snap/core22/current/lib/x86_64-linux-gnu'. (https://snapcraft.io/docs/linters-classic)
- classic: bin/grype: ELF interpreter should be set to '/snap/core22/current/lib64/ld-linux-x86-64.so.2'. (https://snapcraft.io/docs/linters-classic)
- classic: bin/grype: ELF rpath should be set to '/snap/core22/current/lib/x86_64-linux-gnu'. (https://snapcraft.io/docs/linters-classic)
Creating snap package...
Created snap package grype_v0.79.3_amd64.snap

[syncronize-issues-to-jira]

Thank you for reporting us your feedback!

The internal ticket has been created: https://warthogs.atlassian.net/browse/CRAFT-3258.

This message was autogenerated

[lengau]

I've reached out to the Launchpad team about this, as I suspect it's something with launchpad's environment. (maybe being firewalled?) Given the nature of the app I could imagine a very zealous scanner picking up what the code does and shutting it down thinking it's trying to sniff out vulnerabilities in Launchpad, but that's just my uninformed hypothesis

[popey]

@lengau Thanks Alex. That would be wild, given I am working around this issue by grabbing the binary from the same repo, and can do the same security scans with it. :)

[lengau]

@popey agree, but there definitely seems to be something specific to the Launchpad environment. I've made #821 to get us better error messaging out of git that should help isolate this, but it's a long pipeline unfortunately.

[dariuszd21]

Can you try changing the submodule URL to be accessible via https:// ?
Command tries to clone recursively, and it will fail in the environment that does not have SSH keys for GitHub available (such as Launchpad)

https://github.com/anchore/grype/blob/fbdab6e1ecefad6d61a0a2819d97c4ce17d3bf20/.gitmodules#L3

[popey]

Thanks @dariuszd21 - I've certainly seen this before, and would agree this is likely going to be the quick fix. Thanks!