Add SPDX support #445

wagoodman · 2021-06-23T19:15:44Z

This PR adds initial SPDX support for the tag-value and json formats via the spdx and spdx-json output options. Any discovered packages and package metadata that is compatible with the SPDX spec has been added. What has not been added yet is any relationship data as well as file data (files owned by packages and their metadata) --this will be added in a follow up (#444).

Specific additions:

Adds internal/presenter/packages/spdx_*_presenter for both new formats
Adds internal/presenter/packages/model/spdx22 structs that match the ontology needed for the JSON presenter (specific to SPDX 2.2)
Refactors presenter tests for all presenters to use shared helpers (lots of code reuse here)
Adds go-generated code based on the SPDX license list under internal/spdxlicense (there is an integration test to ensure this list always remains up to date)

Closes #213

Signed-off-by: Alex Goodman <[email protected]>

github-actions · 2021-06-23T19:18:33Z

Benchmark Test Results

Benchmark results from the latest changes vs base branch

name                                                   old time/op    new time/op    delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2           757µs ± 1%     935µs ± 3%  +23.53%  (p=0.008 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2        1.03ms ± 1%    1.23ms ± 5%  +19.72%  (p=0.008 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     382µs ± 0%     464µs ± 5%  +21.62%  (p=0.008 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 361µs ± 1%     448µs ± 4%  +23.90%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  380µs ± 1%     438µs ± 4%  +15.01%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  5.10ms ± 1%    6.03ms ± 7%  +18.36%  (p=0.008 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                  553µs ± 1%     684µs ± 3%  +23.74%  (p=0.008 n=5+5)
ImagePackageCatalogers/go-cataloger-2                     190µs ± 0%     237µs ± 8%  +25.00%  (p=0.008 n=5+5)
ImagePackageCatalogers/rust-cataloger-2                   302µs ± 0%     373µs ± 6%  +23.46%  (p=0.008 n=5+5)

name                                                   old alloc/op   new alloc/op   delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2          97.5kB ± 0%    97.6kB ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/python-package-cataloger-2         579kB ± 0%     579kB ± 0%     ~     (p=1.000 n=5+5)
ImagePackageCatalogers/javascript-package-cataloger-2     112kB ± 0%     112kB ± 0%   -0.15%  (p=0.016 n=5+5)
ImagePackageCatalogers/dpkgdb-cataloger-2                 115kB ± 0%     115kB ± 0%   +0.12%  (p=0.008 n=5+5)
ImagePackageCatalogers/rpmdb-cataloger-2                  134kB ± 0%     134kB ± 0%   -0.01%  (p=0.008 n=5+5)
ImagePackageCatalogers/java-cataloger-2                  1.79MB ± 0%    1.79MB ± 0%     ~     (p=0.421 n=5+5)
ImagePackageCatalogers/apkdb-cataloger-2                 1.14MB ± 0%    1.14MB ± 0%     ~     (p=0.183 n=5+5)
ImagePackageCatalogers/go-cataloger-2                    48.3kB ± 0%    53.8kB ± 0%  +11.42%  (p=0.008 n=5+5)
ImagePackageCatalogers/rust-cataloger-2                  88.9kB ± 0%    88.9kB ± 0%   -0.01%  (p=0.008 n=5+5)

name                                                   old allocs/op  new allocs/op  delta
ImagePackageCatalogers/ruby-gemspec-cataloger-2           1.96k ± 0%     1.96k ± 0%     ~     (all equal)
ImagePackageCatalogers/python-package-cataloger-2         5.89k ± 0%     5.89k ± 0%     ~     (all equal)
ImagePackageCatalogers/javascript-package-cataloger-2     1.93k ± 0%     1.93k ± 0%     ~     (all equal)
ImagePackageCatalogers/dpkgdb-cataloger-2                 2.37k ± 0%     2.37k ± 0%     ~     (all equal)
ImagePackageCatalogers/rpmdb-cataloger-2                  3.19k ± 0%     3.19k ± 0%     ~     (all equal)
ImagePackageCatalogers/java-cataloger-2                   22.3k ± 0%     22.3k ± 0%     ~     (p=0.333 n=5+4)
ImagePackageCatalogers/apkdb-cataloger-2                  1.85k ± 0%     1.85k ± 0%     ~     (p=0.444 n=5+5)
ImagePackageCatalogers/go-cataloger-2                     1.40k ± 0%     1.44k ± 0%   +2.72%  (p=0.008 n=5+5)
ImagePackageCatalogers/rust-cataloger-2                   2.75k ± 0%     2.75k ± 0%     ~     (all equal)

dakaneye

wow this was a LOT!

I like consolidating the presenter tests the way you did, all of this code appears to have great test coverage in general.

I didn't go over what each field was in each struct, but the go code itself looks great, and easy to follow.

I'm also not super familiar with the spdx stuff and wanted some additional background on the license and schema things that were added but don't want to block based on that.

dakaneye · 2021-06-24T18:16:03Z

internal/presenter/packages/spdx_helpers.go

+	return strings.Join(parsedLicenses, " AND ")
+}
+
+func noneIfEmpty(value string) string {


nit: this seems like it could be a more global string helper

(feel free to ignore this comment tho)

The returned "NONE" value is pretty coupled to the SPDX spec and it's verbiage --for now I think it belongs with the SPDX helpers

dakaneye · 2021-06-24T18:21:49Z

internal/spdxlicense/generate_license_list.go

@@ -0,0 +1,92 @@
+// +build ignore


what's the purpose of this code?

SPDX spec dictates which authoritative license names can be reported out in an SPDX SBOM. They maintain the authoritative list at https://spdx.org/licenses/licenses.json and the version of this list needs to be additionally reported in the SBOM. This generate_license_list.go responds to the go generate ./... command in this repo based on the //go:generate go run generate_license_list.go comment in license.go.

Specifically the // +build ignore comment in generate_license_list.go is meant to keep this go source file out of the syft build. That is, when running go build ./... this "build tag" is used to tell the compiler to skip this file altogether when making the syft binary. The generate_license_list.go file is still kept specially near the file it is generating for simplicity (license_list.go).

So in order to build syft with the latest license list (pulled from https://spdx.org/licenses/licenses.json) one needs to run:

go generate ./... go build ./...

But if you want to keep the existing license list and just build syft then it you do the same as you would today:

go build ./...

note: I've got an integration test that checks to see if there is a later version of the license list than what we have now --if there is then the test fails and notes in the output as to why.

wagoodman · 2021-06-25T20:29:54Z

@dakaneye re: what's being added schema-wise for SPDX. I pulled in the SPDX JSON schema and added a JSON schema validation test as a CLI test to ensure that, for the image tested against, syft produces valid JSON relative to the published SPDX JSON schema (v2.2).

* add initial spdx support Signed-off-by: Alex Goodman <[email protected]> * expose FileOwner and use in SPDX presenter Signed-off-by: Alex Goodman <[email protected]> * add initial json support for SPDX Signed-off-by: Alex Goodman <[email protected]> * add remaining package fields Signed-off-by: Alex Goodman <[email protected]> * add spdx license list generation + tests Signed-off-by: Alex Goodman <[email protected]> * keep fileOwner unexported from pkg Signed-off-by: Alex Goodman <[email protected]> * restore cli test util Signed-off-by: Alex Goodman <[email protected]> * add external refs to spdx tag-value format Signed-off-by: Alex Goodman <[email protected]> * add golang support to CPE generation Signed-off-by: Alex Goodman <[email protected]> * use tag-value format as default "spdx" format flavor Signed-off-by: Alex Goodman <[email protected]> * add tests around spdx presenters + refactor presenter tests Signed-off-by: Alex Goodman <[email protected]> * add bouncer exception for spdx tools-golang repo Signed-off-by: Alex Goodman <[email protected]> * remove spdx model questions Signed-off-by: Alex Goodman <[email protected]>

wagoodman added 13 commits June 23, 2021 11:20

add initial spdx support

b8b71f5

Signed-off-by: Alex Goodman <[email protected]>

expose FileOwner and use in SPDX presenter

40e93c8

Signed-off-by: Alex Goodman <[email protected]>

add initial json support for SPDX

de0c9bd

Signed-off-by: Alex Goodman <[email protected]>

add remaining package fields

421641f

Signed-off-by: Alex Goodman <[email protected]>

add spdx license list generation + tests

5269c47

Signed-off-by: Alex Goodman <[email protected]>

keep fileOwner unexported from pkg

a6ef8a7

Signed-off-by: Alex Goodman <[email protected]>

restore cli test util

87618c0

Signed-off-by: Alex Goodman <[email protected]>

add external refs to spdx tag-value format

7761619

Signed-off-by: Alex Goodman <[email protected]>

add golang support to CPE generation

71fbadf

Signed-off-by: Alex Goodman <[email protected]>

use tag-value format as default "spdx" format flavor

f8755d9

Signed-off-by: Alex Goodman <[email protected]>

add tests around spdx presenters + refactor presenter tests

421f602

Signed-off-by: Alex Goodman <[email protected]>

add bouncer exception for spdx tools-golang repo

1ae60e7

Signed-off-by: Alex Goodman <[email protected]>

remove spdx model questions

af8318c

Signed-off-by: Alex Goodman <[email protected]>

wagoodman requested a review from luhring June 23, 2021 19:15

wagoodman self-assigned this Jun 23, 2021

wagoodman requested a review from dakaneye June 24, 2021 17:24

dakaneye approved these changes Jun 24, 2021

View reviewed changes

wagoodman merged commit 706322f into main Jun 25, 2021

wagoodman deleted the add-spdx-support branch June 25, 2021 20:30

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add SPDX support #445

Add SPDX support #445

wagoodman commented Jun 23, 2021 •

edited

Loading

github-actions bot commented Jun 23, 2021 •

edited

Loading

dakaneye left a comment

dakaneye Jun 24, 2021

wagoodman Jun 25, 2021

dakaneye Jun 24, 2021

wagoodman Jun 25, 2021

wagoodman commented Jun 25, 2021

Add SPDX support #445

Add SPDX support #445

Conversation

wagoodman commented Jun 23, 2021 • edited Loading

github-actions bot commented Jun 23, 2021 • edited Loading

Benchmark Test Results

dakaneye left a comment

Choose a reason for hiding this comment

dakaneye Jun 24, 2021

Choose a reason for hiding this comment

wagoodman Jun 25, 2021

Choose a reason for hiding this comment

dakaneye Jun 24, 2021

Choose a reason for hiding this comment

wagoodman Jun 25, 2021

Choose a reason for hiding this comment

wagoodman commented Jun 25, 2021

wagoodman commented Jun 23, 2021 •

edited

Loading

github-actions bot commented Jun 23, 2021 •

edited

Loading