Support multiple requirement files according to envionments

[manuel-sommer]

Hi all,

I have multiple requirement files in a folder. This is done because of different environments, e.g.:
/requirements/base.txt
/requirements/development.txt
/requirements/production.txt

The base.txt has all dependencies which are needed in all environments. The other files (e.g. development.txt or production.txt) refer to the base.txt and add additional dependencies, e.g.

# base requirements
-r base.txt

#additional requirements
pylint==2.2.0

Could you advance cyclonedx-python to scan multiple requirement files at once?
Furthermore, cyclonedx-python fails if there is "-r base.txt" in a requirements file?

Traceback (most recent call last):
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 98, in __init__
    req = REQUIREMENT.parseString(requirement_string)
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 1654, in parseString
    raise exc
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 1644, in parseString
    loc, tokens = self._parse( instring, 0 )
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 3417, in parseImpl
    loc, exprtokens = e._parse( instring, loc, doActions )
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 3739, in parseImpl
    return self.expr._parse( instring, loc, doActions, callPreParse=False )
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 1402, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 3400, in parseImpl
    loc, resultlist = self.exprs[0]._parse( instring, loc, doActions, callPreParse=False )
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 1406, in _parseNoCache
    loc,tokens = self.parseImpl( instring, preloc, doActions )
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/pyparsing.py", line 2711, in parseImpl
    raise ParseException(instring, loc, self.errmsg, self)
pkg_resources._vendor.pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/test/.local/bin/cyclonedx-bom", line 8, in <module>
    sys.exit(main())
  File "/home/test/.local/lib/python3.8/site-packages/cyclonedx_py/client.py", line 211, in main
    CycloneDxCmd(args).execute()
  File "/home/test/.local/lib/python3.8/site-packages/cyclonedx_py/client.py", line 88, in execute
    output = self.get_output()
  File "/home/test/.local/lib/python3.8/site-packages/cyclonedx_py/client.py", line 52, in get_output
    parser = self._get_input_parser()
  File "/home/test/.local/lib/python3.8/site-packages/cyclonedx_py/client.py", line 203, in _get_input_parser
    return RequirementsParser(requirements_content=input_data)
  File "/home/test/.local/lib/python3.8/site-packages/cyclonedx/parser/requirements.py", line 32, in __init__
    for requirement in requirements:
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3080, in parse_requirements
    yield Requirement(line)
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/__init__.py", line 3090, in __init__
    super(Requirement, self).__init__(requirement_string)
  File "/home/test/.local/lib/python3.8/site-packages/pkg_resources/_vendor/packaging/requirements.py", line 100, in __init__
    raise InvalidRequirement(
pkg_resources.extern.packaging.requirements.InvalidRequirement: Parse error at "'-r base.'": Expected W:(abcd...)

The following command is an example to this issue:
cyclonedx-bom -r -i base.txt -r -i development.txt --format json -o test.json

Thank you

[jkowalleck]

hello @manuel-sommer .

In case you need an solution to merge multiple CycloneDX SBOM files into one SBOM you could have a look at one of the already existing SBOM tools in the CDX tool center

maybe one of these can help?

• https://github.com/CycloneDX/cyclonedx-cli
• https://github.com/CycloneDX/sbom-combiner

---

regarding

Furthermore, cyclonedx-python fails if there is "-r base.txt" in a requirements file

This syntax is not supported.
Actually, "-r base.txt" is not a universal requirement. It is a pre-requirement-phase statement as used in requirements.in files.
You need to "compile" your input file to an actual static requirements-file - for example with pip-compile from https://pypi.org/project/pip-tools/

---

did this information solve your problems/requests?

[jkowalleck]

@madpah i suggest to stay with the current implementation.

having the parser understand -r ... and other pip-specific syntax is already solved by other tools(pip-compile/pip-tools)
and therefore not in the scope of cyclonedx-python .

therefore i suggest to treat this very issue as a "request for documentation" rather then a "bug report" .
this would mean, we need to improve our docs and make clear, that a requirements.txt is, and what it is not.

any toughs?

[madpah]

Sounds like a good answer @jkowalleck.

[jkowalleck]

Since we would not adopt -r <FILE> syntax inside a requirements-file ,
we might want to allow multiple <INPUT-FILE> CLI options, still. - as proposed by @manuel-sommer
So multiple -i <FILE> CLI parameters would come in and need to be treated.

This would open up a whole new story:

• does input-files order matter?
• what if the multiple input files contradict each other?
• what if the multiple input files use the same library in different ranges/versions?
• do we allow mixing the input-file-types (poetry.lock, pipenv.lock, requirements.txt)? if so, how would we know which input file was of which format? we need to rethink the CLI and this would probalby lead to breaking changes (which i am fine with)

anyways - my (weak opinionated) current vote is: no - dont allow multiple input files as a feature release.
current CycloneDX-Python tool is about describing the environment and which libraries are actually shipped with an environment.
There is only one python environment that is to be considered - and if pip freeze > requrements.txt did not include everything, then you are using it wrong.
CHANGE MY MIND.

speaking of which - we might want to convert this issue to a discussion.
or open a new one here https://github.com/CycloneDX/cyclonedx-python/discussions
to discuss the topic of multiple input files.

[madpah]

@manuel-sommer given our comments above, we've moved this to a discussion at this time to open up to the wider world for input, comment etc.

We're open to the idea being proposed, but feel this needs more rounding out and justification before we get on board.

Please do come back with any further justifications / real-world scenarios you have that would fight for this idea.

Thanks again for your input!

[jkowalleck]

FYIL the upcoming v4 will have various kinds of supporting individual/multiple requirements.txt files
this includes examples like

• cat req1.txt req2.txt | cyclonedx-py requirements
• cyclonedx-py requirements requirements-dev.txt where the requirements-file internally uses -r deep concats