Add API function for using cleanly as a library #39

[TG1999]

Reference: #39
Signed-off-by: Tushar Goel [email protected]

[tdruez]

@TG1999 You resolver_api code is still too correlated with the CLI code.

For example:

json_output = resolver_api(requirement_files=[req_file.name])

ValueError:

No output file specified. Use --json or --json-pdt.

I need a function that, provided a requirements file, returns directly the json as a python object.
Also, those errors are irrelevant in the context of using the tool as a library.

[pombredanne]

Thanks! See some feedback for your review

[tdruez]

@TG1999 I'm getting an error when using this requirements file as input:
https://github.com/nexB/scancode-toolkit/blob/develop/requirements.txt

>>> inspector_output = resolver_api(requirement_files=['requirements.txt'])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Volumes/nexB/repos/scancode.io/lib/python3.9/site-packages/python_inspector/resolve_cli.py", line 446, in resolver_api
    resolved_dependencies, purls = resolve(
  File "/Volumes/nexB/repos/scancode.io/lib/python3.9/site-packages/python_inspector/resolve_cli.py", line 542, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
TypeError: cannot unpack non-iterable NoneType object

---

2 more issues:
Using this input that contains only 1 package entry: reqs.txt

from python_inspector.resolve_cli import resolver_api

>>> resolver_api(requirement_files=["reqs.txt"])
>>> packages = inspector_output.get("packages")
>>> len(packages)
2

>>> packages[0].get("purl")
'pkg:pypi/[email protected]'
>>> packages[1].get("purl")
'pkg:pypi/[email protected]'

>>> packages[0]['parties']
[{'type': 'person', 'role': 'author', 'name': 'William Pearson', 'email': '[email protected]', 'url': None}, 
{'type': 'person', 'role': 'maintainer', 'name': '', 'email': '', 'url': None}]

1. There should be only 1 entry for that package in the resolve packages.

2. One of the 2 parties entry is empty, no name, no email, no URL.

[TG1999]

@tdruez Re:

>>> resolver_api(requirement_files=["reqs.txt"])
>>> packages = inspector_output.get("packages")
>>> len(packages)
2

>>> packages[0].get("purl")
'pkg:pypi/[email protected]'
>>> packages[1].get("purl")
'pkg:pypi/[email protected]'

We have one download_url per package, that's why the purl for both of these are same.

[TG1999]

@tdruez Re:

I'm getting an error when using this requirements file as input:
https://github.com/nexB/scancode-toolkit/blob/develop/requirements.txt

>>> inspector_output = resolver_api(requirement_files=['requirements.txt'])
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/Volumes/nexB/repos/scancode.io/lib/python3.9/site-packages/python_inspector/resolve_cli.py", line 446, in resolver_api
    resolved_dependencies, purls = resolve(
  File "/Volumes/nexB/repos/scancode.io/lib/python3.9/site-packages/python_inspector/resolve_cli.py", line 542, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
TypeError: cannot unpack non-iterable NoneType object

I got this error while using this req file :

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tg1999/Desktop/python-inspector-1/src/python_inspector/resolve_cli.py", line 461, in resolver_api
    resolved_dependencies, purls = resolve(
  File "/home/tg1999/Desktop/python-inspector-1/src/python_inspector/resolve_cli.py", line 548, in resolve
    resolved_dependencies, packages = get_resolved_dependencies(
  File "/home/tg1999/Desktop/python-inspector-1/src/python_inspector/resolve_cli.py", line 598, in get_resolved_dependencies
    raise_error(f"{e!r}")
  File "/home/tg1999/Desktop/python-inspector-1/src/python_inspector/resolve_cli.py", line 276, in raise_exception
    raise Exception(message)
Exception: Exception('Unable to collect setup.py dependencies securely')

I will suggest you to use this
resolver_api(requirement_files=["test.txt"], analyze_setup_py_insecurely=True)

[tdruez]

We have one download_url per package, that's why the purl for both of these are same.

Could you expend a bit your explanation?
I do not understand why single line toml==0.10.2 returns 2 duplicated entries in the packages list.

[tdruez]

I will suggest you to use this
resolver_api(requirement_files=["test.txt"], analyze_setup_py_insecurely=True)

Could you explain why this issue is occurring? What is the specific part of this requirements file that cause thew issue?

[TG1999]

I will suggest you to use this
resolver_api(requirement_files=["test.txt"], analyze_setup_py_insecurely=True)

Could you explain why this issue is occurring? What is the specific part of this requirements file that cause thew issue?

@tdruez rdflib causes this issue, https://github.com/RDFLib/rdflib/blob/main/setup.py

[tdruez]

@TG1999 why "force pushing" into a single commit again? I makes impossible to track the progress made...

[TG1999]

@tdruez we pushed some commits in between that were only for verifying the content of the flaky tests in the CI. And since the verification is now done I squashed the intermediate commits.

[tdruez]

@TG1999 looks good. Can you push a release after the merge?

[pombredanne]

This is a better name IMHO:

Suggested change

	def resolver_api(
	def resolve_dependencies(

[pombredanne]

LGTM!
Thanks!