Cannot install scancode toolkit v31.0.0b3 on Mac (ResolutionImpossible)

[Hritik14]

Description

Unable to install using the instructions given at https://scancode-toolkit.readthedocs.io/en/stable/getting-started/install.html#installation-from-source-code-git-clone

How To Reproduce

wget https://github.com/nexB/scancode-toolkit/releases/download/v31.0.0b3/scancode-toolkit-31.0.0b3_py39-macos.tar.xz
tar -xvf scancode-toolkit-31.0.0b3_py39-macos.tar.xz
./configure

...
ERROR: Cannot install extractcode[full]==30.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    extractcode[full] 30.0.0 depends on extractcode-libarchive>=3.5.1.210525; extra == "full"
    The user requested (constraint) extractcode-libarchive==3.5.1.210531

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
...

System configuration

For bug reports, it really helps us to know:

• What OS are you running on? MacOS
• What version of scancode-toolkit was used to generate the scan file? v31.0.0b3
• What installation method was used to install/run scancode? (pip/source download/other) source

[Hritik14]

I tried with Python 3.9.12. Will try with the supported version and update the issue

[pombredanne]

@Hritik14 Thank you ++
Can you tell me if you are running a Python x86/64 or a Python ARM build?

[Hritik14]

@pombredanne It's Mac M1 processor which is based on arm in some way.
Here's how the prompt looks like

; python
Python 3.9.12 (main, Mar 26 2022, 15:44:31)
[Clang 13.1.6 (clang-1316.0.21.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

Tried again on python 8. Here are the results

; python --version
Python 3.8.10
; ./configure
Looking in links: https://thirdparty.aboutcode.org/pypi/simple/links.html
Obtaining file:///Users/neo/Downloads/scancode-toolkit-31.0.0b3
  Checking if build backend supports build_editable ... done
  Preparing metadata (pyproject.toml) ... done
Collecting ftfy>=4.0.0
...
Collecting patch>=1.16
  Using cached https://thirdparty.aboutcode.org/pypi/patch-1.16-py3-none-any.whl (12 kB)
INFO: pip is looking at multiple versions of dparse2 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of debian-inspector to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of container-inspector to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of commoncode to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of colorama to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of click to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of chardet to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of boolean-py to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of beautifulsoup4 to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of <Python from Requires-Python> to determine which version is compatible with other requirements. This could take a while.
INFO: pip is looking at multiple versions of scancode-toolkit to determine which version is compatible with other requirements. This could take a while.
ERROR: Cannot install extractcode[full]==30.0.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    extractcode[full] 30.0.0 depends on extractcode-libarchive>=3.5.1.210525; extra == "full"
    The user requested (constraint) extractcode-libarchive==3.5.1.210531

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts

[pombredanne]

Is there a way to use a x86/64 build of Python instead? and use that?
Apple support x86/64 on their OS...
May be some of the steps in https://medium.com/swlh/run-x86-terminal-apps-like-homebrew-on-your-new-m1-mac-73bdc9b0f343 ?
If you get it to work I am really looking forward to the steps to document this.

[MarcelBochtler]

l've installed the universal variant of Python 3.8 on my MacBook Pro 14" M1 Max using MacPorts:

sudo port install python38 +universal

❯ arch -x86_64 /opt/local/bin/python3.8 -c 'import os; print(os.uname())'
posix.uname_result(sysname='Darwin', nodename='Marcels-MacBook-Pro.local', release='21.4.0', version='Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000', machine='x86_64')

Running ScanCode 31.0.0b4 works fine when forcing it to use x86_64 under Rosetta 2:

❯ arch -x86_64 /opt/local/bin/bash -c "/Users/marcel/Downloads/scancode-toolkit-31.0.0b4/scancode --help"
Usage: scancode [OPTIONS] <OUTPUT FORMAT OPTION(s)> <input>...

  scan the <input> file or directory for license, origin and packages and save
  results to FILE(s) using one or more output format option.

  Error and progress are printed to stderr.
...

Scanning a file with findings:

~/workspace/ost/ort/utils/spdx/src/main/kotlin main*
❯ arch -x86_64 /opt/local/bin/bash -c "/Users/marcel/Downloads/scancode-toolkit-31.0.0b4/scancode --copyright --license --info --json result.json SpdxConstants.kt"
Setup plugins...
Collect file inventory...
Scan files for: info, licenses, copyrights with 1 process(es)...
[####################] 2
Scanning done.
Summary:        info, licenses, copyrights with 1 process(es)
Errors count:   0
Scan Speed:     0.37 files/sec. 1.16 KB/sec.
Initial counts: 1 resource(s): 1 file(s) and 0 directorie(s)
Final counts:   1 resource(s): 1 file(s) and 0 directorie(s) for 3.11 KB
Timings:
  scan_start: 2022-05-14T120942.780344
  scan_end:   2022-05-14T120946.933299
  setup_scan:licenses: 1.45s
  setup: 1.45s
  scan: 2.68s
  total: 4.15s
Removing temporary files...done.


~/workspace/ost/ort/utils/spdx/src/main/kotlin main* 6s
❯ cat result.json
{"headers": [{"tool_name": "scancode-toolkit", "tool_version": "31.0.0b4", "options": {"input": ["SpdxConstants.kt"], "--copyright": true, "--info": true, "--json": "result.json", "--license": true}, "notice": "Generated with ScanCode and provided on an \"AS IS\" BASIS, WITHOUT WARRANTIES\nOR CONDITIONS OF ANY KIND, either express or implied. No content created from\nScanCode should be considered or used as legal advice. Consult an Attorney\nfor any legal advice.\nScanCode is a free software code scanning tool from nexB Inc. and others.\nVisit https://github.com/nexB/scancode-toolkit/ for support and download.", "start_timestamp": "2022-05-14T120942.780344", "end_timestamp": "2022-05-14T120946.933299", "output_format_version": "2.0.0", "duration": 4.152984857559204, "message": null, "errors": [], "warnings": [], "extra_data": {"system_environment": {"operating_system": "mac", "cpu_architecture": "64", "platform": "macOS-12.3.1-x86_64-arm64-64bit", "platform_version": "Darwin Kernel Version 21.4.0: Fri Mar 18 00:46:32 PDT 2022; root:xnu-8020.101.4~15/RELEASE_ARM64_T6000", "python_version": "3.8.13 (default, May 11 2022, 19:31:38) \n[Clang 13.1.6 (clang-1316.0.21.2.3)]"}, "spdx_license_list_version": "3.16", "files_count": 1}}], "files": [{"path": "SpdxConstants.kt", "type": "file", "name": "SpdxConstants.kt", "base_name": "SpdxConstants", "extension": ".kt", "size": 3184, "date": "2022-04-30", "sha1": "ba11e6097b977c438193b2eeda1142b591fc5912", "md5": "7ff5b1964a6102b25ea2319c683316d8", "sha256": "d67ccbbb7df93b13c7dbb530df4fc1e214cdf612430ced7fd671e37c0a1bab65", "mime_type": "text/plain", "file_type": "ASCII text", "programming_language": "Kotlin", "is_binary": false, "is_text": true, "is_archive": false, "is_media": false, "is_source": true, "is_script": false, "licenses": [{"key": "apache-2.0", "score": 80.37, "name": "Apache License 2.0", "short_name": "Apache 2.0", "category": "Permissive", "is_exception": false, "is_unknown": false, "owner": "Apache Software Foundation", "homepage_url": "http://www.apache.org/licenses/", "text_url": "http://www.apache.org/licenses/LICENSE-2.0", "reference_url": "https://scancode-licensedb.aboutcode.org/apache-2.0", "scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.LICENSE", "scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.yml", "spdx_license_key": "Apache-2.0", "spdx_url": "https://spdx.org/licenses/Apache-2.0", "start_line": 4, "end_line": 17, "matched_rule": {"identifier": "apache-2.0_949.RULE", "license_expression": "apache-2.0", "licenses": ["apache-2.0"], "referenced_filenames": ["/usr/share/common-licenses/Apache-2.0"], "is_license_text": false, "is_license_notice": true, "is_license_reference": false, "is_license_tag": false, "is_license_intro": false, "has_unknown": false, "matcher": "3-seq", "rule_length": 107, "matched_length": 86, "match_coverage": 80.37, "rule_relevance": 100}}, {"key": "apache-2.0", "score": 100.0, "name": "Apache License 2.0", "short_name": "Apache 2.0", "category": "Permissive", "is_exception": false, "is_unknown": false, "owner": "Apache Software Foundation", "homepage_url": "http://www.apache.org/licenses/", "text_url": "http://www.apache.org/licenses/LICENSE-2.0", "reference_url": "https://scancode-licensedb.aboutcode.org/apache-2.0", "scancode_text_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.LICENSE", "scancode_data_url": "https://github.com/nexB/scancode-toolkit/tree/develop/src/licensedcode/data/licenses/apache-2.0.yml", "spdx_license_key": "Apache-2.0", "spdx_url": "https://spdx.org/licenses/Apache-2.0", "start_line": 16, "end_line": 16, "matched_rule": {"identifier": "spdx-license-identifier: apache-2.0", "license_expression": "apache-2.0", "licenses": ["apache-2.0"], "referenced_filenames": [], "is_license_text": false, "is_license_notice": false, "is_license_reference": false, "is_license_tag": true, "is_license_intro": false, "has_unknown": false, "matcher": "1-spdx-id", "rule_length": 6, "matched_length": 6, "match_coverage": 100.0, "rule_relevance": 100}}], "license_expressions": ["apache-2.0", "apache-2.0"], "percentage_of_license_text": 22.06, "copyrights": [{"copyright": "Copyright (c) 2020 HERE Europe B.V.", "start_line": 2, "end_line": 2}], "holders": [{"holder": "HERE Europe B.V.", "start_line": 2, "end_line": 2}], "authors": [], "files_count": 0, "dirs_count": 0, "size_count": 0, "scan_errors": []}]}

[pombredanne]

@tdruez reported to me:

@pombredanne I’m still getting installation issue on clean toolkit install on M1:
pip install scancode-toolkit==31.0.0rc1
ERROR: Could not find a version that satisfies the requirement extractcode-libarchive>=3.5.1.210525; extra == "full" (from extractcode[full]) (from versions: none)
ERROR: No matching distribution found for extractcode-libarchive>=3.5.1.210525; extra == "full"

With this said here is the status of M1 support:

1. the installation of 31.0.0rc1 application archive for macOS should work out of the box now
2. a pip install scancode-toolkit==31.0.0rc1 will NOT work on macOS M1 unless you do either of this first:

• start your terminal in rosetta mode
• or first run this: arch -x86_64 /bin/bash

In essence we still need the rosetta X86 emulation layer for now. The app archive runs it. With a pip install you must do it yourself.

[pombredanne]

This is solved in #3053 and will be released with v31.
Thanks!