feat: add option to fail for incomplete analysis

[maxrake]

This change adds a flag named --fail-incomplete to set a failure code when some analysis results are unknown. This is useful in environments where strictness is desired to prevent adding any new dependency until it passes established policy. This is a mutually exclusive option with --audit.

The name --strict was also considered but not used because that name implies a "strict mode" which would be more than just the opposite of audit mode. For instance, the ability to ignore errors with the --ignore-errors flag should not be possible in a true strict mode. So, the more specific option name was used to limit the effects of specifying it.

Testing

The changes in this PR are available for testing with the maxrake/phylum-ci:stricter Docker image found on Docker Hub.

The --audit and --fail-incomplete options are mutually exclusive:

Details

❯ poetry run phylum-ci -vvaf -p delme --audit --fail-incomplete -d requirements-incomplete.txt
usage: phylum-ci [-h] [-V] [-v | -q] [-d [DEPFILE ...]] [-e [EXCLUDE ...]] [-a] [-f] [-k TOKEN] [-p PROJECT] [-o ORG] [-g GROUP] [-s]
                 [--fail-incomplete | --audit] [--ignore-errors] [-r VERSION] [-t TARGET] [-u URI] [-i]
phylum-ci: error: argument --fail-incomplete: not allowed with argument --audit

What it looks like for the current released version of phylum-ci when analysis is incomplete:

Details

❯ phylum-ci -vvaf -p delme -d requirements-incomplete.txt
DEBUG    Using package: phylum v0.54.0
DEBUG    Logging initialized to level 10 (DEBUG)
DEBUG    Called with args: ['-vvaf', '-p', 'delme', '-d', 'requirements-incomplete.txt']
DEBUG    Phylum CLI version not specified
DEBUG    Found installed Phylum CLI version: v7.1.5
DEBUG    The installed Phylum CLI version is supported
INFO     Using Phylum CLI version: v7.1.5
DEBUG    No CI environment detected
INFO     Confirming prerequisites ...
DEBUG    `git` binary found on the PATH
DEBUG    Git repository root found: /Users/maxrake/dev/phylum/phylum-ci
INFO     All prerequisites met
DEBUG    Existing Phylum CLI instance found: v7.1.5 at /Users/maxrake/.local/bin/phylum
INFO     Using Phylum CLI instance: v7.1.5 at /Users/maxrake/.local/bin/phylum
DEBUG    Project name provided as argument: delme
INFO     Attempting to create a Phylum project with name: delme ...
DEBUG    Org name not provided as argument. Checking Phylum settings file ...
DEBUG    Org name not found in Phylum settings file. Assuming no org ...
DEBUG    Group name not provided as argument. Checking project file ...
DEBUG    Group name not found in `.phylum_project` file or file does not exist. Assuming no group ...
INFO     Project/org/group combo already exists. Continuing with it.
             Project: delme
             Org:     (no org)
             Group:   (no group)
DEBUG    Repository URL not available to set
DEBUG    Dependency files provided as arguments: [requirements-incomplete.txt]
INFO     Parsing requirements-incomplete.txt as auto dependency file. Manifests take longer.
DEBUG    Determining viability of the Phylum sandbox in this environment ...
DEBUG    Executing command: /Users/maxrake/.local/bin/phylum sandbox --allow-run / true
INFO     The Phylum sandbox works in this environment and will be enabled
DEBUG    Using parse command: /Users/maxrake/.local/bin/phylum parse --type auto /Users/maxrake/dev/phylum/phylum-ci/requirements-incomplete.txt
DEBUG    Running command from: /Users/maxrake/dev/phylum/phylum-ci
WARNING  Provided dependency file requirements-incomplete.txt is a lockifest.
         It will be treated as a manifest.
         For more info, see: https://docs.phylum.io/cli/lockfile_generation
WARNING  At least one manifest file was included.
         Forcing analysis to ensure updated dependencies are included.
DEBUG    Valid provided dependency files: [requirements-incomplete.txt]
DEBUG    Dependency files in use: [requirements-incomplete.txt]
INFO     Forced analysis specified or otherwise set. Proceeding with analysis.
INFO     Label to use for analysis: No-CI_stricter_99673df
DEBUG    5 unique current dependencies from 1 file
INFO     Considering all current dependencies ...
INFO     Performing analysis. This may take a few seconds.
DEBUG    Using analysis command: /Users/maxrake/.local/bin/phylum extension run --yes
         /Users/maxrake/.local/pipx/venvs/phylum/lib/python3.12/site-packages/phylum/exts/ci delme No-CI_stricter_99673df
         /var/folders/gh/wnf14j7n4q34y2t36hq2jz800000gn/T/base_69ed6g6i.json /var/folders/gh/wnf14j7n4q34y2t36hq2jz800000gn/T/curr_9q9_ucqm.json
WARNING  There were no failures in the packages that have completed so far
DEBUG    Analysis output:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                                      Phylum OSS Supply Chain Risk Analysis - INCOMPLETE                                                      ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

The analysis contains 1 package(s) Phylum has not yet processed, preventing a complete risk analysis. Phylum is processing these packages currently and should complete soon. Please wait for up to 30 minutes, then re-run the analysis.

View this project in the Phylum UI (https://app.phylum.io/projects/353a4a33-1abe-4e51-9508-d73ea73ca129?label=No-CI_stricter_99673df)
DEBUG    Return code: 0
         More info: https://github.com/phylum-dev/phylum-ci#exit-codes

❯ echo $?
0

What it looks like to submit the same dependency file, but using the new flag provided in this PR:

Details

❯ poetry run phylum-ci -vvaf -p delme --fail-incomplete -d requirements-incomplete.txt
DEBUG    Using package: phylum v0.54.0
DEBUG    Logging initialized to level 10 (DEBUG)
DEBUG    Called with args: ['-vvaf', '-p', 'delme', '--fail-incomplete', '-d', 'requirements-incomplete.txt']
DEBUG    Phylum CLI version not specified
DEBUG    Found installed Phylum CLI version: v7.1.5
DEBUG    The installed Phylum CLI version is supported
INFO     Using Phylum CLI version: v7.1.5
DEBUG    No CI environment detected
INFO     Confirming prerequisites ...
DEBUG    `git` binary found on the PATH
DEBUG    Git repository root found: /Users/maxrake/dev/phylum/phylum-ci
INFO     All prerequisites met
DEBUG    Existing Phylum CLI instance found: v7.1.5 at /Users/maxrake/.local/bin/phylum
INFO     Using Phylum CLI instance: v7.1.5 at /Users/maxrake/.local/bin/phylum
DEBUG    Project name provided as argument: delme
INFO     Attempting to create a Phylum project with name: delme ...
DEBUG    Org name not provided as argument. Checking Phylum settings file ...
DEBUG    Org name not found in Phylum settings file. Assuming no org ...
DEBUG    Group name not provided as argument. Checking project file ...
DEBUG    Group name not found in `.phylum_project` file or file does not exist. Assuming no group ...
INFO     Project/org/group combo already exists. Continuing with it.
             Project: delme
             Org:     (no org)
             Group:   (no group)
DEBUG    Repository URL not available to set
DEBUG    Dependency files provided as arguments: [requirements-incomplete.txt]
INFO     Parsing requirements-incomplete.txt as auto dependency file. Manifests take longer.
DEBUG    Determining viability of the Phylum sandbox in this environment ...
DEBUG    Executing command: /Users/maxrake/.local/bin/phylum sandbox --allow-run / true
INFO     The Phylum sandbox works in this environment and will be enabled
DEBUG    Using parse command: /Users/maxrake/.local/bin/phylum parse --type auto /Users/maxrake/dev/phylum/phylum-ci/requirements-incomplete.txt
DEBUG    Running command from: /Users/maxrake/dev/phylum/phylum-ci
WARNING  Provided dependency file requirements-incomplete.txt is a lockifest.
         It will be treated as a manifest.
         For more info, see: https://docs.phylum.io/cli/lockfile_generation
WARNING  At least one manifest file was included.
         Forcing analysis to ensure updated dependencies are included.
DEBUG    Valid provided dependency files: [requirements-incomplete.txt]
DEBUG    Dependency files in use: [requirements-incomplete.txt]
INFO     Forced analysis specified or otherwise set. Proceeding with analysis.
INFO     Label to use for analysis: No-CI_stricter_99673df
DEBUG    5 unique current dependencies from 1 file
INFO     Considering all current dependencies ...
INFO     Performing analysis. This may take a few seconds.
DEBUG    Using analysis command: /Users/maxrake/.local/bin/phylum extension run --yes /Users/maxrake/dev/phylum/phylum-ci/src/phylum/exts/ci delme
         No-CI_stricter_99673df /var/folders/gh/wnf14j7n4q34y2t36hq2jz800000gn/T/base_nrso_gay.json
         /var/folders/gh/wnf14j7n4q34y2t36hq2jz800000gn/T/curr_o61j5bx3.json
WARNING  There were no failures in the packages that have completed so far
INFO     --fail-incomplete specified. Setting return code to: 5
DEBUG    Analysis output:

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                                                      Phylum OSS Supply Chain Risk Analysis - INCOMPLETE                                                      ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛

The analysis contains 1 package(s) Phylum has not yet processed, preventing a complete risk analysis. Phylum is processing these packages currently and should complete soon. Please wait for up to 30 minutes, then re-run the analysis.

View this project in the Phylum UI (https://app.phylum.io/projects/353a4a33-1abe-4e51-9508-d73ea73ca129?label=No-CI_stricter_99673df)
DEBUG    Return code: 5
         More info: https://github.com/phylum-dev/phylum-ci#exit-codes

❯ echo $?
5

TODO

• Another PR for the phylum-analyze-pr-action repo will be created to
  update the documentation there to match the changes here
  • docs: update exit code section phylum-analyze-pr-action#44