Ubuntu 22.04 - Release 20250214.2.0 - Failing to install python packages

[sbonds]

Description

When by sheer luck I get 20250209.1.0 as my runner, everything works fine.

This appears very similar to existing issue #11602 but my error message is slightly different so I thought I would open a new issue rather than possibly confusing the troubleshooting in issue 11602.

Pipeline YAML

This is a demonstration pipeline so it's intentionally very simple, which makes troubleshooting that much simpler too. That first pip install https://github.com/mkdocs/mkdocs/archive/refs/tags/1.6.1.tar.gz fails.

trigger:
  - main

pool:
  vmImage: ubuntu-latest

jobs:
  - job: install_mkdocs
    displayName: Install software necessary to run mkdocs
    steps:
      - task: Bash@3
        displayName: Install Mkdocs
        inputs:
          targetType: inline
          script: pip install https://github.com/mkdocs/mkdocs/archive/refs/tags/1.6.1.tar.gz

      - task: Bash@3
        displayName: Install Mkdocs-Material plugin
        inputs:
          targetType: inline
          script: pip install --ignore-installed https://github.com/squidfunk/mkdocs-material/archive/refs/tags/9.6.4.tar.gz
        
      - task: Bash@3
        displayName: Install Mkdocs Awesome Pages plugin
        inputs:
          targetType: inline
          script: pip install --ignore-installed https://github.com/lukasgeiter/mkdocs-awesome-nav/archive/refs/tags/v2.10.1.tar.gz
                  
      - task: Bash@3
        displayName: Build site with mkdocs
        inputs:
          targetType: inline
          script: mkdocs build --strict --verbose --site-dir $(Build.ArtifactStagingDirectory)

      - task: PublishPipelineArtifact@1
        displayName: Upload static site content to Azure DevOps
        inputs:
          artifact: $(ARTIFACT_NAME)
          publishLocation: pipeline

Error message

The fatal error is Preparing metadata (pyproject.toml): finished with status 'error' due to ModuleNotFoundError: No module named 'packaging.licenses'. The complete step log is below under "actual behavior".

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

Image: ubuntu-22.04
Version: 20250214.2.0

Raw log from Initialize Job

This is the relevant section from that log, but is not the complete log:

2025-02-16T22:36:05.0603884Z ##[group]Runner Image
2025-02-16T22:36:05.0604127Z Image: ubuntu-22.04
2025-02-16T22:36:05.0604630Z Version: 20250214.2.0
2025-02-16T22:36:05.0604975Z Included Software: https://github.com/actions/runner-images/blob/ubuntu22/20250214.2/images/ubuntu/Ubuntu2204-Readme.md
2025-02-16T22:36:05.0605382Z Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20250214.2
2025-02-16T22:36:05.0605849Z ##[endgroup]
2025-02-16T22:36:05.0606080Z ##[group]Runner Image Provisioner
2025-02-16T22:36:05.0606658Z 2.0.422.1
2025-02-16T22:36:05.0606864Z ##[endgroup]
2025-02-16T22:36:05.0611228Z Current image version: '20250214.2.0'
2025-02-16T22:36:05.2260850Z Agent running as: 'vsts'

Is it regression?

https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20250209.1

Expected behavior

This is the log from the mkdocs install during a working run:

2025-02-15T23:23:42.2853409Z ##[section]Starting: Install Mkdocs
2025-02-15T23:23:42.2868237Z ==============================================================================
2025-02-15T23:23:42.2869029Z Task         : Bash
2025-02-15T23:23:42.2869242Z Description  : Run a Bash script on macOS, Linux, or Windows
2025-02-15T23:23:42.2869708Z Version      : 3.250.1
2025-02-15T23:23:42.2870121Z Author       : Microsoft Corporation
2025-02-15T23:23:42.2870370Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
2025-02-15T23:23:42.2870716Z ==============================================================================
2025-02-15T23:23:42.7677477Z Generating script.
2025-02-15T23:23:42.7691748Z Script contents:
2025-02-15T23:23:42.7692902Z pip install https://github.com/mkdocs/mkdocs/archive/refs/tags/1.4.3.tar.gz
2025-02-15T23:23:42.7695568Z ========================== Starting Command Output ===========================
2025-02-15T23:23:42.7707748Z [command]/usr/bin/bash /home/vsts/work/_temp/9101d8bf-0c25-4d25-b58c-8e2810924e91.sh
2025-02-15T23:23:43.4406009Z Defaulting to user installation because normal site-packages is not writeable
2025-02-15T23:23:43.8770193Z Collecting https://github.com/mkdocs/mkdocs/archive/refs/tags/1.4.3.tar.gz
2025-02-15T23:23:44.3224295Z   Downloading https://github.com/mkdocs/mkdocs/archive/refs/tags/1.4.3.tar.gz
2025-02-15T23:23:44.5534472Z      | 4.0 MB 19.6 MB/s 0:00:00
2025-02-15T23:23:47.0890608Z   Installing build dependencies: started
2025-02-15T23:23:50.2537920Z   Installing build dependencies: finished with status 'done'
2025-02-15T23:23:50.2580397Z   Getting requirements to build wheel: started
2025-02-15T23:23:50.3557440Z   Getting requirements to build wheel: finished with status 'done'
2025-02-15T23:23:50.5471829Z   Installing backend dependencies: started
2025-02-15T23:23:54.1691078Z   Installing backend dependencies: finished with status 'done'
2025-02-15T23:23:54.1720788Z   Preparing metadata (pyproject.toml): started
2025-02-15T23:23:54.3323434Z   Preparing metadata (pyproject.toml): finished with status 'done'
2025-02-15T23:23:54.3679126Z Requirement already satisfied: pyyaml>=5.1 in /usr/lib/python3/dist-packages (from mkdocs==1.4.3) (5.4.1)
2025-02-15T23:23:54.5149289Z Collecting markdown<3.4,>=3.2.1
2025-02-15T23:23:54.6188074Z   Downloading Markdown-3.3.7-py3-none-any.whl (97 kB)
2025-02-15T23:23:54.6707574Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 97.8/97.8 KB 2.6 MB/s eta 0:00:00
2025-02-15T23:23:54.6708732Z Requirement already satisfied: jinja2>=2.11.1 in /usr/lib/python3/dist-packages (from mkdocs==1.4.3) (3.0.3)
2025-02-15T23:23:54.6852764Z Collecting ghp-import>=1.0
2025-02-15T23:23:54.6967110Z   Downloading ghp_import-2.1.0-py3-none-any.whl (11 kB)
2025-02-15T23:23:54.7245474Z Collecting pyyaml-env-tag>=0.1
2025-02-15T23:23:54.7337784Z   Downloading pyyaml_env_tag-0.1-py3-none-any.whl (3.9 kB)
2025-02-15T23:23:54.7388980Z Requirement already satisfied: packaging>=20.5 in /usr/local/lib/python3.10/dist-packages (from mkdocs==1.4.3) (24.2)
2025-02-15T23:23:54.7400167Z Requirement already satisfied: click>=7.0 in /usr/lib/python3/dist-packages (from mkdocs==1.4.3) (8.0.3)
2025-02-15T23:23:54.7686099Z Collecting mergedeep>=1.3.4
2025-02-15T23:23:54.7776866Z   Downloading mergedeep-1.3.4-py3-none-any.whl (6.4 kB)
2025-02-15T23:23:54.9625185Z Collecting watchdog>=2.0
2025-02-15T23:23:54.9706669Z   Downloading watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl (79 kB)
2025-02-15T23:23:54.9850086Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.1/79.1 KB 7.3 MB/s eta 0:00:00
2025-02-15T23:23:55.0339401Z Collecting python-dateutil>=2.8.1
2025-02-15T23:23:55.0418867Z   Downloading python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
2025-02-15T23:23:55.0712581Z      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 229.9/229.9 KB 9.5 MB/s eta 0:00:00
2025-02-15T23:23:55.0916669Z Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil>=2.8.1->ghp-import>=1.0->mkdocs==1.4.3) (1.16.0)
2025-02-15T23:23:55.0960256Z Building wheels for collected packages: mkdocs
2025-02-15T23:23:55.0977792Z   Building wheel for mkdocs (pyproject.toml): started
2025-02-15T23:23:56.2526144Z   Building wheel for mkdocs (pyproject.toml): finished with status 'done'
2025-02-15T23:23:56.2674951Z   Created wheel for mkdocs: filename=mkdocs-1.4.3-py3-none-any.whl size=3654419 sha256=26fc6fab93bc114e267ea483ef145cffae34ac94db07244707a50ca3f76fa120
2025-02-15T23:23:56.2677018Z   Stored in directory: /tmp/pip-ephem-wheel-cache-lu2m2wmi/wheels/66/c5/e6/bca822c11657520d670caea27b5298ba4783740e4dab023daa
2025-02-15T23:23:56.2717075Z Successfully built mkdocs
2025-02-15T23:23:56.5305218Z Installing collected packages: watchdog, pyyaml-env-tag, python-dateutil, mergedeep, markdown, ghp-import, mkdocs
2025-02-15T23:23:56.9814219Z Successfully installed ghp-import-2.1.0 markdown-3.3.7 mergedeep-1.3.4 mkdocs-1.4.3 python-dateutil-2.9.0.post0 pyyaml-env-tag-0.1 watchdog-6.0.0
2025-02-15T23:23:57.1422382Z 
2025-02-15T23:23:57.1529754Z ##[section]Finishing: Install Mkdocs

Actual behavior

The pip process fails with Preparing metadata (pyproject.toml): finished with status 'error' due to ModuleNotFoundError: No module named 'packaging.licenses'

2025-02-16T22:17:42.8664047Z ##[section]Starting: Install Mkdocs
2025-02-16T22:17:42.8676378Z ==============================================================================
2025-02-16T22:17:42.8676783Z Task         : Bash
2025-02-16T22:17:42.8676968Z Description  : Run a Bash script on macOS, Linux, or Windows
2025-02-16T22:17:42.8677253Z Version      : 3.250.1
2025-02-16T22:17:42.8677477Z Author       : Microsoft Corporation
2025-02-16T22:17:42.8677702Z Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/bash
2025-02-16T22:17:42.8678172Z ==============================================================================
2025-02-16T22:17:43.3292095Z Generating script.
2025-02-16T22:17:43.3303592Z Script contents:
2025-02-16T22:17:43.3307023Z pip install https://github.com/mkdocs/mkdocs/archive/refs/tags/1.4.3.tar.gz
2025-02-16T22:17:43.3310484Z ========================== Starting Command Output ===========================
2025-02-16T22:17:43.3316840Z [command]/usr/bin/bash /home/vsts/work/_temp/a4aac199-1f08-46e9-92e1-457238595508.sh
2025-02-16T22:17:43.9615207Z Defaulting to user installation because normal site-packages is not writeable
2025-02-16T22:17:44.3692912Z Collecting https://github.com/mkdocs/mkdocs/archive/refs/tags/1.4.3.tar.gz
2025-02-16T22:17:44.9064766Z   Downloading https://github.com/mkdocs/mkdocs/archive/refs/tags/1.4.3.tar.gz
2025-02-16T22:17:45.1615018Z      | 4.0 MB 17.3 MB/s 0:00:00
2025-02-16T22:17:47.5155012Z   Installing build dependencies: started
2025-02-16T22:17:50.5499177Z   Installing build dependencies: finished with status 'done'
2025-02-16T22:17:50.5547284Z   Getting requirements to build wheel: started
2025-02-16T22:17:50.6494856Z   Getting requirements to build wheel: finished with status 'done'
2025-02-16T22:17:50.8328174Z   Installing backend dependencies: started
2025-02-16T22:17:54.0974611Z   Installing backend dependencies: finished with status 'done'
2025-02-16T22:17:54.1012666Z   Preparing metadata (pyproject.toml): started
2025-02-16T22:17:54.2207567Z   Preparing metadata (pyproject.toml): finished with status 'error'
2025-02-16T22:17:54.2300303Z   error: subprocess-exited-with-error
2025-02-16T22:17:54.2301008Z   
2025-02-16T22:17:54.2563428Z   × Preparing metadata (pyproject.toml) did not run successfully.
2025-02-16T22:17:54.2564861Z   │ exit code: 1
2025-02-16T22:17:54.2565133Z   ╰─> [14 lines of output]
2025-02-16T22:17:54.2565362Z       Traceback (most recent call last):
2025-02-16T22:17:54.2565820Z         File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
2025-02-16T22:17:54.2566066Z           main()
2025-02-16T22:17:54.2566292Z         File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
2025-02-16T22:17:54.2566526Z           json_out['return_val'] = hook(**hook_input['kwargs'])
2025-02-16T22:17:54.2566786Z         File "/usr/lib/python3/dist-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
2025-02-16T22:17:54.2567078Z           return hook(metadata_directory, config_settings)
2025-02-16T22:17:54.2567539Z         File "/tmp/pip-build-env-lctqrfd4/overlay/local/lib/python3.10/dist-packages/hatchling/build.py", line 117, in prepare_metadata_for_build_wheel
2025-02-16T22:17:54.2568479Z           f.write(builder.config.core_metadata_constructor(builder.metadata))
2025-02-16T22:17:54.2568808Z         File "/tmp/pip-build-env-lctqrfd4/overlay/local/lib/python3.10/dist-packages/hatchling/metadata/spec.py", line 546, in construct_metadata_file_2_4
2025-02-16T22:17:54.2569052Z           if metadata.core.license:
2025-02-16T22:17:54.2569317Z         File "/tmp/pip-build-env-lctqrfd4/overlay/local/lib/python3.10/dist-packages/hatchling/metadata/core.py", line 677, in license
2025-02-16T22:17:54.2569594Z           from packaging.licenses import canonicalize_license_expression
2025-02-16T22:17:54.2569803Z       ModuleNotFoundError: No module named 'packaging.licenses'
2025-02-16T22:17:54.2570173Z       [end of output]
2025-02-16T22:17:54.2570590Z   
2025-02-16T22:17:54.2570915Z   note: This error originates from a subprocess, and is likely not a problem with pip.
2025-02-16T22:17:54.2571187Z error: metadata-generation-failed
2025-02-16T22:17:54.2571258Z 
2025-02-16T22:17:54.2571544Z × Encountered error while generating package metadata.
2025-02-16T22:17:54.2571982Z ╰─> See above for output.
2025-02-16T22:17:54.2572054Z 
2025-02-16T22:17:54.2572250Z note: This is an issue with the package mentioned above, not pip.
2025-02-16T22:17:54.2572446Z hint: See above for details.
2025-02-16T22:17:54.3634324Z 
2025-02-16T22:17:54.3688979Z ##[error]Bash exited with code '1'.
2025-02-16T22:17:54.3714818Z ##[section]Finishing: Install Mkdocs

Repro steps

Create a pipeline running the "ubuntu-latest" running which uses a pip install command like pip install https://github.com/mkdocs/mkdocs/archive/refs/tags/1.6.1.tar.gz

Run the pipeline. If you randomly get a runner version 20250214.2.0 it will fail. If you get 20250209.1.0 it will succeed.

[Prabhatkumar59]

Hi @sbonds - Thank you for bringing this issue to our attention. We will look into this issue and will update you after investigating.


[sbonds]

My workaround was to use ubuntu 20.04. I removed the workaround since I noticed in issue #11602 that there was a solution to try. So far, all of my pipeline run attempts with "ubuntu-latest" are getting the 20250209.1.0 image. I'm not sure if this was the fix or if I'm just having "bad" luck in getting this known-good image.

I guess the good news is that ubuntu-latest is working reliably now, but I'd like to see how a newer-than-20250209.1.0 image works before considering it fixed.

[sbonds]

I'm still only getting 20250209.1.0 runners so I can't confirm that a newer runner version works.

[kishorekumar-anchala]

Hi @sbonds - Kindly try to run the pipeline with new rollout and confirm ?

[sbonds]

Runner image version 20250223.1.0 fails with the same error message.

[sbonds]

This seems to be an issue with pip 22.0.2. If I update to pip 25.0.1, it works. I also tested with pip 22.0.2 at a user-level install just in case there was something special with a non-system-level pip install, and pip 22.0.2 also failed in that config.

This also seems to be specific to an URL-based pip install. When I use the system pip 22.0.2 to install mkdocs via "pip install mkdocs==1.6.1" it works fine.