[REGRESSION]: AzureFileCopyV6 uses wrong path for azcopy.exe #19705

DHowett · 2024-03-28T16:06:40Z

New issue checklist

Task name

AzureFileCopy

Breaking task version

6

Last working task version

5

Regression Description

Builds consuming AzureFileCopy@6 fail with:

FAILING OUTPUT

& azcopy copy "C:\__w\1\s\_out\*"

The term 'azcopy' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

PASSING OUTPUT

& "AzCopy\AzCopy.exe" copy "C:\__w\1\s\_out\*"

Notice the different path to AzCopy.exe.

I have pinpointed where in the task this change was made. This diff is the reason AzCopy is not found. The utility code is now ignoring the $azCopyExeLocation variable.

diff --git a/Utility.ps1 b/Utility.ps1
index 94beb25e9f..0d8272dd3d 100644
--- a/Utility.ps1
+++ b/Utility.ps1
@@ -242,23 +231,22 @@ function Upload-FilesToAzureContainer
         if ($useSanitizerActivate) {
             # Splitting arguments on space, but not on space inside quotes
             $sanitizedArguments = [regex]::Split($additionalArguments, ' (?=(?:[^"]|"[^"]*")*$)')
             Write-Output "##[command] & azcopy copy `"$sourcePath`" `"$containerURL`" $sanitizedArguments"
-            & azcopy copy $sourcePath $containerURL$containerSasToken $sanitizedArguments
+            & azcopy copy $sourcePath $containerURL $sanitizedArguments
         } else {
-            Write-Output "##[command] & `"$azCopyExeLocation`" copy `"$sourcePath`" `"$containerURL`"  $additionalArguments"
-            $uploadToBlobCommand = "& `"$azCopyExeLocation`" copy `"$sourcePath`" `"$containerURL`" $additionalArguments"
-            Invoke-Expression $uploadToBlobCommand
+            Write-Output "##[command] & azcopy copy `"$sourcePath`" `"$containerURL`" $additionalArguments"
+            $azCopyCommand = "azcopy copy `"$sourcePath`" `"$containerURL`"  $additionalArguments"
+            Invoke-Expression -Command $azCopyCommand
         }

Regressed in #19650

This change was made in every instance where azcopy is invoked. /cc @v-mohithgc

Environment type (Please select at least one enviroment where you face this issue)

Self-Hosted
Microsoft Hosted
VMSS Pool
Container

Azure DevOps Server type

dev.azure.com (formerly visualstudio.com)

Azure DevOps Server Version (if applicable)

No response

Operation system

Windows

Relevant log output

Log output included above.

Full task logs with system.debug enabled

Log output included above.

Repro steps

No response

The text was updated successfully, but these errors were encountered:

geekzter · 2024-03-28T17:10:46Z

@DHowett Thanks for creating the issue in this repo. Is this happening on Microsoft hosted or agent or a self-hosted agent? If the latter, what PowerShell version is installed?
@v-mohithgc @manolerazvan

DHowett · 2024-03-28T17:13:21Z

Technically self-hosted; Microsoft internal (OneBranch). However... the issues I pointed out in the script are true regardless of the agent and powershell version, I believe. 🙂

DHowett · 2024-03-28T19:19:53Z

Yes, judging by the contents of the task's make.json:

    "externals": {
        "archivePackages": [
            {
                "url": "https://vstsagenttools.blob.core.windows.net/tools/azcopy/10.12/AzCopy.zip",
                "dest": "./"
            }
        ],
// ...

... and the contents of that archive ...

$ tar tf .\AzCopy.zip
AzCopy/AzCopy.exe

the path to azcopy.exe must contain AzCopy/, otherwise the task is actually taking a dependency on the azcopy.exe available in the environment. This means that the task, instead of using AzCopy 10.12 which it ships with, will use whatever AzCopy is on the machine already. That can result in significant regressions if the agent contains an older AzCopy.

rdumitru · 2024-03-30T17:57:01Z

We're also encountering the exact same issue for our internal Microsoft pipelines. v4 worked really well, but we've been waiting for v6 to use it with workload identity service connections.

DevTestUser123 · 2024-04-04T06:19:20Z

Still getting same error:

failed with error: 'The term 'azcopy' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.' For more info please refer to https://aka.ms/azurefilecopyreadme

@v-mohithgc @DHowett

v-mohithgc · 2024-04-04T06:52:13Z

Still getting same error:

failed with error: 'The term 'azcopy' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.' For more info please refer to https://aka.ms/azurefilecopyreadme

@v-mohithgc @DHowett

Hi @DevTestUser123, the changes have been merged today and it will take 5-7 working days to be deployed on all the regions.
The new version of the task (ie 6.238.0) will contain the fix.

DHowett · 2024-04-11T14:47:55Z

@DevTestUser123 Look, friend, I just reported the bug and the fine folks here are all subscribed to the thread. You don't need to ping all of us for every question.

v-mohithgc · 2024-04-16T13:17:22Z

The new version of this task is out, which will fix the issue. please check and confirm back. Thanks

DevTestUser123 · 2024-04-16T14:30:32Z

No still not working on self hosted agent.

v-mohithgc · 2024-04-17T04:57:45Z

No still not working on self hosted agent.

What is the task version picked up by your agent?

DevTestUser123 · 2024-04-17T05:09:05Z

6.238.9

No still not working on self hosted agent.

What is the task version picked up by your agent?

v-mohithgc · 2024-04-17T05:16:21Z

6.238.9

No still not working on self hosted agent.

What is the task version picked up by your agent?

Many users have confirmed that the new version of the task is working fine (in both Microsoft and self hosted agents), I suspect your error might be a different issue, mostly to do with your pipeline configurations.
If possible, send the debug logs to [email protected]

rdumitru · 2024-04-17T15:23:20Z

New version just dropped for us, but it's failing with this error:
Failed to perform Auto-login: PSContextCredentialexec: "pwsh": executable file not found in %PATH%.

v-snalawade · 2024-04-22T12:33:20Z

New version just dropped for us, but it's failing with this error: Failed to perform Auto-login: PSContextCredentialexec: "pwsh": executable file not found in %PATH%.

what is powershell version installed?

cyclelabs-ryanberger · 2024-04-23T19:20:24Z

We are also still receiving this error on a self-hosted Windows 2019 Server based image. Any ideas on how to resolve? We are using a WIF for our Azure DevOps Service connection for this task. This was part of why we upgraded to version 6.

PowerShell Version: 5.1.14393.5582
AzFileCopy Version: 6.238.9

`Starting: AzureFileCopy
==============================================================================
Task         : Azure file copy
Description  : Copy files to Azure Blob Storage or virtual machines
Version      : 6.238.9
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/devops/pipelines/tasks/deploy/azure-file-copy
==============================================================================
Added TLS 1.2 in session.
Import-Module -Name C:\Users\AzDevOps\Documents\WindowsPowerShell\Modules\Az.Accounts\2.17.0\Az.Accounts.psd1 -Global
Clear-AzContext -Scope CurrentUser -Force -ErrorAction SilentlyContinue
Clear-AzContext -Scope Process
Connect-AzAccount System.Collections.Hashtable

Set-AzContext System.Collections.Hashtable
Import-Module -Name C:\Users\AzDevOps\Documents\WindowsPowerShell\Modules\Az.Resources\6.16.1\Az.Resources.psd1 -Global
Import-Module -Name C:\Users\AzDevOps\Documents\WindowsPowerShell\Modules\Az.Storage\6.1.3\Az.Storage.psd1 -Global
Import-Module -Name C:\Users\AzDevOps\Documents\WindowsPowerShell\Modules\Az.Compute\7.2.0\Az.Compute.psd1 -Global
Import-Module -Name C:\Users\AzDevOps\Documents\WindowsPowerShell\Modules\Az.Network\7.4.1\Az.Network.psd1 -Global
##[warning]The names of some imported commands from the module 'Microsoft.Azure.PowerShell.Cmdlets.Network' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
##[warning]The names of some imported commands from the module 'Az.Network' include unapproved verbs that might make them less discoverable. To find the commands with unapproved verbs, run the Import-Module command again with the Verbose parameter. For a list of approved verbs, type Get-Verb.
 mime: C:\a\_tasks\AzureFileCopy_eb72cb01-a7e5-427b-a8a1-1b31ccac8a43\6.238.9\MimeMapping.json
Account                              SubscriptionName TenantId                             Environment
-------                              ---------------- --------                             -----------
REDACTED AZURE TENANT/SUBSCRIPTION

Uploading files from source path: 'C:\a\1\s\*.exe' to storage account: 'REDACTED' in container: 'REDACTED' with blob prefix: 'REDACTED-2-buildPipeline-20240423.7'
 & "AzCopy\AzCopy.exe" copy "C:\a\1\s\*.exe" "[REDACTED"](REDACTED)  --log-level=INFO --recursive
INFO: Scanning...

Failed to perform Auto-login: PSContextCredentialexec: "pwsh": executable file not found in %PATH%.`

DevTestUser123 · 2024-04-26T13:01:58Z

Which is the required PowerShell version to use so that I cant get an error?

ghost · 2024-04-27T20:32:28Z

Also getting the error

Failed to perform Auto-login: PSContextCredentialexec: "pwsh": executable file not found in %PATH%.`

Switching to version 5 works (I had to do this first https://stackoverflow.com/questions/52769758/azure-blob-storage-authorization-permission-mismatch-error-for-get-request-wit )

DevTestUser123 · 2024-04-28T12:30:12Z

Also getting the error

Failed to perform Auto-login: PSContextCredentialexec: "pwsh": executable file not found in %PATH%.`

Switching to version 5 works (I had to do this first https://stackoverflow.com/questions/52769758/azure-blob-storage-authorization-permission-mismatch-error-for-get-request-wit )

I am already using version 5 still didn't work for me, also gave required storage permission

cyclelabs-ryanberger · 2024-04-28T13:34:48Z

The fix for us was installing powershell 7 on our self hosted image. I saw that the error was referencing pwsh.exe (powershell 7) and I only had 5 on our agent. I added 7 alongside of it and this fixed the error for us.

…

On Sun, Apr 28, 2024 at 8:30 AM DevTestUser123 ***@***.***> wrote: Also getting the error Failed to perform Auto-login: PSContextCredentialexec: "pwsh": executable file not found in %PATH%.` Switching to version 5 works (I had to do this first https://stackoverflow.com/questions/52769758/azure-blob-storage-authorization-permission-mismatch-error-for-get-request-wit ) we are already using version 5 still didn't work for me, also gave required storage permission — Reply to this email directly, view it on GitHub <#19705 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOLST6UQTHQFZ2TJDVYFU33Y7TTWVAVCNFSM6AAAAABFNB7CQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBRGQ3DSNZRHE> . You are receiving this because you commented.Message ID: ***@***.***>

DevTestUser123 · 2024-06-06T09:18:55Z

I am not able to see convert option for previously created service connection, can someone pls help ?

It should be like this as per DOC:

But there is no convert option showing currently:

v-schhabra · 2024-06-20T07:54:21Z

I am not able to see convert option for previously created service connection, can someone pls help ?

It should be like this as per DOC:

But there is no convert option showing currently:

Could you please try following other options to create WIF service connection?

ghost · 2024-06-26T05:46:15Z

The fix for us was installing powershell 7 on our self hosted image. I saw that the error was referencing pwsh.exe (powershell 7) and I only had 5 on our agent. I added 7 alongside of it and this fixed the error for us.
…
On Sun, Apr 28, 2024 at 8:30 AM DevTestUser123 @.> wrote: Also getting the error Failed to perform Auto-login: PSContextCredentialexec: "pwsh": executable file not found in %PATH%.` Switching to version 5 works (I had to do this first https://stackoverflow.com/questions/52769758/azure-blob-storage-authorization-permission-mismatch-error-for-get-request-wit ) we are already using version 5 still didn't work for me, also gave required storage permission — Reply to this email directly, view it on GitHub <#19705 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOLST6UQTHQFZ2TJDVYFU33Y7TTWVAVCNFSM6AAAAABFNB7CQKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBRGQ3DSNZRHE . You are receiving this because you commented.Message ID: @.>

Thank you! this fixed it for me 😄

Had to restart my self hosted agent service afterwards and all was good.

DHowett added bug regression This used to work, but a change in the service/tasks broke it. labels Mar 28, 2024

DHowett mentioned this issue Mar 28, 2024

Support for AAD Workload Identity Azure/azure-storage-azcopy#2545

Closed

github-actions bot added Area: Release triage labels Mar 28, 2024

v-mohithgc mentioned this issue Apr 3, 2024

AzureFileCopyV6: Fix usage of wrong path for azcopy.exe #19715

Merged

2 tasks

v-mohithgc added p1 Priority 1 (should have immediate attention) Area:RM RM task team and removed triage labels Apr 3, 2024

v-schhabra closed this as completed Jun 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[REGRESSION]: AzureFileCopyV6 uses wrong path for azcopy.exe #19705

[REGRESSION]: AzureFileCopyV6 uses wrong path for azcopy.exe #19705

DHowett commented Mar 28, 2024

geekzter commented Mar 28, 2024

DHowett commented Mar 28, 2024

DHowett commented Mar 28, 2024

rdumitru commented Mar 30, 2024

DevTestUser123 commented Apr 4, 2024 •

edited

Loading

v-mohithgc commented Apr 4, 2024

DHowett commented Apr 11, 2024

v-mohithgc commented Apr 16, 2024

DevTestUser123 commented Apr 16, 2024

v-mohithgc commented Apr 17, 2024

DevTestUser123 commented Apr 17, 2024 •

edited

Loading

v-mohithgc commented Apr 17, 2024 •

edited

Loading

rdumitru commented Apr 17, 2024

v-snalawade commented Apr 22, 2024

cyclelabs-ryanberger commented Apr 23, 2024 •

edited

Loading

DevTestUser123 commented Apr 26, 2024 •

edited

Loading

ghost commented Apr 27, 2024 •

edited by ghost

Loading

DevTestUser123 commented Apr 28, 2024 •

edited

Loading

cyclelabs-ryanberger commented Apr 28, 2024 via email

DevTestUser123 commented Jun 6, 2024 •

edited

Loading

v-schhabra commented Jun 20, 2024

ghost commented Jun 26, 2024 •

edited by ghost

Loading

[REGRESSION]: AzureFileCopyV6 uses wrong path for azcopy.exe #19705

[REGRESSION]: AzureFileCopyV6 uses wrong path for azcopy.exe #19705

Comments

DHowett commented Mar 28, 2024

New issue checklist

Task name

Breaking task version

Last working task version

Regression Description

FAILING OUTPUT

PASSING OUTPUT

Environment type (Please select at least one enviroment where you face this issue)

Azure DevOps Server type

Azure DevOps Server Version (if applicable)

Operation system

Relevant log output

Full task logs with system.debug enabled

Repro steps

geekzter commented Mar 28, 2024

DHowett commented Mar 28, 2024

DHowett commented Mar 28, 2024

rdumitru commented Mar 30, 2024

DevTestUser123 commented Apr 4, 2024 • edited Loading

v-mohithgc commented Apr 4, 2024

DHowett commented Apr 11, 2024

v-mohithgc commented Apr 16, 2024

DevTestUser123 commented Apr 16, 2024

v-mohithgc commented Apr 17, 2024

DevTestUser123 commented Apr 17, 2024 • edited Loading

v-mohithgc commented Apr 17, 2024 • edited Loading

rdumitru commented Apr 17, 2024

v-snalawade commented Apr 22, 2024

cyclelabs-ryanberger commented Apr 23, 2024 • edited Loading

DevTestUser123 commented Apr 26, 2024 • edited Loading

ghost commented Apr 27, 2024 • edited by ghost Loading

DevTestUser123 commented Apr 28, 2024 • edited Loading

cyclelabs-ryanberger commented Apr 28, 2024 via email

DevTestUser123 commented Jun 6, 2024 • edited Loading

v-schhabra commented Jun 20, 2024

ghost commented Jun 26, 2024 • edited by ghost Loading

DevTestUser123 commented Apr 4, 2024 •

edited

Loading

DevTestUser123 commented Apr 17, 2024 •

edited

Loading

v-mohithgc commented Apr 17, 2024 •

edited

Loading

cyclelabs-ryanberger commented Apr 23, 2024 •

edited

Loading

DevTestUser123 commented Apr 26, 2024 •

edited

Loading

ghost commented Apr 27, 2024 •

edited by ghost

Loading

DevTestUser123 commented Apr 28, 2024 •

edited

Loading

DevTestUser123 commented Jun 6, 2024 •

edited

Loading

ghost commented Jun 26, 2024 •

edited by ghost

Loading