Connect-AzAccount: inconsistent behaviour when using WhatIf flag - breaks tests #12511
Comments
ghost
added
question
|
@rodolfograve , what's the expected result when using WhatIf? I didn't figure out the meaning of -Whatif to Connect-AzAccount. |
|
Hi, and thanks for the response. WhatIf is used very frequently for testing purposes, as a way to make the commands not to perform any action that would require a real resource to be affected in any way. I agree that in the case of Connect-AzAccount it's not obvious what WhatIf should do if you look at it from the "hypothetical" excecution point of view. However, it's easier to define if we come to it from the "how would you test a script that makes use of Connect-AzAccount" angle. I suggest something like: "Connect-AzAccount -WhatIf" should return a fake PSAzureProfile object, either with well-known, hard-coded, fake Tenant and Subscription ids, or using random values. In terms of code, my suggestion is that we add an else here: azure-powershell/src/Accounts/Accounts/Account/ConnectAzureRmAccount.cs Lines 329 to 338 in f1b7543 This is pseudo-code. I haven't looked at the classes involved to see if they have a constructor we could use, etc. but it should be possible to make those modifications: Another option is that I'm just wrong trying to use -WhatIf for this purpose, but I haven't found any other way to write tests for PowerShell scripts that target resources. Any suggestions that work around this issue are very welcome. |
|
@rodolfograve , many az powershell cmdlets support @mikefrobbins , @dcaro , @markcowl for your awareness |
|
@dingmeng-xue , thanks again. I think there are two separate issues/topics here:
I think #1 is a fundamental issue. As the author of a script that uses a script that uses a script (...) that uses Connect-AzAccount, I don't expect things to fail only because I added the WhatIf flag. Everyone expects WhatIf to change the behaviour of the script so that it doesn't perform any real operations, but not that it changes other aspects of the integration of the script with other scripts. If possible, I would like to focus on that aspect as the most important. As for #2, the main reason is that we are trying to avoid having credentials embedded in the code. This creates a security problem, but more importantly, it makes your tests fails if the "right" environment is not available:
In an attempt to illustrate this one (examples are always dangerous), imagine I want to test the Do-Something command below. Please, keep in mind that in practice the scripts are much more complicated and nested: Function Connect-ToMySubscription {
[CmdletBinding(SupportsShouldProcess, ConfirmImpact="Low")]
Param()
$Credentials = Get-MyCredentialsFromEnvironment # So that the script connects to the right subscription according to the environment
Connect-AzAccount -Credentials $Credentials -Tenant "MyProductionTenant" -Subscription "MyProductionSubscription" -ServicePrincipal
}
Function Get-Something {
[CmdletBinding(SupportsShouldProcess, ConfirmImpact="Low")]
Param()
Connect-ToMySubscription
# Execute some Get- Az commands
$Result = Get-....
return $Result
}I think issue #1 is well evidenced here. Get-Something would return different things with -WhatIf than without. #2 is harder to show but I hope you can see why we would like to avoid any valid Service Principal credentials available on a developer's workstation, as that would mean they could impersonate that Service Principal. This goes against some of our security guidelines:
And it would be a shame to water down our very secure approach to managing secrets only because we are not able to test the scripts :-) |
|
Hi @dingmeng-xue . Do you have any updates on this? Maybe @mikefrobbins , @dcaro , @markcowl? |
|
I'm revisiting
|
|
Thanks @dingmeng-xue . That definition of -WhatIf does not mention anything about returning values and what happens when scripts interact using -WhatIf. I think I gave a couple of arguments in my previous comment. Is there any chance the Azure PowerShell team can take a look and address those arguments? Test-ability is a very important feature of a framework. If you see Azure PowerShell as something people use interactively then you won't care too much, but there is a huge potential for people to write complicated scripts and even entire modules that build on top of Azure PowerShell (as we are!), and the current behaviour for -WhatIf is a deal breaker for that scenario. |
|
@rodolfograve , I understand your point from test perspective. But I'm still confused about expected behavior. Do you have any example from PowerShell core modules for my reference? Below I tried |
|
@dingmeng-xue apologies for the delay, I was disconnected for a few days. I can't find any examples of commands that return a value when -WhatIf, so it looks like I'm making an unfair request to the PowerShell Az team. That does not mean the existing pattern is right, tough. I can't think of any other way in which we can test our scripts. At this point all I can say is that I would appreciate any guidance on:
Thanks a lot for all your help so far. |
|
Sorry that we have no test approach using a mock up
I cannot figure out other approaches. If you know any way in practice, please let me know. |
|
Close this issue now. Please contact us if you need further help. |
Uh oh!
There was an error while loading. Please reload this page.
Description
The Connect-AzAccount function writes an object to the standard output when not using -WhatIf. However, it doesn't when -WhatIf is present.
Since WhatIf is key when writing tests, this breaks tests for commands that use Connect-AzAccount and return a value of its own. This is because of PowerShell's behaviour to treat any standard output as a value returned by the function.
I believe this is the section causing the issue:
azure-powershell/src/Accounts/Accounts/Account/ConnectAzureRmAccount.cs
Lines 329 to 338 in f1b7543
WriteObject(...) is inside the if (ShouldProcess...) and there is no else to return some fake value.
Writing tests for PowerShell scripts is already hard enough, particularly for those scripts that target some infrastructure (most of them?). Hopefully key frameworks like PowerShell Az can relieve some of the pain by providing first-class support for testing.
To be clear, the issue here is that my tests would pass (they use WhatIf) but my scripts will fail when used for real.
Steps to reproduce
Environment data
Module versions
Debug output
Error output
The text was updated successfully, but these errors were encountered: