|
| 1 | +# Run Tests using Visual Studio task |
| 2 | + |
| 3 | +### Overview |
| 4 | + |
| 5 | +VSTest task can be used to run tests on Build agent machines. Apart from MSTest based tests, you can also run tests written using test frameworks like NUnit, xUnit, Mocha, Jasmine, etc. using the appropriate test adapters to Visual Studio. The task uses vstest.console.exe to execute tests and the command-line options available are documented [here](https://msdn.microsoft.com/en-us/library/jj155796.aspx) |
| 6 | + |
| 7 | +#### Execution Options |
| 8 | + |
| 9 | +Use the following options to select tests and control how the tests are run |
| 10 | + |
| 11 | +- **Test Assembly:** This is a required field. Use this to specify one or more test file names from which the tests should be picked. |
| 12 | + * Paths are relative to the 'Search Folder' input. |
| 13 | + * Multiple paths can be specified, one on each line. |
| 14 | + * Uses the minimatch patterns. Learn more about minimatch [here](https://aka.ms/minimatchexamples) |
| 15 | + |
| 16 | + Example 1: |
| 17 | + Most commonly your test projects follow a naming pattern such as `Product.Tests.dll`, `ProductTests.dll`, `Product.Test.dll`, `Product.UnitTests.dll` or similar. These dlls reside in your `bin` directory. To include all such test dlls use this pattern: |
| 18 | + |
| 19 | + ``` |
| 20 | + **\bin\**\*test.dll |
| 21 | + **\bin\**\*tests.dll |
| 22 | + ``` |
| 23 | + |
| 24 | + Example 2: |
| 25 | + When it is impossible to determine a naming convention for the tested dlls a wide include pattern can be used (notice the * before .dll). Such pattern can be followed by exclude patterns (starting with `!`) that excludes additional dlls. |
| 26 | + This pattern includes all dlls that have `test` in their name, and excludes all dlls from intermediate build `obj` directory: |
| 27 | + |
| 28 | + ``` |
| 29 | + **\*test*.dll |
| 30 | + !**\obj\** |
| 31 | + ``` |
| 32 | + |
| 33 | + This pattern is likely to include more dlls than you expect as many other dll names include *test* in their name, such as `MSTest.TestFramework.dll`, `Microsoft.VisualStudio.TestPlatform.ObjectModel.dll` etc. Please review your test log to see which dlls are included, and add appropriate excludes, such as: |
| 34 | + |
| 35 | + ``` |
| 36 | + **\*test*.dll |
| 37 | + !**\obj\** |
| 38 | + !**\*.resources.dll |
| 39 | + !**\*TestAdapter.dll |
| 40 | + !**\*Microsoft.*TestPlatform*.dll |
| 41 | + !**\*testhost*.dll |
| 42 | + !**\testcentric.engine.metadata.dll |
| 43 | + ``` |
| 44 | + |
| 45 | + |
| 46 | +- **Search Folder:** Use this to specify the folder to search for the test files. Defaults to `$(System.DefaultWorkingDirectory)` |
| 47 | + |
| 48 | +- **Test Filter Criteria:** Filters tests from within the test assembly files. For example, “Priority=1 | Name=MyTestMethod”. This option works the same way as the console option /TestCaseFilter of vstest.console.exe |
| 49 | + |
| 50 | +- **Run Settings File:** Path to a runsettings or testsettings file can be specified here. The path can be to a file in the repository or a path to file on disk. Use $(Build.SourcesDirectory) to access the root project folder. [Click here](https://msdn.microsoft.com/library/jj635153.aspx) for more information on these files. |
| 51 | + |
| 52 | +- **Override TestRun Parameters:** Override parameters defined in the TestRunParameters section of the runsettings file. For example: Platform=$(platform);Port=8080 |
| 53 | +[Click here](https://blogs.msdn.com/b/visualstudioalm/archive/2015/09/04/supplying-run-time-parameters-to-tests.aspx) for more information on overriding parameters. |
| 54 | + |
| 55 | +- **Code Coverage Enabled:** If set, this will collect code coverage information during the run and upload the results to the server. This is supported for .Net and C++ projects only. [Click here](https://msdn.microsoft.com/library/jj159530.aspx) to learn more about how to customize code coverage and manage inclusions and exclusions. |
| 56 | + |
| 57 | +- **Run in Parallel:** If set, tests will run in parallel leveraging available cores of the machine. [Click here](https://aka.ms/paralleltestexecution) to learn more about how tests are run in parallel. |
| 58 | + |
| 59 | +- **VSTest version:** Choose which version of Visual Studio (vstest.console.exe) to run tests with. |
| 60 | + |
| 61 | +- **Path to Custom Test Adapters:** Path to the testadapter for the framework in which the specified tests are written. Provided directory and all subdirectories are checked for testadapters. If there is a packages folder in the sources directory, it is automatically searched for testadapters. As a result, any testadapter downloaded as a Nuget package will be used without any input. For example, ‘$(Build.SourcesDirectory)\Fabrikam\packages’ |
| 62 | + |
| 63 | +- **Other console options:** Other options that can be provided to vstest.console.exe. For example, if you are using VSIX extensions, you can provide “/UseVsixExtensions:true”. These options are not supported and will be ignored when running tests using the ‘Multi agent’ parallel setting of an agent job or when running tests using ‘Test plan’ or 'Test run' option or when a custom batching option is selected. In these cases, the options can be specified using a runsettings file instead. |
| 64 | + |
| 65 | +#### Test Impact Analysis |
| 66 | + |
| 67 | +- **Run Only Impacted Tests:** If set, then only the relevant set of managed automated tests that need to be run to validate a given code change will be run. |
| 68 | + |
| 69 | +- **Number Of Builds After Which To Run All Tests:** This is an override that can be used to set the periodicity at which to automatically run the complete automated test suite. |
| 70 | + |
| 71 | +The Test Impact Analysis feature is available through the v2.\* (preview) version of the task. |
| 72 | + |
| 73 | +The feature is presently scoped to the following: |
| 74 | +- Dependencies |
| 75 | + - **Requires use of the v2.\* (preview) of the VSTest task in the build definition.** |
| 76 | + - **Requires VS 2015 Update 3 or VS 2017 RC and above on the build agent** |
| 77 | +- Supported |
| 78 | + - Managed code |
| 79 | + - CI and in PR workflows |
| 80 | + - IIS interactions |
| 81 | + - Automated Tests (unit tests, functional tests) - the tests and the application must be running on the same machine. |
| 82 | + - Build vNext, with multiple Test Tasks |
| 83 | + - Local and Hosted build agents (you will need VS 2015 Update 3 or VS2017 RC and above – please see “Dependencies”) |
| 84 | + - Git, GitHub, External Git, TFVC repos |
| 85 | +- Not yet supported |
| 86 | + - Remote testing (where the test is exercising an app deployed to a different machine) |
| 87 | + - No xplat support (Windows only). |
| 88 | + - No UWP support. |
| 89 | + |
| 90 | + Learn more about Test Impact [here](https://aka.ms/tialearnmore) |
| 91 | + |
| 92 | + |
| 93 | +#### Advanced Execution Options |
| 94 | + |
| 95 | +- **Batch tests:** A batch is a group of tests. A batch of tests runs at a time and results are published for that batch. If the phase in which the task runs is set to use multiple agents, each agent picks up any available batches of tests to run in parallel. Choose one of the below mentioned options for batching. |
| 96 | + - **Based on number of tests and agents:** Simple batching based on the number of tests and agents participating in the test run. |
| 97 | + - **Based on past running time of tests:** This batching considers past running time to create batches of tests such that each batch has approximately equal running time. |
| 98 | + - **Based on test assemblies:** Tests from an assembly are batched together. |
| 99 | + |
| 100 | + Learn more about batching options [here](https://aka.ms/vstestbatchingoptions) |
| 101 | + |
| 102 | +- **Do not distribute tests and replicate instead when multiple agents are used in the phase:** Choosing this option will not distribute tests across agents when the task is running in a multi-agent phase. Each of the selected test(s) will be repeated on each agent. The option is not applicable when the agent phase is configured to run with no parallelism or with the multi-config option. |
| 103 | + |
| 104 | +#### Reporting Options |
| 105 | +Use the following options to report desired information for the test run that can be used when analyzing runs. |
| 106 | + |
| 107 | +- **Test Run Title:** Provide a name for the test run. |
| 108 | + |
| 109 | +- **Platform:** Build platform against which the test run should be reported. This field is used for reporting purposes only. If you are using the Build – Visual Studio template, this is already defined for you. For example, x64 or x86. If you have defined a variable for platform in your build task, use that here. |
| 110 | + |
| 111 | +- **Configuration:** Build configuration against which the Test Run should be reported. Field is used for reporting purposes only. If you are using the Build – Visual Studio template, this is already defined for you. For example, Debug or Release. If you have defined a variable for configuration in your build task, use that here. |
| 112 | + |
| 113 | +- **Upload test attachments:** If set, any test run level attachments such as the TRX file will be uploaded. |
0 commit comments