vstest.console.exe not detecting data collectors for code coverage

[davidkeaveny]

I am attempting to run code coverage on a .NET Framework 4.7.2 application using vstest.console.exe, but when I run the tests from the command line, the tests themselves pass but a coverage report is not generated.

My test project file looks like this:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net472</TargetFramework>
    <DebugType>full</DebugType>
  <ItemGroup>
    <PackageReference Include="coverlet.collector" Version="6.0.4">
      <PrivateAssets>all</PrivateAssets>
      <IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
    </PackageReference>
    <PackageReference Include="FluentAssertions" Version="5.10.0" />
    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.13.0" />
    <PackageReference Include="NUnit" Version="3.14.0" />
    <PackageReference Include="NUnit3TestAdapter" Version="3.17.0">
      <PrivateAssets>all</PrivateAssets>
    </PackageReference>
  </ItemGroup>
</Project>

and if I run the following command from the solution root folder then all tests pass:

> vstest.console.exe MyProject.Tests\bin\Debug\net472\MyProject.Tests.dll

VSTest version 17.13.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
NUnit Adapter 3.17.0.0: Test execution started
Running all tests in C:\work\MyProject.Tests\bin\Debug\net472\MyProject.Tests.dll

All good so far; however, if I then change the command to run code coverage:

> vstest.console.exe MyProject.Tests\bin\Debug\net472\MyProject.Tests.dll /EnableCodeCoverage

VSTest version 17.13.0 (x64)

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Data collection : Unable to find a datacollector with friendly name 'Code Coverage'.
Data collection : Could not find data collector 'Code Coverage'
NUnit Adapter 3.17.0.0: Test execution started

So I then go and create CodeCoverage.runsettings as follows:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
    <DataCollectionRunSettings>
        <DataCollectors>
            <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
                <Configuration>
                    <CodeCoverage>
                        <ModulePaths>
                            <Exclude>
                                <ModulePath>.*FluentValidation.*</ModulePath>
                                <ModulePath>.*NUnit3.*</ModulePath>
                            </Exclude>
                        </ModulePaths>
                        <UseVerifiableInstrumentation>True</UseVerifiableInstrumentation>
                        <AllowLowIntegrityProcesses>True</AllowLowIntegrityProcesses>
                        <CollectFromChildProcesses>True</CollectFromChildProcesses>
                        <CollectAspDotNet>False</CollectAspDotNet>
                    </CodeCoverage>
                </Configuration>
            </DataCollector>
        </DataCollectors>
    </DataCollectionRunSettings>
</RunSettings>

and run my new command:

> vstest.console.exe MyProject.Tests\bin\Debug\net472\MyProject.Tests.dll /EnableCodeCoverage /Settings:CodeCoverage.runsettings

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
Data collection : Unable to find a datacollector with friendly name 'Code Coverage'.
Data collection : Could not find data collector 'Code Coverage'
NUnit Adapter 3.17.0.0: Test execution started

so no change at all, whether or not I specify the .runsettings file.

How can I get coverage to run? Should I be referencing coverlet.collector at all, or is that .NET Core-only assembly?

[nohwnd]

@fhnaseer please help here.

[fhnaseer]

@davidkeaveny Can you run it with /diag:log.vstest.txt and share log files?

Also, did you get vstest.console.exe from nuget package or somewhere else?

[davidkeaveny]

@fhnaseer at this point I'm assuming vstest.console.exe is coming from the Microsoft.NET.Test.Sdk package, as it has the same version number (17.13.0)?

I'm attaching the diagnostic logs.

debug.datacollector.25-04-16_08-44-10_77151_5.txt
debug.host.25-04-16_08-44-11_34042_5.txt
debug.txt

[fhnaseer]

@davidkeaveny You are using Professional edition of Visual Studio. Code Coverage components are not shipped with Professional edition.

Please download this package => https://www.nuget.org/packages/Microsoft.TestPlatform/17.13.0
And then run vstest.console.exe from microsoft.testplatform.17.13.0\tools\net462\Common7\IDE\Extensions\TestPlatform\vstest.console.exe and try again.

[davidkeaveny]

@fhnaseer the good news is that I can now see a .coverage report being generated in the TestResults folder. The bad news is that VSTest appears to be ignoring my CodeCoverage.runsettings - for example, if I specify Cobertura as the output format then it still writes a binary format instead of XML. I'm still using the same /EnableCodeCoverage /Settings:CodeCoverage.runsettings flags, just specifying the path to vstest.console.exe to the version provided in the Microsoft.TestPlatform NuGet package as you specified above.

If I change my command to use /EnableCodeCoverage /Collect:"Code Coverage;Format=Cobertura as suggested by https://learn.microsoft.com/en-us/visualstudio/test/customizing-code-coverage-analysis?view=vs-2022, instead of specifying my .runsettings file then I do see the code coverage report being generated in Cobertura format as .cobertura.xml. If I deliberately mis-spell the filename (e.g. /EnableCodeCoverage /Settings:CodeCoveragexxx.runsettings) then VSTest reports that the file cannot be found, so it is at least trying to load it, it just doesn't use the settings in the file when generating the coverage report.

[fhnaseer]

I am not sure if /EnableCodeCoverage and Collect are compatible with each other. That is something to investigate from our side.

But you can skip /EnableCodeCoverage and just use Collect and /Settings and it should work for you. You can try out the following options,

vstest.console.exe /Collect:"Code Coverage;Format=Cobertura"
vstest.console.exe /Collect:"Code Coverage" /Settings:CodeCoverage.runsettings
vstest.console.exe /Settings:CodeCoverage.runsettings => assuming that runsettings has coverage data collector

Also, you can use dotnet test and then you do not need to download microsoft.testplatform nuget package.

dotnet test --collect:"Code Coverage;format=Cobertura"
dotnet test --collect:"Code Coverage" --settings "CodeCoverage.runsettings"