[PackageValidation] Add support to do baseline package validation when the new one has dropped supported TFMs

[joperezr]

Related: #22901 (comment)

It is becoming more and more common for newer versions of packages (particularly for new major versions of them) to drop support for unsupported Target Frameworks. Even packages produced from dotnet itself does this. An example of this is the following:

I have a Package Foo v1.0.0 which originally supported net6.0. After a few years, I ship Foo v2.0.0 which drops support of net6.0 as that is out of support now and instead I target net8.0 now, but I still want my surface area of Foo (net8.0) to be compatible with the previous one I shipped (net6.0) in order to not break my existing customers that are using package 1.0.0.

Like explained in the case above, a lot of developers will care about not making breaking changes to their API even when dropping support of target frameworks, as they will still want that other libraries that consumed their 1.0.0 version would still work in an app that targets net8.0 and that lifts Foo to v2.0.0. While PackageValidation feature will work in Foo today, you are currently not able to do baseline package validation (meaning ensuring you don't break a previously shipped API), since that relies on the new package never dropping support of all TFMs. The way it works today if you don't drop TFMs is that the following comparisons are made which ensures you maintain compatibility:

• (baseline package) net6.0 compared to (new package) net6.0
• (new package) net6.0 compared to (new package) net8.0

This is the way we ensure today that you are not breaking old APIs, but this feature request is asking to see if we can support this while dropping the net6.0 in the new package so that instead we compare:

• (baseline package) net6.0 compared to (new package) net8.0

Proposal

One way to do this, would be to add a special set of properties that allow you to specify this remapping when performing baseline validation like:

    <EnablePackageValidation>true</EnablePackageValidation>
    <PackageValidationBaselineVersion>1.0.0</PackageValidationBaselineVersion>
  <PropertyGroup>

  <ItemGroup>
    <!-- new -->
    <PackageValidationBaselineTargetFrameworkMapping Include="net6.0" MapTo="net8.0" />

cc: @ViktorHofer @dotMorten

[danmoseley]

@ViktorHofer do you have an idea how much work this would be?

right now, aspnetcore is essentially relying on PR reviews to avoid breaking API, which isn't reliable enough

[ViktorHofer]

@joperezr and I talked about this offline and I support the change and asked him to file an issue. Adding a new input and flowing it through the UI and business layer is trivial. The tricky part is interpreting what now is "left" (baseline) and "right" (current). I expect APICompat to have various places that expect left to be the baseline. With the proposed change the direction would be swapped.

Maybe there's an alternative to swapping the direction of the compatibility tuple that I'm not thinking of.

Just as a guess, I don't think this should be much work. Less than a week for someone like Jo or me who know the code base.

[joperezr]

I’m not really following why you’d need to swap places. A way to think about it, is imagine both of these assemblies where on the same package. In that case, you’d have a comparison happening between net6 (left) and net8 (right). This is the same case, except the net6 assembly is on the baseline while the net8 is on the new package. Treating them as if they were on the same package would be the right approach IMO, as that would still allow you to make compatible additions to the net8 surface, while still making sure you are staying compatible with the old net6 one.

[ViktorHofer]

Treating them as if they were on the same package would be the right approach IMO

That would circumvent the baseline conditions: https://github.com/search?q=repo%3Adotnet%2Fsdk%20IsBaselineComparison&type=code

It's important to treat assemblies from baseline packages as a baseline comparison. That input is used when logging failures so that the user knows which assembly from which package failed and in a few other cases.

[joperezr]

Can you help me understand what is the difference (apart from logging)? If it's just logging, I think most people (or at least us) would be okay with that drawback as the messages would still make sense (there is no net6 target in the project so I can assume it is comparing it against the old package)

[ViktorHofer]

I was hoping that I explained that above. If we would treat a baseline assembly not as baseline, multiple things will be incorrect:

• Failures logged will point to an incorrect location (we log the path to the nuget package)
• Suppressions written into suppression files will be missing an IsBaselineSuppression entry.

I'm sure we can find a better way to support this scenario without sacrificing on the correctness of the outputs.

[joperezr]

Got it. Yeah fair enough. I was mostly trying to understand why there was a need to swap left and right, as well as pointing out that if it would be cheaper to implement this in a way that worked but meant some outputs would be wrong, then that would be an okay trade off as a stopgap (at least for us as something is better than nothing :P). To @danmoseley's question above, do you have any idea on the cost of what it would look like to support this?

[ViktorHofer]

Yeah rough estimate from above:

Just as a guess, I don't think this should be much work. Less than a week for someone like Jose or me who know the code base.

[ericstj]

I'm surprised we don't cover this already. I would expect API Compat to see that the new package has a net8.0 asset and it would resolve the baseline package for net8.0 and find it's net6.0 asset. The gist of the algorithm for identifying tuples with a baseline was "1. find all interesting TFMs 2. resolve both left and right for those TFMs 3. compare"

I see that today we're only looking at the baseline packages for interesting frameworks --

sdk/src/Compatibility/ApiCompat/Microsoft.DotNet.PackageValidation/Validators/BaselinePackageValidator.cs

Line 31 in 4d6b1e0

foreach (NuGetFramework baselineTargetFramework in options.BaselinePackage.FrameworksInPackage)

That's wrong since it won't capture interesting differences that might exist in the new package - even in lieu of removing support - like introducing a bug when adding new multi-targeting.

I think the fix here should just be to look at the union of frameworks from the old baseline and latest package.