[mono][tvos] Libraries tests failed to build on Helix due to missing `System.Runtime.Serialization.Formatters.*` files

[ivanpovazan]

Build Information

Build: https://dev.azure.com/dnceng-public/public/_build/results?buildId=786206&view=results

Affected pipeline:

• runtime-extra-platforms

Build error leg:

• tvos_arm64_release_allsubsets_mono

Work items failing:

• System.Resources.Extensions.Tests
• System.Formats.Nrbf.Tests.WorkItemExecution

Relevant failure:

Device target detected but app not found, skipping signing..

Welcome to .NET 9.0!
---------------------
SDK Version: 9.0.100-preview.7.24407.12

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate, run 'dotnet dev-certs https --trust'
Learn about HTTPS: https://aka.ms/dotnet-https

----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
MSBuild version 17.12.0-preview-24374-02+48e81c6f1 for .NET
/private/tmp/helix/working/A62F0915/w/A5930953/e/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/tmp/helix/working/A62F0915/p/build/microsoft.netcore.app.runtime.tvos-arm64/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.pdb" because it was not found.
/private/tmp/helix/working/A62F0915/w/A5930953/e/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/tmp/helix/working/A62F0915/p/build/microsoft.netcore.app.runtime.tvos-arm64/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.xml" because it was not found.
Signing bundle /tmp/helix/working/A62F0915/w/A5930953/e/System.Resources.Extensions.Tests.app
No embedded provisioning profile found at /tmp/helix/working/A62F0915/w/A5930953/e/System.Resources.Extensions.Tests.app/embedded.mobileprovision! Failed to sign the app!

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorPattern": "System\\.Runtime\\.Serialization\\.Formatters\\..* because it was not found",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=786206
Error message validated: [System\.Runtime\.Serialization\.Formatters\..* because it was not found]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 8/27/2024 12:45:02 PM UTC

Report

Build	Definition	Test	Pull Request
792568	dotnet/runtime	System.Runtime.Tests.WorkItemExecution	#106858
786206	dotnet/runtime	System.Resources.Extensions.Tests.WorkItemExecution

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	1	2

[dotnet-policy-service]

Tagging subscribers to 'os-tvos': @vitek-karas, @kotlarmilos, @ivanpovazan, @steveisok, @akoeplinger
See info in area-owners.md if you want to be subscribed.

[dotnet-policy-service]

Tagging subscribers to this area: @directhex, @matouskozak
See info in area-owners.md if you want to be subscribed.

[ivanpovazan]

This could have been introduces with: #106737
I will follow-up with how we build tvos apps on Helix and why this started failing.
/cc: @ericstj

[dotnet-policy-service]

Tagging subscribers to 'binaryformatter-migration': @adamsitnik, @bartonjs, @jeffhandley, @terrajobst

[ericstj]

cc @adamsitnik

Why is it trying to copy PDB and XML?

/private/tmp/helix/working/A62F0915/w/B419093C/e/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/tmp/helix/working/A62F0915/p/build/microsoft.netcore.app.runtime.tvos-arm64/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.xml" because it was not found.
/private/tmp/helix/working/A62F0915/w/B419093C/e/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/tmp/helix/working/A62F0915/p/build/microsoft.netcore.app.runtime.tvos-arm64/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.pdb" because it was not found.

Here's the source:

runtime/src/mono/msbuild/apple/data/ProxyProjectForAOTOnHelix.proj

Lines 111 to 117 in ef0c712

	<ItemGroup>
	<_RuntimePackFiles Condition="%(_AppleUsedRuntimePackFiles.Extension) == '.dll' and %(_AppleUsedRuntimePackFiles.FileName) != 'System.Private.CoreLib'" Include="@(_AppleUsedRuntimePackFiles->'$(MicrosoftNetCoreAppRuntimePackLibDir)%(FileName)%(Extension)')" />
	<_RuntimePackFiles Condition="%(_AppleUsedRuntimePackFiles.Extension) != '.dll' or %(_AppleUsedRuntimePackFiles.FileName) == 'System.Private.CoreLib'" Include="@(_AppleUsedRuntimePackFiles->'$(MicrosoftNetCoreAppRuntimePackNativeDir)%(FileName)%(Extension)')" />
	</ItemGroup>
	<Copy SourceFiles="@(_RuntimePackFiles)"
	DestinationFolder="$(OriginalPublishDir)" />

Probably there is a difference in the items copied for a project reference vs the framework reference. I suspect there are some assumptions in that copying logic that are broken by a project reference being preferred.

[ivanpovazan]

Some more information about building ios/tvos test on CI.
We split the build into:

1. Build of runtime, libs and tests on build machines
2. AOT + bundling test apps on Helix machines

For 1. we collect all required files for a particular test, zip it and send over to Helix. The logic of picking up the right set of files is a bit tricky and it happens in this MSBuild target:

runtime/eng/testing/tests.ioslike.targets

Line 60 in c1a9f26

<Target Name="_BundleAOTTestAppleAppForHelix" DependsOnTargets="PrepareForAppleBuildApp">

So what I think that happens is that the System.Runtime.Serialization.Formatters.xml System.Runtime.Serialization.Formatters.pdb get passed over as required files through _AppleUsedRuntimePackFiles item group, but they are not carried over to / or restored on Helix, this probably happens since these file are present both in output of the test project build and in the runtime pack. Will investigate further.

---

FWIW to repro the failure locally one can do the following steps:

1. Build runtime/libs/tests:

./build.sh -s mono+libs+libs.tests -os tvos -arch arm64 -c Release /p:ArchiveTests=true /p:DevTeamProvisioning=- /p:RunAOTCompilation=true /p:RunSmokeTestsOnly=False /p:EnableAdditionalTimezoneChecks=true  /p:BuildDarwinFrameworks=true /p:IsManualOrRollingBuild=true /p:BuildTestsOnHelix=true /p:BuildDarwinFrameworks=true /p:UsePortableRuntimePack=true -binaryLog

2. Unzip the folder that would be sent to Helix, and cd to the publish dir

mkdir artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests
tar -xvf artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests.zip -C artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests
pushd artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests/publish

3. AOT the test and create the app bundle (use your repo root instead <Path-to-Your-Repo-Root>)

../../../../../../dotnet.sh msbuild ProxyProjectForAOTOnHelix.proj /p:RuntimeSrcDir=<Path-to-Your-Repo-Root> /p:RuntimeConfig=Release /p:RunAOTCompilation=true /p:UsePortableRuntimePack=true /p:TargetOS=tvos /p:TargetArchitecture=arm64 /p:MonoEnableLLVM=true /p:DevTeamProvisioning=- /p:Configuration=Release /bl:test.binlog

The step 3. should fail with:

/Users/ivan/repos/runtime-mono/artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/Users/ivan/repos/runtime-mono/artifacts/bin/microsoft.netcore.app.runtime.tvos-arm64/Release/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.pdb" because it was not found.
/Users/ivan/repos/runtime-mono/artifacts/helix/tests/tvos.AnyCPU.Release/System.Resources.Extensions.Tests/publish/ProxyProjectForAOTOnHelix.proj(116,5): error MSB3030: Could not copy the file "/Users/ivan/repos/runtime-mono/artifacts/bin/microsoft.netcore.app.runtime.tvos-arm64/Release/runtimes/tvos-arm64/native/System.Runtime.Serialization.Formatters.xml" because it was not found.