setPreferredAudioLanguage does not work when tracks are grouped together with adaptation-set-switching

[Pitel]

Version

Media3 1.4.1

More version details

No response

Devices that reproduce the issue

Pixel 6 running Android 16 beta

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Not tested

Reproduction steps

Load DASH manifest with multiple audio AdaptationSets linked together using <SupplementalProperty schemeIdUri="urn:mpeg:dash:adaptation-set-switching:2016" value="1,2,3"/>. Observe using EventLogger that they're in the same group:

10:12:27.634 Even...ger  D  tracks [eventTime=0.54, mediaPos=4040.00, window=0, period=0
10:12:27.634 Even...ger  D    group [
10:12:27.634 Even...ger  D      [X] Track:0, id=SFA_Coco_15209_org_V1_0100, mimeType=video/avc, container=video/mp4, bitrate=99000, codecs=avc1.4d400b, drm=[playready,widevine,cenc], res=320x180, fps=12.5, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:1, id=SFA_Coco_15209_org_V1_0200, mimeType=video/avc, container=video/mp4, bitrate=200000, codecs=avc1.4d400c, drm=[playready,widevine,cenc], res=320x180, fps=25.0, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:2, id=SFA_Coco_15209_org_V1_0300, mimeType=video/avc, container=video/mp4, bitrate=298000, codecs=avc1.4d400c, drm=[playready,widevine,cenc], res=320x180, fps=25.0, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:3, id=SFA_Coco_15209_org_V1_0500, mimeType=video/avc, container=video/mp4, bitrate=496000, codecs=avc1.4d400d, drm=[playready,widevine,cenc], res=384x216, fps=25.0, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:4, id=SFA_Coco_15209_org_V1_0800, mimeType=video/avc, container=video/mp4, bitrate=792000, codecs=avc1.4d4015, drm=[playready,widevine,cenc], res=512x288, fps=25.0, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:5, id=SFA_Coco_15209_org_V1_1200, mimeType=video/avc, container=video/mp4, bitrate=1198000, codecs=avc1.4d401e, drm=[playready,widevine,cenc], res=704x396, fps=25.0, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:6, id=SFA_Coco_15209_org_V1_1600, mimeType=video/avc, container=video/mp4, bitrate=1596000, codecs=avc1.4d401e, drm=[playready,widevine,cenc], res=852x480, fps=25.0, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:7, id=SFA_Coco_15209_org_V1_2200, mimeType=video/avc, container=video/mp4, bitrate=2194000, codecs=avc1.4d401f, drm=[playready,widevine,cenc], res=1024x576, fps=25.0, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:8, id=SFA_Coco_15209_org_V1_4000, mimeType=video/avc, container=video/mp4, bitrate=3987000, codecs=avc1.4d401f, drm=[playready,widevine,cenc], res=1280x720, fps=25.0, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:9, id=SFA_Coco_15209_org_V1_6000, mimeType=video/avc, container=video/mp4, bitrate=5985000, codecs=avc1.4d4028, drm=[playready,widevine,cenc], res=1920x1080, fps=25.0, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D    ]
10:12:27.634 Even...ger  D    group [
10:12:27.634 Even...ger  D      [X] Track:0, id=SFA_Coco_15209_dan_A1_048_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=54000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=da, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:1, id=SFA_Coco_15209_dan_A1_128_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=134000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=da, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:2, id=SFA_Coco_15209_dan_A1_192_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=197000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=da, roleFlags=[main], supported=YES
10:12:27.634 Even...ger  D      [X] Track:3, id=SFA_Coco_15209_fin_A1_048_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=54000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=fi, roleFlags=[alternate], supported=YES
10:12:27.634 Even...ger  D      [X] Track:4, id=SFA_Coco_15209_fin_A1_128_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=134000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=fi, roleFlags=[alternate], supported=YES
10:12:27.634 Even...ger  D      [X] Track:5, id=SFA_Coco_15209_fin_A1_192_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=197000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=fi, roleFlags=[alternate], supported=YES
10:12:27.635 Even...ger  D      [X] Track:6, id=SFA_Coco_15209_nor_A1_048_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=54000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=no, roleFlags=[alternate], supported=YES
10:12:27.635 Even...ger  D      [X] Track:7, id=SFA_Coco_15209_nor_A1_128_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=134000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=no, roleFlags=[alternate], supported=YES
10:12:27.635 Even...ger  D      [X] Track:8, id=SFA_Coco_15209_nor_A1_192_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=197000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=no, roleFlags=[alternate], supported=YES
10:12:27.635 Even...ger  D      [X] Track:9, id=SFA_Coco_15209_swe_A1_048_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=54000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=sv, roleFlags=[alternate], supported=YES
10:12:27.635 Even...ger  D      [X] Track:10, id=SFA_Coco_15209_swe_A1_128_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=134000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=sv, roleFlags=[alternate], supported=YES
10:12:27.635 Even...ger  D      [X] Track:11, id=SFA_Coco_15209_swe_A1_192_1, mimeType=audio/mp4a-latm, container=audio/mp4, bitrate=197000, codecs=mp4a.40.2, drm=[playready,widevine,cenc], channels=2, sample_rate=48000, language=sv, roleFlags=[alternate], supported=YES
10:12:27.635 Even...ger  D    ]
10:12:27.635 Even...ger  D    group [
10:12:27.635 Even...ger  D      [ ] Track:0, id=46997_DA_1, mimeType=application/x-media3-cues, container=application/mp4, bitrate=2000, codecs=application/ttml+xml stpp, language=da-dk, roleFlags=[subtitle], supported=YES
10:12:27.635 Even...ger  D    ]
10:12:27.635 Even...ger  D    group [
10:12:27.635 Even...ger  D      [ ] Track:0, id=46997_FI_1, mimeType=application/x-media3-cues, container=application/mp4, bitrate=2000, codecs=application/ttml+xml stpp, language=fi-fi, roleFlags=[subtitle], supported=YES
10:12:27.635 Even...ger  D    ]
10:12:27.635 Even...ger  D    group [
10:12:27.635 Even...ger  D      [ ] Track:0, id=46997_NO_1, mimeType=application/x-media3-cues, container=application/mp4, bitrate=2000, codecs=application/ttml+xml stpp, language=no-nob-no, roleFlags=[subtitle], supported=YES
10:12:27.635 Even...ger  D    ]
10:12:27.635 Even...ger  D    group [
10:12:27.635 Even...ger  D      [X] Track:0, id=46997_SV_1, mimeType=application/x-media3-cues, container=application/mp4, bitrate=2000, codecs=application/ttml+xml stpp, language=sv-se, roleFlags=[subtitle], supported=YES
10:12:27.635 Even...ger  D    ]
10:12:27.635 Even...ger  D  ]

Now try switching languages using

exo.setTrackSelectionParameters(
  exo.getTrackSelectionParameters().buildUpon()
    .setPreferredAudioLanguage("fi")
    .build()
);

Expected result

Audio language will change.

Actual result

Still only the first (Danish) audio language is being played.

But if I use a custom DashManifestParser which ignores those adaptation-set-switching propperties, it splits the tracks in groups, and language switching works. But is not seamless, whis is the point of that adaptation-set-switching propperty.

Media

Not sure if I can share it, but you can use any publicly available MPD manifest with multiple audio adaptation sets with different languages, just add the <SupplementalProperty schemeIdUri="urn:mpeg:dash:adaptation-set-switching:2016" value="1,2,3"/> to all of them to link them together (the value should always contain ids of the other adaptation sets).

Bug Report

• You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[tonihei]

Representations in the same AdaptationSet must represent "perceptually equivalent content" according to the DASH spec (ISO/IEC 23009-1:2022(E), section 4.3). So I think putting audio representations with different languages in the same adaptation set is not allowed. ExoPlayer puts all these Format objects in the same TrackGroup assuming they are interchangeable and will also not check the languages individually when selecting tracks. There is probably a error in logcat as well saying something like Different languages combined in one TrackGroup to highlight the issue.

AdaptationSets marked with adaptation-set-switching are more interesting because the spec says that they must "provide the same content or the content may be offered such that seamless switching is desired (for example in case of multiple viewpoints)" (section 5.3.3.5). ExoPlayer works on the assumption that they provide the same content with the same semantics as if they were in the same AdaptationSet. However it seems it's technically allowed to set this flag to signal that the representations are time-aligned and segments are not overlapping (so could offer seamless switching on demand).

One easy fix on your side would be to remove the adaptation-set-switching flag if the content is not actually the same (unless it's used for anything in particular?). At the same time, we can make a change in ExoPlayer to not merge the groups if content type, language or role flags are different.

[Pitel]

Yes, there is a warning in logcat about trackgroups and languages that I previously missed.

I'll advice our client to remove the adaptation-set-switching for the time being. They are using this just for different dubbing of the assets. (They should also use role: dub and not just alternate, I guess...)

But I think it also makes sense to make that change in the ExoPlayer's grouping you mentioned. 👍🏻 But I'd be worried if it doesn't brake something for someone else.

I was also reading the DASH spec, I found this paragraph in the page-break:

Suggesting one should not use adaptation-set-switching if AdaptationSets have different parameters. But i couldn't find what exactly is meant by parameters. They provide an example with spatial resolution, and in my opnion, language is also one of those important parameters. So maybe our client's manifest is indeed incorrect. But then I also tried running it through DASH-IF's validator, and it didn't mention any mistakes.

[tonihei]

Suggesting one should not use adaptation-set-switching if AdaptationSets have different parameters. But i couldn't find what exactly is meant by parameters.

I agree that this is strangely under-defined term here 🤔. The fact that they explicitly mention multiple viewpoints as an example (at the beginning of 5.3.3.5 that I quoted above) suggests to me that automatic switching based on bitrate shouldn't be assumed for all these adaptation-set-switching groups.