Implement an FfmpegVideoRenderer

[EE-GSlomin]

I apologize for filing this as an "issue", unfortunately github doesn't provide too many avenues of communication.

I have considered writing my own FfmpegVideoRenderer (or perhaps using libx264 directly) to software decode unsupported formats for viewing on the phone. I was mostly curious if this has been attempted before, and dropped due to technical reasons or perhaps someone had already planned on working on it?

[ojw28]

We don't have any plans to provide an Ffmpeg video renderer. I'm fairly sure it's perfectly feasible to implement one, although note that in many cases software decoders will not be as power efficient or as performant as those provided by the platform. Marking as an enhancement in any case.

[EE-GSlomin]

Appreciate the response. I'll definitely take a crack at this then. The idea was that it would only be used as a fallback.

There was one thing I wasn't 100% sure of though. Most cases of unsupported video in my use case is generally because of extreme resolutions (3-8MP cameras). Would it make more sense to scale the image using ffmpeg/x264? Or rely on the rasterizer to downscale the final result?

[andrewlewis]

• Last time I checked, libx264 was only for encoding (not decoding) but FFmpeg does have an H.264/AVC decoder so it makes sense to wrap that. This would also give the option of using other codecs. Some parts of the existing ffmpeg audio extension could be reused.
• The vp9 extension may be a useful starting point.
• For scaling and color space conversion: it might be necessary to do some experiments to find out whether this is best done inside FFmpeg or separately on its output. In general, the decoder has to keep full-resolution frames in memory. The vp9 extension has code for doing color space conversion from YCbCr to RGB using GLES, rendering to a GLSurfaceView.

[EE-GSlomin]

Yep you would be correct. I always assumed it supported decoding, but I never dug into the libx264 side of it too much until now.

Sounds like I'll have to do a bit of benchmarking to come up with the ideal solution. I will definitely look at the VP9 extension more. Thanks for the advice.

[michalliu]

It would be great to have a ffmpeg video render

[EE-GSlomin]

I ended up not pursuing this on my end because of the licensing issues with FFMPEG.

[goffioul]

I've started writing a skeleton of FfmpegVideoRenderer. At the moment it's mainly a lot of placeholders with plumbing to a ffmpeg mpeg2video decoder. However the decoding fails in ffmpeg, because of invalid input data.

The "render" method is very similar to other available renderers: wait for input format, then drain output buffers, then feed input buffers. What I noticed is that the data returned by "readSource" (inherited from BaseRenderer) only contains the picture data (with start code 00000100), but I also need to feed all data to the ffmpeg decoder, in particular the sequence segment (with start code 000001b3). I have no idea why I only get the picture segments, and not the entire MPEG2 stream. Any hint appreciated.

[goffioul]

I've compared the data that is processed by MediaCodecRender (my Android device has a MPEG2 decoder builtin), and it appears that its calls to "readSource" also only get the picture segments. Is that a property of the implementation of H262Reader? (my sample clip is a MPEG/TS with MPEG2/MP2). If yes, is there a workaround to make the renderer to get the full MPEG2 data stream?

[goffioul]

My initial assessment was incorrect. The MPEG2 stream is segmented by H262Reader into TrackOutput on the START_PICTURE boundaries. That's why I see that start code when I inspect the starting bytes of the data received my renderer. However the problem is that the data between the start of the MPEG2 stream and the first START_PICTURE is lost and does not reach the renderer. And this is where the sequence header is located (the stream starts with sequence, extended sequence, GOP and then the first START_PICTURE), which is required by ffmpeg decoder.

[ojw28]

You'll probably find what you're looking for in the Format that's provided to the renderer. Specifically in Format.initializationData.

[goffioul]

Thanks, I hadn't realized that the initialization data contained the sequence and sequence extension headers. They're pre-parsed by the mpeg2video codec automatically when set as extradata on the context. Now I can get my FfmpegVideoRenderer to work.

[michalliu]

Hi @goffioul could you share you code regarding FfmpegVideoRenderer thanks. I'd like to implement the very same functionality.