iMX6 hardware decode issue on Android

[Eric-BTG]

Hi,

I'm trying to use i.MX6 VPU to decode 1080p h264 stream on android platform.
I'm facing performances issue, stream is very laggy. I can see only few
images.
(However it works smoothly for 480p streams). I have tried various gst
version/plugins/pipelines and still have this issue. Help would be very
appreciated :)


My board is a Wandboard Quad, I run freescale's kk-4.4.2 android portage. My
kernel is freescale's 3.0.35 with Wandboard config.
My test stream is Big Buck Bunny trailer
(http://video.blendertestbuilds.de/download.blender.org/peach/trailer_1080p.mov).
I tried custom android app and https://github.com/sdroege/gst-player and
https://github.com/jaroslavas/Gstreamer-Android-example.


In few words my experiments are:

-I tried androidmedia plugins:
For this I patched androidmedia plugin (gstamc.c) to force slice_height to
GST_ROUND_UP_16 (height) (same fix as NVidia Tegra 3 Bug 748867
<https://bugzilla.gnome.org/show_bug.cgi?id=748867> ).
App is built for armeabi-v7a with armv7 gst plugins.
A lot of "Frame is too late, dropping" event occurs. Logcat log is joined
for gst-player run gst-player-androidmedia-gst-1.log
<http://gstreamer-devel.966125.n4.nabble.com/file/n4673449/gst-player-androidmedia-gst-1.log>
.
(I tried gst 1.4.5 and 1.5.90).


-I tried dv imxvpudec (gstreamer-imx) 0.11.1 plugin:
Result is almost the same, log is joined gst-player-imxvpudec-0.log
<http://gstreamer-devel.966125.n4.nabble.com/file/n4673449/gst-player-imxvpudec-0.log>
. (I plan to describe imxvpudec android issue in github project).


I don't understand where is the bottleneck. (I'm low CPU and low mem). I
can't figure out if this is the decode part or further in opengl display.

Any feedback on imx6 and android would be very appreciated.

Regards,

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Carlos Rafael Giani]

I see no indication that imxvpudec was ever used. Instead, it apparently
is using OpenMAX. Run this with GST_DEBUG=2,*imx*:5 .
Also, ditch the 3.0.35 kernel as soon as you can. It is known to be full
of severe bugs. Wandboard should have 3.14 by now.

On 09/03/2015 11:24 AM, Eric-BTG wrote:
> Hi,
>
> I'm trying to use i.MX6 VPU to decode 1080p h264 stream on android platform.
> I'm facing performances issue, stream is very laggy. I can see only few
> images.
> (However it works smoothly for 480p streams). I have tried various gst
> version/plugins/pipelines and still have this issue. Help would be very
> appreciated :)
>
>
> My board is a Wandboard Quad, I run freescale's kk-4.4.2 android portage. My
> kernel is freescale's 3.0.35 with Wandboard config.
> My test stream is Big Buck Bunny trailer
> (http://video.blendertestbuilds.de/download.blender.org/peach/trailer_1080p.mov).
> I tried custom android app and https://github.com/sdroege/gst-player and
> https://github.com/jaroslavas/Gstreamer-Android-example.
>
>
> In few words my experiments are:
>
> -I tried androidmedia plugins:
> For this I patched androidmedia plugin (gstamc.c) to force slice_height to
> GST_ROUND_UP_16 (height) (same fix as NVidia Tegra 3 Bug 748867
> <https://bugzilla.gnome.org/show_bug.cgi?id=748867> ).
> App is built for armeabi-v7a with armv7 gst plugins.
> A lot of "Frame is too late, dropping" event occurs. Logcat log is joined
> for gst-player run gst-player-androidmedia-gst-1.log
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4673449/gst-player-androidmedia-gst-1.log>
> .
> (I tried gst 1.4.5 and 1.5.90).
>
>
> -I tried dv imxvpudec (gstreamer-imx) 0.11.1 plugin:
> Result is almost the same, log is joined gst-player-imxvpudec-0.log
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4673449/gst-player-imxvpudec-0.log>
> . (I plan to describe imxvpudec android issue in github project).
>
>
> I don't understand where is the bottleneck. (I'm low CPU and low mem). I
> can't figure out if this is the decode part or further in opengl display.
>
> Any feedback on imx6 and android would be very appreciated.
>
> Regards,
>
> Eric
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eric-BTG]

Thanks for your reply,

In fact I have 2 logs, gst-player-androidmedia-gst-1.5.90.log
<http://gstreamer-devel.966125.n4.nabble.com/file/n4673455/gst-player-androidmedia-gst-1.log>
with androidmedia which uses freescale's OpenMAX libs.

And second log gst-player-imxvpudec-0.11.1-gst-1.5.90.log
<http://gstreamer-devel.966125.n4.nabble.com/file/n4673455/gst-player-imxvpudec-0.log>
in which one I include imxvpu in plugin declaration. In this log we can see
vpu-lib alloc.

In this case, to start decode I must chmod /dev/mxc_vpu.
(If not I get this error):


Here ith the log with gst_debug_set_threshold_from_string("2,*imx*:5",
TRUE):
imxvpudec-debug-imx-5.log
<http://gstreamer-devel.966125.n4.nabble.com/file/n4673455/imxvpudec-debug-imx-5.log>

Regards,

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673455.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Carlos Rafael Giani]

On 09/03/2015 02:23 PM, Eric-BTG wrote:
> Thanks for your reply,
>
> In fact I have 2 logs, gst-player-androidmedia-gst-1.5.90.log
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4673455/gst-player-androidmedia-gst-1.log>
> with androidmedia which uses freescale's OpenMAX libs.
>
> And second log gst-player-imxvpudec-0.11.1-gst-1.5.90.log
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4673455/gst-player-imxvpudec-0.log>
> in which one I include imxvpu in plugin declaration. In this log we can see
> vpu-lib alloc.
>
> In this case, to start decode I must chmod /dev/mxc_vpu.
> (If not I get this error):
>
>
> Here ith the log with gst_debug_set_threshold_from_string("2,*imx*:5",
> TRUE):
> imxvpudec-debug-imx-5.log
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4673455/imxvpudec-debug-imx-5.log>
>
> Regards,
>
> Eric
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673455.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

This is a 1080p video, and you are using glimagesink. It is possible
that the CPU-based frame copies are using up too much bandwidth/CPU
time. Re-run with 2,*imx*:5,*imxvpu*:9 .
Also, if possible, try what happens if you replace the glimagesink with
a fakesink (but set the fakesink's sync property to TRUE). If the
framedrops then stop, then it is most likely the CPU copies that cause
the problem.

[Eric-BTG]

Hum.. unfortunately I must use glimagesink because of android :-/

Here is the log with 2,*imx*:5,*imxvpu*:9:
imxvpudec-debug-imx-5-imxvpu-9.log
<http://gstreamer-devel.966125.n4.nabble.com/file/n4673457/imxvpudec-debug-imx-5-imxvpu-9.log>

I will try fakesink and let you know.

Concerning 3.0.35 kernel, I must deal with it because it would implies
changing android's portage..

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673457.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Eric-BTG]

I did fakesink experiment. I can't replace glimagesink with fakesink as is
because my app needs a real sink to map to displayed texture. So I created a
sink displaying a viteotest while I decode stream to give to fakesink.

I use custom app, and my pipeline is:
"filesrc location=/sdcard/Movies/trailer_1080p.mov ! qtdemux ! queue !
h264parse ! imxvpudec ! queue ! fakesink sync=true videotestsrc !
videoconvert ! autovideosink"

And as you expected no more frames are dropped.. Log is
imxvpudec-fakesink.log
<http://gstreamer-devel.966125.n4.nabble.com/file/n4673465/imxvpudec-fakesink.log>


Do you have a idea to cope with CPU copies? Or it means that I have to patch
glimagesink to make GPU based copies?

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673465.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Bing Song]

VPU decoder will output physical continues video frame buffer, glimagesink can map this kind of buffer using one private API, needn't video frame buffer copy.

Regards,
Song Bing.

________________________________________
From: gstreamer-devel <gstreamer-devel-***@lists.freedesktop.org> on behalf of Eric-BTG <***@bt-ground.com>
Sent: Thursday, September 3, 2015 11:15 PM
To: gstreamer-***@lists.freedesktop.org
Subject: Re: iMX6 hardware decode issue on Android

I did fakesink experiment. I can't replace glimagesink with fakesink as is
because my app needs a real sink to map to displayed texture. So I created a
sink displaying a viteotest while I decode stream to give to fakesink.

I use custom app, and my pipeline is:
"filesrc location=/sdcard/Movies/trailer_1080p.mov ! qtdemux ! queue !
h264parse ! imxvpudec ! queue ! fakesink sync=true videotestsrc !
videoconvert ! autovideosink"

And as you expected no more frames are dropped.. Log is
imxvpudec-fakesink.log
<http://gstreamer-devel.966125.n4.nabble.com/file/n4673465/imxvpudec-fakesink.log>


Do you have a idea to cope with CPU copies? Or it means that I have to patch
glimagesink to make GPU based copies?

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673465.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
gstreamer-***@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Carlos Rafael Giani]

You are referring to the Vivante video texture extension. While it does
allow for zerocopy by using DMA to the video framebuffer, it also
requires custom, Vivante-specific code, which glimagesink doesn't have.
This is why I wrote eglvivsink in gstreamer-imx. Ideally, it would be
possible to hook up the Vivante stuff through GLUploadMeta, but the
discussion about this has been going on for a while:
http://lists.freedesktop.org/archives/gstreamer-bugs/2014-April/123097.html

Am 2015-09-04 um 14:32 schrieb Bing Song:
> VPU decoder will output physical continues video frame buffer, glimagesink can map this kind of buffer using one private API, needn't video frame buffer copy.
>
> Regards,
> Song Bing.
>
> ________________________________________
> From: gstreamer-devel <gstreamer-devel-***@lists.freedesktop.org> on behalf of Eric-BTG <***@bt-ground.com>
> Sent: Thursday, September 3, 2015 11:15 PM
> To: gstreamer-***@lists.freedesktop.org
> Subject: Re: iMX6 hardware decode issue on Android
>
> I did fakesink experiment. I can't replace glimagesink with fakesink as is
> because my app needs a real sink to map to displayed texture. So I created a
> sink displaying a viteotest while I decode stream to give to fakesink.
>
> I use custom app, and my pipeline is:
> "filesrc location=/sdcard/Movies/trailer_1080p.mov ! qtdemux ! queue !
> h264parse ! imxvpudec ! queue ! fakesink sync=true videotestsrc !
> videoconvert ! autovideosink"
>
> And as you expected no more frames are dropped.. Log is
> imxvpudec-fakesink.log
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4673465/imxvpudec-fakesink.log>
>
>
> Do you have a idea to cope with CPU copies? Or it means that I have to patch
> glimagesink to make GPU based copies?
>
> Eric
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673465.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Sebastian Dröge]

On Fr, 2015-09-04 at 15:29 +0200, Carlos Rafael Giani wrote:
> You are referring to the Vivante video texture extension. While it
> does allow for zerocopy by using DMA to the video framebuffer, it
> also requires custom, Vivante-specific code, which glimagesink
> doesn't have.
> This is why I wrote eglvivsink in gstreamer-imx. Ideally, it would be
> possible to hook up the Vivante stuff through GLUploadMeta, but the
> discussion about this has been going on for a while:
> http://lists.freedesktop.org/archives/gstreamer-bugs/2014-April/12309
> 7.html

Alternatively if that makes more sense, it would also be possible to
add special code paths for that kind of stuff to glimagesink (or more
specific the glupload object).

--
Sebastian Dröge, Centricular Ltd · http://www.centricular.com

[Eric-BTG]

Hi,

A feedback from my experiments.

So I had 2 possibilities: use imxeglvivsink or use Freescale code path to
glimagesink (or patch myself glupload). I tried both:


*# imxeglvivsink:* I used posted patch. It works but display is not smooth.
CPU is avg 3%. I'm stuck, I don't know where is the bottle neck. I can
provide logs if that helps.

*# Freescale glimagesink patchs:* I tried
"1.4.5-Use-viv-direct-texture-to-bind-buffer.patch" +
"egl-workaround-for-eglCreateContext-isn-t-thread-safe.patch" from Freescale
latest Yocto release.
I can't use it as is because it maps texture from "vpudec" bufs output only
(Freescale's implem only). Mapping is done from memory allocated with
GstAllocatorPhyMem which is part of gst1.0-fsl-plugins. I don't have those
plugins since I use Carlos imxvpudec..


*# Patch glupload in glimagesink:* So I patched glupload (1.4.5) to use
direct Vivante texture binding from imxvpudec's physical buffers.
(Basically I merged eglvivsink's gles2_renderer with Freescale's patch)

I got physical buffers meta data from GST_IMX_PHYS_MEM_META_GET macro and
used Vivante glTexDirectVIVMap to map incoming buffers to the gl texture.

It works but it is still laggy, (a bit less smooth than imxeglvivsink).


My texture format is I420, could you tell me more about transparent
conversion to RGBA?

Display is laggy but CPU is very low ~3%, do you have a explanation on
what's going on?


I traced eglSwapBuffers execution:
imxeglvivsink: avg 32.5ms, med 30ms
glimagesink: avg 33.9ms, med 30ms

Do you have a experiment to suggest to identify the issue? Any hint is
welcomed :)

Regards,

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673691.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Nicolas Dufresne]

Le vendredi 04 septembre 2015 à 15:29 +0200, Carlos Rafael Giani a
écrit :
> You are referring to the Vivante video texture extension. While it
> does
> allow for zerocopy by using DMA to the video framebuffer, it also
> requires custom, Vivante-specific code, which glimagesink doesn't
> have.
> This is why I wrote eglvivsink in gstreamer-imx. Ideally, it would be
> possible to hook up the Vivante stuff through GLUploadMeta, but the
> discussion about this has been going on for a while:
> http://lists.freedesktop.org/archives/gstreamer-bugs/2014-April/12309
> 7.html

I think the discussion is not relevant to implementing the upload. We
where just trying to explain that you need to do some caps negotiation
and not just attach the meta. The reason is that after the upload, you
most likely have an RGBA texture (the caps MAY needs to reflect that).

Nicolas

[Carlos Rafael Giani]

Am 2015-09-04 um 15:51 schrieb Nicolas Dufresne:
> Le vendredi 04 septembre 2015 à 15:29 +0200, Carlos Rafael Giani a
> écrit :
>> You are referring to the Vivante video texture extension. While it
>> does
>> allow for zerocopy by using DMA to the video framebuffer, it also
>> requires custom, Vivante-specific code, which glimagesink doesn't
>> have.
>> This is why I wrote eglvivsink in gstreamer-imx. Ideally, it would be
>> possible to hook up the Vivante stuff through GLUploadMeta, but the
>> discussion about this has been going on for a while:
>> http://lists.freedesktop.org/archives/gstreamer-bugs/2014-April/12309
>> 7.html
> I think the discussion is not relevant to implementing the upload. We
> where just trying to explain that you need to do some caps negotiation
> and not just attach the meta. The reason is that after the upload, you
> most likely have an RGBA texture (the caps MAY needs to reflect that).
>
> Nicolas
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

Well the main problem was that the Vivante stuff can implicitely do
color space conversion, transparently exposing YUV textures as RGB(A)
ones to OpenGL ES shaders, and this is not possible with this meta,
since you cannot specify two caps (the input format - typically some YUV
format - and the format that the glimagesink textures - some RGB format).

[Nicolas Dufresne]

Le vendredi 04 septembre 2015 à 16:02 +0200, Carlos Rafael Giani a
écrit :
> Well the main problem was that the Vivante stuff can implicitely do
> color space conversion, transparently exposing YUV textures as RGB(A)
> ones to OpenGL ES shaders, and this is not possible with this meta,
> since you cannot specify two caps (the input format - typically some
> YUV format - and the format that the glimagesink textures - some RGB
> format).

That's something you can negotiate. This is the same thing with VAAPI
btw. Use a caps feature "meta:GLUpload" and if you have this feature,
set format=RGBA. Same as gtreamer-vaapi here.

Nicolas

[Carlos Rafael Giani]

So, if I get this right, then the decoder would check if downstream
supports this caps feature. If so, it sets the caps to format=RGBA and
feature=gluploadmeta. Internally, the buffer holds I420 data, but the
Vivante direct texture based upload code will then transparently convert
to RGBA. So far so good.

But what if there is a branch downstream, for example because of a tee
element, and one element supports this caps feature, the other doesn't?

Am 2015-09-04 um 17:08 schrieb Nicolas Dufresne:
> Le vendredi 04 septembre 2015 à 16:02 +0200, Carlos Rafael Giani a
> écrit :
>> Well the main problem was that the Vivante stuff can implicitely do
>> color space conversion, transparently exposing YUV textures as RGB(A)
>> ones to OpenGL ES shaders, and this is not possible with this meta,
>> since you cannot specify two caps (the input format - typically some
>> YUV format - and the format that the glimagesink textures - some RGB
>> format).
> That's something you can negotiate. This is the same thing with VAAPI
> btw. Use a caps feature "meta:GLUpload" and if you have this feature,
> set format=RGBA. Same as gtreamer-vaapi here.
>
> Nicolas
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Matthew Waters]

Tee enforces that all branches support the exact same caps so it is
impossible to have a tee output both gluploadmeta buffers and sysmem
buffers (in terms of caps). However, the uploadmeta can still be placed
on the buffers when negotiated as sysmem but the format will be
incorrect and need converting which is the bug/discussion that was
linked before. In that case, It would be simpler to just not add the
upload meta and go through sysmem.

The other alternative is to add code to glupload (as was mentioned
before and would be much preferred), or create an Vivante specific
UploadMethod and at runtime, add it to the a list of possible uploaders
when the application/plugins are loaded.

On 06/09/15 17:50, Carlos Rafael Giani wrote:
> So, if I get this right, then the decoder would check if downstream
> supports this caps feature. If so, it sets the caps to format=RGBA and
> feature=gluploadmeta. Internally, the buffer holds I420 data, but the
> Vivante direct texture based upload code will then transparently
> convert to RGBA. So far so good.
>
> But what if there is a branch downstream, for example because of a tee
> element, and one element supports this caps feature, the other doesn't?
>
> Am 2015-09-04 um 17:08 schrieb Nicolas Dufresne:
>> Le vendredi 04 septembre 2015 à 16:02 +0200, Carlos Rafael Giani a
>> écrit :
>>> Well the main problem was that the Vivante stuff can implicitely do
>>> color space conversion, transparently exposing YUV textures as RGB(A)
>>> ones to OpenGL ES shaders, and this is not possible with this meta,
>>> since you cannot specify two caps (the input format - typically some
>>> YUV format - and the format that the glimagesink textures - some RGB
>>> format).
>> That's something you can negotiate. This is the same thing with VAAPI
>> btw. Use a caps feature "meta:GLUpload" and if you have this feature,
>> set format=RGBA. Same as gtreamer-vaapi here.
>>
>> Nicolas
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-***@lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>

[Nicolas Dufresne]

Le dimanche 06 septembre 2015 à 09:50 +0200, Carlos Rafael Giani a
écrit :
> But what if there is a branch downstream, for example because of a
> tee element, and one element supports this caps feature, the other
> doesn't?

"tee" element makes the allocation query fail, so that will never be
used (we always double check the caps features against the allocation
for backward compatibility). If tee finally had code to intersect the
allocation query, the upload meta would only be negotiated if all the
branches supports it (it's an intersection). If you add a branch
without the negotiated feature at run-time (and this was supported at
all) it would fail with not-negotiated. We could think of a
renegotiation path eventually though (leave the new requested pad idle
until upstream dare to renegotiated).

[Carlos Rafael Giani]

Would it be possible for you to use imxeglvivsink instead of glimagesink?

Am 2015-09-03 um 17:15 schrieb Eric-BTG:
> I did fakesink experiment. I can't replace glimagesink with fakesink as is
> because my app needs a real sink to map to displayed texture. So I created a
> sink displaying a viteotest while I decode stream to give to fakesink.
>
> I use custom app, and my pipeline is:
> "filesrc location=/sdcard/Movies/trailer_1080p.mov ! qtdemux ! queue !
> h264parse ! imxvpudec ! queue ! fakesink sync=true videotestsrc !
> videoconvert ! autovideosink"
>
> And as you expected no more frames are dropped.. Log is
> imxvpudec-fakesink.log
> <http://gstreamer-devel.966125.n4.nabble.com/file/n4673465/imxvpudec-fakesink.log>
>
>
> Do you have a idea to cope with CPU copies? Or it means that I have to patch
> glimagesink to make GPU based copies?
>
> Eric
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673465.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eric-BTG]

Hi,

Interresting stuff !! :) Thanks for your interrest.

I did explore eglvivsink possibility but facing problems when linking it to
android egl vivante. I can't use it as it because I have no framebuffer on
my android platform.

I get EGL/eglvivante.h header from
https://github.com/rogeriorps/gpu-samples-mx6/blob/master/lesson05/include/EGL/eglvivante.h

I'm able to build libgstimxeglvivsink.a with it setting egl platform to fb
but can't link it further.

Fb and gl symbols are missing:

../src/eglvivsink/gles2_renderer.c:725: error: undefined reference to
'glTexDirectVIVMap'
../src/eglvivsink/gles2_renderer.c:740: error: undefined reference to
'glTexDirectVIV'
../src/eglvivsink/gles2_renderer.c:772: error: undefined reference to
'glTexDirectInvalidateVIV'
../src/eglvivsink/egl_platform_fb.c:67: error: undefined reference to
'fbGetDisplayByIndex'
../src/eglvivsink/egl_platform_fb.c:148: error: undefined reference to
'fbCreateWindow'
../src/eglvivsink/egl_platform_fb.c:150: error: undefined reference to
'fbGetWindowGeometry'
../src/eglvivsink/egl_platform_fb.c:188: error: undefined reference to
'fbDestroyWindow'

I will have to link against freescale .so android egl lib. Writing fb
functions to create/destroy/get geometry of android native window does not
seems to be a big issue.

I think this is a good canditate to solve my CPU copies issue. Its looks
easyer than implementing zero copy in glimagesink.

I will look deeper to use imxeglvivsink and write android missing glue.

Thanks!

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673484.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Bing Song]

You can refer Freescale latest Yocto release. It already implemented vpudec ! glimagesink without video frame buffer copy. Their share video buffer with absolute physical address. Will share video buffer with DMA FD in the future.

Why need replace Freescale media player with GStreamer player on Android? Do you have any special requirement?

Regards,
Song Bing.

________________________________________
From: gstreamer-devel <gstreamer-devel-***@lists.freedesktop.org> on behalf of Eric-BTG <***@bt-ground.com>
Sent: Friday, September 4, 2015 10:44 PM
To: gstreamer-***@lists.freedesktop.org
Subject: Re: iMX6 hardware decode issue on Android

Hi,

Interresting stuff !! :) Thanks for your interrest.

I did explore eglvivsink possibility but facing problems when linking it to
android egl vivante. I can't use it as it because I have no framebuffer on
my android platform.

I get EGL/eglvivante.h header from
https://github.com/rogeriorps/gpu-samples-mx6/blob/master/lesson05/include/EGL/eglvivante.h

I'm able to build libgstimxeglvivsink.a with it setting egl platform to fb
but can't link it further.

Fb and gl symbols are missing:

../src/eglvivsink/gles2_renderer.c:725: error: undefined reference to
'glTexDirectVIVMap'
../src/eglvivsink/gles2_renderer.c:740: error: undefined reference to
'glTexDirectVIV'
../src/eglvivsink/gles2_renderer.c:772: error: undefined reference to
'glTexDirectInvalidateVIV'
../src/eglvivsink/egl_platform_fb.c:67: error: undefined reference to
'fbGetDisplayByIndex'
../src/eglvivsink/egl_platform_fb.c:148: error: undefined reference to
'fbCreateWindow'
../src/eglvivsink/egl_platform_fb.c:150: error: undefined reference to
'fbGetWindowGeometry'
../src/eglvivsink/egl_platform_fb.c:188: error: undefined reference to
'fbDestroyWindow'

I will have to link against freescale .so android egl lib. Writing fb
functions to create/destroy/get geometry of android native window does not
seems to be a big issue.

I think this is a good canditate to solve my CPU copies issue. Its looks
easyer than implementing zero copy in glimagesink.

I will look deeper to use imxeglvivsink and write android missing glue.

Thanks!

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673484.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Carlos Rafael Giani]

Is this release 4.0.6 of gst1.0-fsl-plugins? I did not see anything like
this in there. Do you use the glupload method nicolas suggested?

Am 2015-09-05 um 04:44 schrieb Bing Song:
> You can refer Freescale latest Yocto release. It already implemented vpudec ! glimagesink without video frame buffer copy. Their share video buffer with absolute physical address. Will share video buffer with DMA FD in the future.
>
> Why need replace Freescale media player with GStreamer player on Android? Do you have any special requirement?
>
> Regards,
> Song Bing.
>
> ________________________________________
> From: gstreamer-devel <gstreamer-devel-***@lists.freedesktop.org> on behalf of Eric-BTG <***@bt-ground.com>
> Sent: Friday, September 4, 2015 10:44 PM
> To: gstreamer-***@lists.freedesktop.org
> Subject: Re: iMX6 hardware decode issue on Android
>
> Hi,
>
> Interresting stuff !! :) Thanks for your interrest.
>
> I did explore eglvivsink possibility but facing problems when linking it to
> android egl vivante. I can't use it as it because I have no framebuffer on
> my android platform.
>
> I get EGL/eglvivante.h header from
> https://github.com/rogeriorps/gpu-samples-mx6/blob/master/lesson05/include/EGL/eglvivante.h
>
> I'm able to build libgstimxeglvivsink.a with it setting egl platform to fb
> but can't link it further.
>
> Fb and gl symbols are missing:
>
> ../src/eglvivsink/gles2_renderer.c:725: error: undefined reference to
> 'glTexDirectVIVMap'
> ../src/eglvivsink/gles2_renderer.c:740: error: undefined reference to
> 'glTexDirectVIV'
> ../src/eglvivsink/gles2_renderer.c:772: error: undefined reference to
> 'glTexDirectInvalidateVIV'
> ../src/eglvivsink/egl_platform_fb.c:67: error: undefined reference to
> 'fbGetDisplayByIndex'
> ../src/eglvivsink/egl_platform_fb.c:148: error: undefined reference to
> 'fbCreateWindow'
> ../src/eglvivsink/egl_platform_fb.c:150: error: undefined reference to
> 'fbGetWindowGeometry'
> ../src/eglvivsink/egl_platform_fb.c:188: error: undefined reference to
> 'fbDestroyWindow'
>
> I will have to link against freescale .so android egl lib. Writing fb
> functions to create/destroy/get geometry of android native window does not
> seems to be a big issue.
>
> I think this is a good canditate to solve my CPU copies issue. Its looks
> easyer than implementing zero copy in glimagesink.
>
> I will look deeper to use imxeglvivsink and write android missing glue.
>
> Thanks!
>
> Eric
>
>
>
> --
> View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673484.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Bing Song]

gst1.0-fsl-plugins only included vpudec. Freescale has special patch for glimagesink to directly upload texture. The patch applied in gstreamer1.0-plugins-bad in Yocto.

Regards,
Song Bing.

-----Original Message-----
From: gstreamer-devel [mailto:gstreamer-devel-***@lists.freedesktop.org] On Behalf Of Carlos Rafael Giani
Sent: Saturday, September 05, 2015 4:49 PM
To: gstreamer-***@lists.freedesktop.org
Subject: Re: iMX6 hardware decode issue on Android

Is this release 4.0.6 of gst1.0-fsl-plugins? I did not see anything like this in there. Do you use the glupload method nicolas suggested?

Am 2015-09-05 um 04:44 schrieb Bing Song:
> You can refer Freescale latest Yocto release. It already implemented vpudec ! glimagesink without video frame buffer copy. Their share video buffer with absolute physical address. Will share video buffer with DMA FD in the future.
>
> Why need replace Freescale media player with GStreamer player on Android? Do you have any special requirement?
>
> Regards,
> Song Bing.
>
> ________________________________________
> From: gstreamer-devel <gstreamer-devel-***@lists.freedesktop.org>
> on behalf of Eric-BTG <***@bt-ground.com>
> Sent: Friday, September 4, 2015 10:44 PM
> To: gstreamer-***@lists.freedesktop.org
> Subject: Re: iMX6 hardware decode issue on Android
>
> Hi,
>
> Interresting stuff !! :) Thanks for your interrest.
>
> I did explore eglvivsink possibility but facing problems when linking
> it to android egl vivante. I can't use it as it because I have no
> framebuffer on my android platform.
>
> I get EGL/eglvivante.h header from
> https://github.com/rogeriorps/gpu-samples-mx6/blob/master/lesson05/inc
> lude/EGL/eglvivante.h
>
> I'm able to build libgstimxeglvivsink.a with it setting egl platform
> to fb but can't link it further.
>
> Fb and gl symbols are missing:
>
> ../src/eglvivsink/gles2_renderer.c:725: error: undefined reference to
> 'glTexDirectVIVMap'
> ../src/eglvivsink/gles2_renderer.c:740: error: undefined reference to
> 'glTexDirectVIV'
> ../src/eglvivsink/gles2_renderer.c:772: error: undefined reference to
> 'glTexDirectInvalidateVIV'
> ../src/eglvivsink/egl_platform_fb.c:67: error: undefined reference to
> 'fbGetDisplayByIndex'
> ../src/eglvivsink/egl_platform_fb.c:148: error: undefined reference to
> 'fbCreateWindow'
> ../src/eglvivsink/egl_platform_fb.c:150: error: undefined reference to
> 'fbGetWindowGeometry'
> ../src/eglvivsink/egl_platform_fb.c:188: error: undefined reference to
> 'fbDestroyWindow'
>
> I will have to link against freescale .so android egl lib. Writing fb
> functions to create/destroy/get geometry of android native window does
> not seems to be a big issue.
>
> I think this is a good canditate to solve my CPU copies issue. Its
> looks easyer than implementing zero copy in glimagesink.
>
> I will look deeper to use imxeglvivsink and write android missing glue.
>
> Thanks!
>
> Eric
>
>
>
> --
> View this message in context:
> http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue
> -on-Android-tp4673449p4673484.html
> Sent from the GStreamer-devel mailing list archive at Nabble.com.
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-***@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

_______________________________________________
gstreamer-devel mailing list
gstreamer-***@lists.freedesktop.org
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Eric-BTG]

Hi,

Here are my imxeglvivsink experiments.

Since I have no framebuffer I have to stub 4 eglvivsink/egl_platform_fb.c
functions: fbGetDisplayByIndex, fbCreateWindow, fbGetWindowGeometry and
fbDestroyWindow to build imxeglvivsink for android.

Here is my stub patch (test purpose, hardcoded values):

On top of 898e51dbdb01926d6423d0d31a9530ec6deb5192 (Version 0.10.1) (can't
build further)


I have to use injected window_handle instead of creating it throught
framebuffer. Carlos why did you used fbCreateWindow to create the window
instead of using given window_handle?

My configure output:


I can use the sink with videotestsrc:
"videotestsrc ! imxeglvivsink" works as expected.

But I can't link imxvpudec to imxeglvivsink.

"filesrc location=/sdcard/Movies/trailer_1080p.mov ! qtdemux ! h264parse !
imxvpudec ! imxeglvivsink"

Outputs:
09-08 14:21:54.529 2888-2952/com.test.test E/GStreamer+GST_PIPELINE﹕
0:00:06.859754001 0x53530d80 ./grammar.y:617:gst_parse_perform_link could
not link imxvpudec0 to imxeglvivsink0
09-08 14:21:54.529 2888-2952/com.test.test E/GStreamer+default﹕
0:00:06.860123334 0x53530d80
src/main/jni/GStreamerPipeline.cpp:49:createGstPipeline Failed to build
gstreamer pipeline: could not link imxvpudec0 to imxeglvivsink0


So, I tried to use color space conversion from IPU with
imxipuvideotransform:

"filesrc location=/sdcard/Movies/trailer_1080p.mov ! qtdemux ! h264parse !
imxvpudec ! imxipuvideotransform ! imxeglvivsink"

I have a lot of IPU blitter errors:
E/GStreamer+imxipublitter﹕ 0:00:20.097881669 0x5352eb20
../src/ipu/blitter.c:482:gst_imx_ipu_blitter_blit_frame:<imxipublitter0>
queuing IPU task failed: Invalid argument
E/GStreamer+imxipublitter﹕ 0:00:20.104503001 0x5352eb20
../src/ipu/blitter.c:504:gst_imx_ipu_blitter_blit_frame:<imxipublitter0>
queuing IPU task failed: Invalid argument

Full log: ipu_blitter-full-log.log
<http://gstreamer-devel.966125.n4.nabble.com/file/n4673535/ipu_blitter-full-log.log>

As there was a lot work done on ipu blitter since 0.10.1 I will try to build
0.11.1 and let you know.


Regards,

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673535.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Eric-BTG]

Hi! Some good news.

Understood why I was not able to link imxvpudec to imxeglvivsink: GL_VIV_XXX
viv textures was not defined in my GLES2/gl2.h / gl2ext.h since it comes
from Android's NDK.

I took it from
https://github.com/Freescale/chromium-imx/blob/master/src/content/common/gpu/media/imx_gl_viv_direct_texture.h

I built 0.11.1 (b94ea3c014 + PR#57) and managed blitter and compositor
dependencies with .la files.


I can now decode and display 1080p frames but still have lags (few). I don't
have any dropped frames.

Here is a trace of egl_platform_fb eglSwapBuffers function execution:


eglSwapBuffers takes 32.5ms average.

My thought is that there is a colorspace conversion between vpudec output
and native window.
ANativeWindow_getFormat returns 1:WINDOW_FORMAT_RGBA_8888. For now I'm
trying to set my native window in YUV format.

Regards,

Eric



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/iMX6-hardware-decode-issue-on-Android-tp4673449p4673568.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.