Gstreamer with h264 annex b

[sivan]

How can make the gstreamer on linux to stream raw annex b h264 stream?



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-with-h264-annex-b-tp4670991.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Sebastian Dröge]

Post by sivan
How can make the gstreamer on linux to stream raw annex b h264 stream?

That would be video/x-h264,stream-format=byte-stream. How do you want to
stream it?

[sivan]

Here is my command to gstreamer :

gst-launch-1.0 -v --gst-debug-level=3 v4l2src device=/dev/video0 !
'video/x-raw,width=640,height=480' ! x264enc byte-stream=true !
video/x-h264,stream-format=byte-stream,alignment=au, profile=baseline !
rtph264pay ! udpsink host=192.168.1.170 port=4320 sync=0

When I open it with wireshark, I see the format is different then annex b:

I don't see the 00 00 00 01 sequence that suppose to be in the beginning of
each frame



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-with-h264-annex-b-tp4670991p4671003.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Sebastian Dröge]

Post by sivan
gst-launch-1.0 -v --gst-debug-level=3 v4l2src device=/dev/video0 !
'video/x-raw,width=640,height=480' ! x264enc byte-stream=true !
video/x-h264,stream-format=byte-stream,alignment=au, profile=baseline !
rtph264pay ! udpsink host=192.168.1.170 port=4320 sync=0
I don't see the 00 00 00 01 sequence that suppose to be in the beginning of
each frame

Check the beginning of the dump here:
gst-launch-1.0 videotestsrc ! x264enc ! "video/x-h264,stream-format=byte-stream" ! fakesink dump=true num-buffers=3

It contains the 00 00 00 01 sequence at the beginning of each buffer.

If you put it into RTP, it will be in the RTP mapping which will not
have that sequence at the beginning of every RTP packet... and
rtph264depay can output stream-format=byte-stream or avc, where the
latter would also not have that sequence.

[sivan]

I do use RTP

If I understand you, you mean that using RTP changes the header of the NALU
?

How would I know where a frame begins?



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-with-h264-annex-b-tp4670991p4671006.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Sebastian Dröge]

Post by sivan
I do use RTP
If I understand you, you mean that using RTP changes the header of the NALU
?
How would I know where a frame begins?

RTP adds its own header in the beginning. Read
https://tools.ietf.org/html/rfc6184 for more details about how H264 is
put into RTP packets.

[sivan]

anyway I guess I just need to use the sdk for iOS to extract h264 frames

It seems better solution then to try to write it on my own now :)



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-with-h264-annex-b-tp4670991p4671008.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[sivan]

The bigger picture is:

I have e medical device with camera. The device is running linux and we use
gstreamer to stream the video in h264.

I have iOS app which receives the stream, decodes it (with hardware
acceleration in iOS 8) and pass the RGB frames to video conference sdk.

I understand that streamer 1.5 sdk will handle the hardware decoding too?

How can I use the current gstreamer to get the h264 frames and I also need
the spa and ops from the stream




--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-with-h264-annex-b-tp4670991p4671009.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[sivan]

I mean the sps and pps :)



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-with-h264-annex-b-tp4670991p4671011.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Michał Wróbel]

Post by sivan
I don't see the 00 00 00 01 sequence that suppose to be in the beginning of
each frame

That's perfectly OK.

Actually, RFC6184 specifies that 00 00 00 01 sequence inside an RTP
packet would be an error:

"This payload specification can only be used to carry the "naked" H.264
NAL unit stream over RTP and not the bitstream format discussed in Annex
B of H.264."

Please, don't violate this rule. Even if many H.264 stream receivers
seem to parse such packets without any error or warning.

Post by sivan
How can I use the current gstreamer to get the h264 frames and I also need the sps and pps from the stream

According to RFC6184, it is acceptable to have SPS and PPS inside an RTP
packet stream. It is called "in-band signalling of parameter sets".
However, it is not recommended when an unreliable medium is used:

"If in-band signaling of parameter sets is used, the sender SHOULD take
the error characteristics into account and use mechanisms to provide a
high probability for delivering the parameter sets correctly.
Mechanisms that increase the probability for a correct reception include
packet repetition, FEC, and retransmission. The use of an unreliable,
out-of-band control protocol has similar disadvantages as the in-band
signaling (possible loss) and, in addition, may also lead to
difficulties in the synchronization (see below). Therefore, it is NOT
RECOMMENDED."

It may be done to work reliably, but requires some special attention.
Many receivers/decoders will wait for SPS/PPS before attempting to
decode the video. Some do not - they try their best to decode, most of
the time without success, which results in a glitch when starting to
receive the stream.

An typical out-of-the-band signaling of parameter sets is to provide
description of the session via SDP (in a file or on-line via RTSP -
check out gst-rtsp-server).


Having that said... if you still want to just stream RTP with in-band
signalling, here's a gst-launch oneliner for x264enc:

gst-launch-1.0 videotestsrc ! 'video/x-raw, framerate=30/1, width=640,
height=480' ! queue ! x264enc tune=4 ! rtph264pay ! udpsink
host=224.0.0.1 port=5000

However, here it is x264enc which inserts SPS/PPS into the stream at
each keyframe.


If you have a stream without SPS/PPS (from some other encoder), use
config-interval property of rtph264pay.

Best regards,
Michał Wróbel
--
Michał Wróbel
Software developer

***@flytronic.pl
mobile phone: [phone]

Flytronic sp. z o.o.
UAV R&D Center
www.flytronic.pl
tel. [phone]
fax [phone]
ul. Bojkowska 43, 44-100 Gliwice
NIP:969 151 39 93, REGON: 240851769, KRS 0000298330, Sąd Rejonowy w Gliwicach, X Wydział Gospodarczy Krajowego Rejestru Sądowego

[Chuck Crisler]

There are situations where in-band signaling is absolutely needed and
required for successful video operations. Specifically, if you have video
originating with a mobile device that has different sizes for different
orientations, then when the device is rotated 90 degrees, you need to send
the SPS/PPS pair to help correct the final display. Also, the application
that I have developed has 1 system that receives video from a network
source and then propagates it to other systems that join at different
times. If there is only a SPS/PPS at the start, later joining endpoints
won't ever succeed in decoding video. I have extended the rtph264depay
element to also have a config-interval property, which solved that problem
because there are IP cameras (Axis) that don't send the in-band signaling.

Though UDP is defined as 'unreliable', repeating the SPS/PPS every 2-5
seconds doesn't have any real impact of bandwidth usage and if you are
losing enough packets that they never get through, the lack of SPS/PPS
packets isn't your real problem.

Post by Michał Wróbel

Post by sivan
When I open it with wireshark, I see the format is different then annex
I don't see the 00 00 00 01 sequence that suppose to be in the beginning

of

Post by sivan
each frame

That's perfectly OK.
Actually, RFC6184 specifies that 00 00 00 01 sequence inside an RTP
"This payload specification can only be used to carry the "naked" H.264
NAL unit stream over RTP and not the bitstream format discussed in Annex
B of H.264."
Please, don't violate this rule. Even if many H.264 stream receivers
seem to parse such packets without any error or warning.

Post by sivan
How can I use the current gstreamer to get the h264 frames and I also

need the sps and pps from the stream
According to RFC6184, it is acceptable to have SPS and PPS inside an RTP
packet stream. It is called "in-band signalling of parameter sets".
"If in-band signaling of parameter sets is used, the sender SHOULD take
the error characteristics into account and use mechanisms to provide a
high probability for delivering the parameter sets correctly.
Mechanisms that increase the probability for a correct reception include
packet repetition, FEC, and retransmission. The use of an unreliable,
out-of-band control protocol has similar disadvantages as the in-band
signaling (possible loss) and, in addition, may also lead to
difficulties in the synchronization (see below). Therefore, it is NOT
RECOMMENDED."
It may be done to work reliably, but requires some special attention.
Many receivers/decoders will wait for SPS/PPS before attempting to
decode the video. Some do not - they try their best to decode, most of
the time without success, which results in a glitch when starting to
receive the stream.
An typical out-of-the-band signaling of parameter sets is to provide
description of the session via SDP (in a file or on-line via RTSP -
check out gst-rtsp-server).
Having that said... if you still want to just stream RTP with in-band
gst-launch-1.0 videotestsrc ! 'video/x-raw, framerate=30/1, width=640,
height=480' ! queue ! x264enc tune=4 ! rtph264pay ! udpsink
host=224.0.0.1 port=5000
However, here it is x264enc which inserts SPS/PPS into the stream at
each keyframe.
If you have a stream without SPS/PPS (from some other encoder), use
config-interval property of rtph264pay.
Best regards,
Michał Wróbel
--
Michał Wróbel
Software developer
mobile phone: [phone]
Flytronic sp. z o.o.
UAV R&D Center
www.flytronic.pl
tel. [phone]
fax [phone]
ul. Bojkowska 43, 44-100 Gliwice
NIP:969 151 39 93, REGON: 240851769, KRS 0000298330, Sąd Rejonowy w
Gliwicach, X Wydział Gospodarczy Krajowego Rejestru Sądowego
--
Niniejsza wiadomość jest przeznaczona tylko dla jej adresata i nie moÅŒe
być ujawniona osobom trzecim. Jeśli nie jest Pan/i zamierzonym adresatem
niniejszej wiadomości bądź osobą przez niego upowaÅŒnioną do jej odbioru lub
przekazania adresatowi, informujemy, ÅŒe jej rozpowszechnianie, dystrybucja,
kopiowanie lub inne działanie o podobnym charakterze jest zabronione.
JeÅŒeli wiadomość dotarła do Pan/i omyłkowo, prosimy o poinformowanie
nadawcy oraz usunięcie wiadomości z poczty bez otwierania załączników.
Dziękujemy.
This e-mail is intended solely for the addressee(s) and must not be
disclosed to third parties. If you are not the addressee of this e-mail or
has not been authorized by the addressee to receive or forward this
message, we inform that its disclosure, distribution, copying or any other
similar action is unlawful. If you have received this e-mail in error,
please notify the sender immediately by return e-mail. Please then delete
the e-mail from your mail box without opening attachments. Thank you.
--
_______________________________________________
gstreamer-devel mailing list
http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[sivan]

First of all thanks for the help and knowledge sharing here

Really appreciate it !!!

I'll describe the big picture:

I have a medical device with tiny video camera. The device is running linux
and gstreamer to stream the video.

The gstreamer on the device is configured to stream RTP h264

I have iOS app which receives the stream, and I am decoding the frames with
the VideoToolBox framework for better performance through hardware
acceleration.

What would be the best solution for achieving this ?

It seems that gstreamer sdk for iOS 1.5 will do it all for me. isn't it so?



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-with-h264-annex-b-tp4670991p4671044.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Michał Wróbel]

Post by sivan
I have iOS app which receives the stream, and I am decoding the frames with
the VideoToolBox framework for better performance through hardware
acceleration.
What would be the best solution for achieving this ?
It seems that gstreamer sdk for iOS 1.5 will do it all for me. isn't it so?

Did you read Sebastian's post about this?

https://coaxion.net/blog/2014/09/gstreamer-with-hardware-video-codecs-on-ios/

There's some discussion in the comments as well.

It seems that if you wanted to have it now, you would need to compile
gstreamer from git master as 1.5 is not yet released.

[sivan]

I wrote the code that uses the VideoToolBox to decode h264 frames

My problem is that with RTP I can receive sliced frames in different
packets, so now I need to add code that handles that option - of sliced
frames received in different packets

I suppose that the gstreamer 1.5 for iOS handles that. is that so ?



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-with-h264-annex-b-tp4670991p4671051.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Sebastian Dröge]

Post by sivan
I wrote the code that uses the VideoToolBox to decode h264 frames
My problem is that with RTP I can receive sliced frames in different
packets, so now I need to add code that handles that option - of sliced
frames received in different packets
I suppose that the gstreamer 1.5 for iOS handles that. is that so ?

Take a look at the code:
http://cgit.freedesktop.org/gstreamer/gst-plugins-bad/tree/sys/applemedia/vtdec.c

The decoder requires AVC stream-format (so not byte-stream), and AU
alignment (i.e. one frame per buffer). Other stream formats and
alignments are converted to this one by rtph264depay and h264parse if
required.

[sivan]

I am working with gstreamer for linux on one side (the device), and working
with gstreamer for iOS on the client side

I need on the iOS side the RGB frames

Why is it complicated ?

I guess it would take only few lines of code on the iOS side to make it
happen right?

Is there any example like this out there ?



--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Gstreamer-with-h264-annex-b-tp4670991p4671056.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.