Debugging video jerking problem

[Wolfgang Grandegger]

Hello,

I'm currently debugging a jerking problem. When I move from 50 to 60
FPS, the displayed video starts jerking. Just a little bit but it's
clearly visible when the camera pans. Here is my pipeline:

gst-launch-1.0 -v \
udpsrc port=50004 buffer-size=180000000 do-timestamp=1 \
caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, \
encoding-name=(string)JPEG, payload=(int)26, framerate=(fraction)60/1" \
! rtpjitterbuffer latency=20 \
! rtpjpegdepay \
! vaapijpegdec \
! timeoverlay text="A Text ..."
! tee name=t
t. ! queue ! vaapisink
t. ! queue ! vaapih264enc ! mp4mux ! filesink location=/tmp/test.mp4

I have connected a signal for under- and overruns to both, the display and
record queue. What I see when the jerking starts is that I get underruns
on the display (vaapisink) queue. Does that mean that the packets do not
arrive in time due to pre-processing time? The jerking goes away if I remove
the tee for recording or run the video at just 50 FPS.
Any ideas what could cause the jerking and how it can be debugged or even
avoided?

Any help is appreciated.

Thanks,

Wolfgang.

[Nicolas Dufresne]

Post by Wolfgang Grandegger
Hello,
I'm currently debugging a jerking problem. When I move from 50 to 60
FPS, the displayed video starts jerking. Just a little bit but it's
gst-launch-1.0 -v \
udpsrc port=50004 buffer-size=180000000 do-timestamp=1 \
caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, \
encoding-name=(string)JPEG, payload=(int)26, framerate=(fraction)60/1" \
! rtpjitterbuffer latency=20 \
! rtpjpegdepay \
! vaapijpegdec \
! timeoverlay text="A Text ..."
! tee name=t
t. ! queue ! vaapisink
t. ! queue ! vaapih264enc ! mp4mux ! filesink location=/tmp/test.mp4
I have connected a signal for under- and overruns to both, the display and
record queue. What I see when the jerking starts is that I get underruns
on the display (vaapisink) queue. Does that mean that the packets do not
arrive in time due to pre-processing time? The jerking goes away if I remove
the tee for recording or run the video at just 50 FPS.
Any ideas what could cause the jerking and how it can be debugged or even
avoided?

My guess would be that when going for 60 to 50 FPS you increase the
latency. This is because H264 decoding latency is in number of frames.
So the longer the duration of a frame, the longer he latency. Ideally,
you would through your RTP signalling make the caps changes when the
FPS changes so that latency can be recalculated. You can also use
pipeline latency, or processing-latency on sinks.

Post by Wolfgang Grandegger
Any help is appreciated.
Thanks,
Wolfgang.
_______________________________________________
gstreamer-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Wolfgang Grandegger]

Hello,

Post by Nicolas Dufresne

Post by Wolfgang Grandegger
Hello,
I'm currently debugging a jerking problem. When I move from 50 to 60
FPS, the displayed video starts jerking. Just a little bit but it's
gst-launch-1.0 -v \
udpsrc port=50004 buffer-size=180000000 do-timestamp=1 \
caps="application/x-rtp, media=(string)video, clock-rate=(int)90000, \
encoding-name=(string)JPEG, payload=(int)26, framerate=(fraction)60/1" \
! rtpjitterbuffer latency=20 \
! rtpjpegdepay \
! vaapijpegdec \
! timeoverlay text="A Text ..."
! tee name=t
t. ! queue ! vaapisink
t. ! queue ! vaapih264enc ! mp4mux ! filesink location=/tmp/test.mp4
I have connected a signal for under- and overruns to both, the display and
record queue. What I see when the jerking starts is that I get underruns
on the display (vaapisink) queue. Does that mean that the packets do not
arrive in time due to pre-processing time? The jerking goes away if I remove
the tee for recording or run the video at just 50 FPS.
Any ideas what could cause the jerking and how it can be debugged or even
avoided?

My guess would be that when going for 60 to 50 FPS you increase the
latency. This is because H264 decoding latency is in number of frames.

Not sure if I understand you correctly. I do not change FPS on the fly.
I convert my MJPEG video from 50 to 60 FPS on the remote side with

$ ffmpeg ... -filter:v "setpts=0.83333333*PTS" -r 60 ...

and then send it to the network. The video with 50 FPS is recorded
and displayed fine. The one with 60 FPS causes jerking on the display.
The jerking is also visible on the recorded video... hm, can I assume
that the "rtpjitterbuffer" calculated jerky timestamps then?

And am using H264 encoding to record the video, not decoding.

Post by Nicolas Dufresne
So the longer the duration of a frame, the longer he latency. Ideally,
you would through your RTP signalling make the caps changes when the
FPS changes so that latency can be recalculated. You can also use
pipeline latency, or processing-latency on sinks.

OK, will look into that.

Wolfgang.