Asynchrounous Audio Sample Rate Conversion with gstreamer

[Maik Scholz]

Hi,

in some use cases (broadcast digital radio or IP radio), the transmitter
is sending an audio stream with a nominal sampling rate (e.g. 48kHz).
When a receiver, expecting 48KHz, wants to play these samples on a local
sound sink, then there is always the problem,
that the transmitter clock slightly differs from the receiver clock.
For that issue, I need an asynchronous sampling rate converter!

Is this possible the available gstreamer plugins?
Do audioresample support asynchronous clocks?

_My example:_
My Transmitter (sending samples with *(48-1)*kHz):
gst-launch-0.10 audiotestsrc freq=300 ! audio/x-raw-int,
endianness="(int)1234", signed="(boolean)true", width="(int)16",
depth="(int)16", rate="(int)*47*000", channels="(int)1" ! tcpclientsink
host=localhost port=3000

My Receiver: (Expeting 48kHz)
gst-launch-0.10 tcpserversrc host=localhost port=3000 ! audio/x-raw-int,
endianness="(int)1234", signed="(boolean)true", width="(int)16",
depth="(int)16", rate="(int)48000", channels="(int)1" ! audioconvert !
audioresample ! audio/x-raw-int, endianness="(int)1234",
signed="(boolean)true", width="(int)16", depth="(int)16",
rate="(int)48000", channels="(int)1" ! autoaudiosink

Because the transmitter clock (47kHz) differs from the receiver clock
(48kHz), the sound drops over short time.

Thank you for any hint in advance.

Maik

[Sebastian Dröge]

Hi, 
in some use cases (broadcast digital radio or IP radio), the
transmitter is sending an audio stream with a nominal sampling rate
(e.g. 48kHz).
When a receiver, expecting 48KHz, wants to play these samples on a
local sound sink, then there is always the problem, that the
transmitter clock slightly differs from the receiver clock.
For that issue, I need an asynchronous sampling rate converter!
Is this possible the available gstreamer plugins?
Do audioresample support asynchronous clocks?

audioresample not, it's not really audioresample's job to do this kind
of work (but in your case you can use audioresample to resample the
audio as you know the ratio between sender and receiver rate).

This is something that can be implemented on various layers, e.g. with
RTP this would happen based on the RTP timestamps and arrival times
inside the rtpjitterbuffer. You don't have any timestamps inside the
stream so that won't do it.

In your case you would have to implement something similar somewhere
downstream of the tcpserversrc. Something that correlates the amount of
data received per time unit with the duration of that time unit and
then extrapolates the relative rate difference between the server and
receiver clock, and then accordingly timestamps the media, or directly
resamples the media (for which you can again use audioresample here).
One main problem in your case is that you use TCP, and due to all the
buffering that happens in TCP you will have it harder to calculate this
than with UDP. And as you don't have timestamps inside the stream, you
will have to assume that your media is continuous which might also not
be true.


And then (if you don't resample anywhere but just adjust timestamps),
it's the job of the audio sink in the end of make up for the difference
between timestamps and actual samples received. It will be default just
"skip" (skew) but you can configure it also to resample, this can be
done with the slave-method property on all audio sinks.
However the resampling that is currently implemented there is very
minimal and audioresample will give you better quality, but that's
something that should be fixed.


Does that answer your question? There's no standard solution for this
(as your streaming protocol is not and has many use case specific
assumptions) and you have to implement something yourself with the help
of existing GStreamer code.

[Sebastian Dröge]

Post by Maik Scholz
 
gst-launch-0.10 audiotestsrc freq=300 ! audio/x-raw-int,
endianness="(int)1234", signed="(boolean)true", width="(int)16",
depth="(int)16", rate="(int)47000", channels="(int)1" ! tcpclientsink
host=localhost port=3000
My Receiver: (Expeting 48kHz)
gst-launch-0.10 tcpserversrc host=localhost port=3000 ! audio/x-raw-
int, endianness="(int)1234", signed="(boolean)true", width="(int)16",
depth="(int)16", rate="(int)48000", channels="(int)1" ! audioconvert
! audioresample ! audio/x-raw-int, endianness="(int)1234",
signed="(boolean)true", width="(int)16", depth="(int)16",
rate="(int)48000", channels="(int)1" ! autoaudiosink

Also some notes regarding your specific pipelines. First of all, don't
use GStreamer 0.10 but use a recent 1.x version, e.g. 1.8.1. 0.10 is no
longer maintained since more than 3 years and you're unlikely to get
much support for it anywhere.

Then, you should make sure that downstream of tcpserversrc always a
multiple of full samples arrives. That is, in your case an even buffer
size. This is not guaranteed because TCP is a stream protocol.
The audioparse element can ensure that this is the case for you.

[Maik Scholz]

Hi,
thank you very much for the hints.
With your help, my pipeline is running quite well with ONE audio source.

My target is, having multiply audio sources with different sample rates and
different external clock sources.
That includes small sample rate mismatches (+- 50Hz) and drifts.

*My mixer pipeline prototype:*
GSTLAUNCH=gst-launch-1.0
STREAMDECHOST=127.0.0.1
PORT_SRC1=41001
PORT_SRC2=42001
PORT_SINK_DEBUG_1=45002

${GSTLAUNCH} \
audiomixer name=sink_audiomixer latency=256000000 ! \
tee name=sink_tee \
\
sink_tee. ! \
queue name=outtcp_queue ! \
tcpserversink name=tcpserversink host="0.0.0.0"
port=${PORT_SINK_DEBUG_1} \
\
sink_tee. ! \
queue name=outdsp_queue ! \
autoaudiosink name=autoaudiosink sync=true
slave-method=resample \
\
sink_tee. ! \
queue name=file_queue ! \
audioconvert name=fileconvert ! \
filesink name=file_filesink location=/dev/null \
\
udpsrc name=src1_src port=${PORT_SRC1} do-timestamp=true ! \
audio/x-raw,format=S16LE,channels=2,rate=32000 ! \
queue name=src1_queue min-threshold-time=0 ! \
audiorate name=src1_audiorate tolerance=256000000
skip-to-first=false ! \
audioconvert name=src1_audioconvert ! \
audioresample name=src1_audioresample quality=1 ! \
audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
tee name=src1_tee \
\
src1_tee. ! \
volume name=src1_volume volume=0.5 ! \
sink_audiomixer. \
\
udpsrc name=src2_src port=${PORT_SRC2} do-timestamp=true ! \
audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
queue name=src2_queue min-threshold-time=0 ! \
audiorate name=src2_audiorate tolerance=256000000
skip-to-first=false ! \
audioconvert name=src2_audioconvert ! \
audioresample name=src2_audioresample quality=4 ! \
audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
tee name=src2_tee \
\
src2_tee. ! \
volume name=src2_volume volume=0.5 ! \
sink_audiomixer.


*My SRC1 audio transmitter (32000 - 50Hz):*
gst-launch-1.0 audiotestsrc freq=100 !
audio/x-raw,format=S16LE,channels=2,rate=31950 ! udpsink host="127.0.0.1"
port=41001

*My SRC2 audio transmitter (48000 + 50Hz):*
gst-launch-1.0 audiotestsrc freq=300 !
audio/x-raw,format=S16LE,channels=2,rate=48050 ! udpsink host="127.0.0.1"
port=42001

When I run these pipelines with even the latest gst version, then I hear
"clicks" from time to time.
I expect, that the audiomixer has some problems with the differ sampling
frequencies.

My assumption is, I need a asynchronous SRC for each input
(src1_audioresample+src2_audioresample).
How can I implement this with available elements. Or if not available, what
is the best way to implement these?

Why is the audiomixer located in the bad plugins? What means bad?




--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Asynchrounous-Audio-Sample-Rate-Conversion-with-gstreamer-tp4677252p4677765.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.

[Carlos Rafael Giani]

In most cases, there is only one clock (-> audio clock = pipeline
clock), but in your case, the pipeline clock is the monotonic system
clock, so audiobasesink needs to compensate for the drift.
The clicks are most likely coming from the "skew" drift compensation
method in the audiobasesink. This method looks at the drift, and if a
threshold is reached, moves the playout pointer (= an offset withing a
ring buffer), effectively inserting nullsamples or discarding samples,
depending on the skew direction.

You can hook in your own drift compensation method if you set the
audiobasesink's "slave-method" property to "custom". We use that in a
project to finetune the audio clock's speed via a PLL by an integer
number of ppm. It would be the same with an ASRC.

Post by Maik Scholz
Hi,
thank you very much for the hints.
With your help, my pipeline is running quite well with ONE audio source.
My target is, having multiply audio sources with different sample rates and
different external clock sources.
That includes small sample rate mismatches (+- 50Hz) and drifts.
*My mixer pipeline prototype:*
GSTLAUNCH=gst-launch-1.0
STREAMDECHOST=127.0.0.1
PORT_SRC1=41001
PORT_SRC2=42001
PORT_SINK_DEBUG_1=45002
${GSTLAUNCH} \
audiomixer name=sink_audiomixer latency=256000000 ! \
tee name=sink_tee \
\
sink_tee. ! \
queue name=outtcp_queue ! \
tcpserversink name=tcpserversink host="0.0.0.0"
port=${PORT_SINK_DEBUG_1} \
\
sink_tee. ! \
queue name=outdsp_queue ! \
autoaudiosink name=autoaudiosink sync=true
slave-method=resample \
\
sink_tee. ! \
queue name=file_queue ! \
audioconvert name=fileconvert ! \
filesink name=file_filesink location=/dev/null \
\
udpsrc name=src1_src port=${PORT_SRC1} do-timestamp=true ! \
audio/x-raw,format=S16LE,channels=2,rate=32000 ! \
queue name=src1_queue min-threshold-time=0 ! \
audiorate name=src1_audiorate tolerance=256000000
skip-to-first=false ! \
audioconvert name=src1_audioconvert ! \
audioresample name=src1_audioresample quality=1 ! \
audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
tee name=src1_tee \
\
src1_tee. ! \
volume name=src1_volume volume=0.5 ! \
sink_audiomixer. \
\
udpsrc name=src2_src port=${PORT_SRC2} do-timestamp=true ! \
audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
queue name=src2_queue min-threshold-time=0 ! \
audiorate name=src2_audiorate tolerance=256000000
skip-to-first=false ! \
audioconvert name=src2_audioconvert ! \
audioresample name=src2_audioresample quality=4 ! \
audio/x-raw,format=S16LE,channels=2,rate=48000 ! \
tee name=src2_tee \
\
src2_tee. ! \
volume name=src2_volume volume=0.5 ! \
sink_audiomixer.
*My SRC1 audio transmitter (32000 - 50Hz):*
gst-launch-1.0 audiotestsrc freq=100 !
audio/x-raw,format=S16LE,channels=2,rate=31950 ! udpsink host="127.0.0.1"
port=41001
*My SRC2 audio transmitter (48000 + 50Hz):*
gst-launch-1.0 audiotestsrc freq=300 !
audio/x-raw,format=S16LE,channels=2,rate=48050 ! udpsink host="127.0.0.1"
port=42001
When I run these pipelines with even the latest gst version, then I hear
"clicks" from time to time.
I expect, that the audiomixer has some problems with the differ sampling
frequencies.
My assumption is, I need a asynchronous SRC for each input
(src1_audioresample+src2_audioresample).
How can I implement this with available elements. Or if not available, what
is the best way to implement these?
Why is the audiomixer located in the bad plugins? What means bad?
--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Asynchrounous-Audio-Sample-Rate-Conversion-with-gstreamer-tp4677252p4677765.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.
_______________________________________________
gstreamer-devel mailing list
https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

[Maik Scholz]

Hi,

My pipeline simplified:
SRC1: sine with 32kHz*+50Hz*=>
NETWORK=>
udpsrc(expected sample rate is 32kHz)=>
queue=>
audiorate=>
audioconvert=>
audioresample=>|

|audiomixer=>queue=>autoaudiosink(sync=true,slave-method=resample)=>Audio-Out
audioresample=>|
audioconvert=>
audiorate=>
queue=>
udpsrc(expected sample rate is 48kHz)=>
NETWORK=>
SRC2: sine with 48kHz*-50Hz*=>

Post by Carlos Rafael Giani
You can hook in your own drift compensation method if you set the
audiobasesink's "slave-method" property to "custom". We use that in a
project to finetune the audio clock's speed via a PLL by an integer
number of ppm. It would be the same with an ASRC.

Sorry, I have some concerns about this proposal.
How should this compensate the clock drifts between (Audio-Out) and
two different inputs (SRC1 and SRC2)?

My expectation is, that I need something in the SRC pipeline parts.
This compensates each SRC clock to the main pipeline clock.
The ASRC in the basesink/autoaudiosink is responsible to synchronize the
main pipeline clock to the sink clock.

Post by Carlos Rafael Giani
Unlike the adder element audiomixer properly synchronises all input

streams.
What does this mean in this context? Is there a ASRC inside the audiomixer?

An idea for would be adding a custom element in front of each
"audioresample" element.
This "ASRC-SampleRate-Checker", could observe the SRCx buffer timestamps and
calculate the real input sample rate.
By knowing this, the element could adapt/change the rate in the output caps
in a way,
that the following audioresample is compensating the source clock
difference.
Could this work (inaudible)?

If there is any other solution for this, I am happy for all proposals.

Maik











--
View this message in context: http://gstreamer-devel.966125.n4.nabble.com/Asynchrounous-Audio-Sample-Rate-Conversion-with-gstreamer-tp4677252p4677771.html
Sent from the GStreamer-devel mailing list archive at Nabble.com.