Elastic Buffer

[TheDaChicken]

Fixing #330

Use a circular buffer to store packets to be played later temporarily. This allows libnrsc5 to handle buffering properly by outputting at a fixed ratio. This allows missing packets to be silenced or handled properly.

Replace audio callback with a buffering system that involves a polling function instead with any buffer size + implement latency buffering.

New functions:
1. nrsc5_open_program
2. nrsc5_close_program
3. nrsc5_reset_program
4. nrsc5_read_program_blocking
5. nrsc5_read_program_nonblocking

Please let me know what you think. Code review as much as you want! :D

[FoxxMD]

Thanks for the putting in the hard work! If you need testers I'm happy to build this into my workflow and see how it performs

[argilo]

Thanks for the proposed fix. It looks like there is a problem with the Python wrapper, as it now crashes.

I'm not sure that such a drastic change is needed. An alternative approach would be to use a 64-frame circular buffer (as the NRSC-5 specification hints at), with silence inserted for any missing audio frames. When a batch of samples arrives through the rtlsdr callback, a corresponding batch of audio samples could immediately be returned from the circular buffer, and then decoding of the RF samples (which can take a variable amount of time) could proceed. I think this approach would result in a constant output rate, and allow the digital output to be properly synchronized with analog, without needing any change to the API.

[TheDaChicken]

Thanks for the proposed fix. It looks like there is a problem with the Python wrapper, as it now crashes.

That would make sense since I didn't test the Python wrapper. Probably due to me removing the AUDIO callback.

I'm not sure that such a drastic change is needed. An alternative approach would be to use a 64-frame circular buffer (as the NRSC-5 specification hints at), with silence inserted for any missing audio frames. When a batch of samples arrives through the rtlsdr callback, a corresponding batch of audio samples could immediately be returned from the circular buffer, and then decoding of the RF samples (which can take a variable amount of time) could proceed. I think this approach would result in a constant output rate, and allow the digital output to be properly synchronized with analog, without needing any change to the API.

I agree about the drastic change being pretty annoying. I tried to not add silence. I'll do more testing and update the PR with changes. I know it is possible to constantly send very SMALL amounts of audio packets. However, it was VERY on edge but that was without silence. You have more experience reading whitepapers. I read it so many times.

[argilo]

You have more experience reading whitepapers. I read it so many times.

The specification only describes how to build a transmitter, so one has to infer how to build a receiver.

The "PDU control word" is what hints at a 64-frame circular buffer, with the "Starting Sequence Number" field describing where to begin inserting frames into the buffer when decoding a PDU.

The "Audio Transport" section also describes how elastic buffering is supposed to work at a high level. The receiver is supposed to use an elastic buffer so that it outputs exactly 32 audio frames per PDU, even though the incoming PDUs can have anywhere from 24 to 40 audio frames each. This is the buffer that nrsc5 is currently missing, which causes our output audio rate to be variable.

[TheDaChicken]

The specification only describes how to build a transmitter, so one has to infer how to build a receiver.

The "PDU control word" is what hints at a 64-frame circular buffer, with the "Starting Sequence Number" field describing where to begin inserting frames into the buffer when decoding a PDU.

Yes. Should I assume packets are meant to be ordered per stream? If its not reset buffer? It would be nice to support the alignment of many streams. I don't have a recording of any enhanced streams to test alignment.

Should I start playback once it receives full PDU Sequence? The maximum would be 64 in the buffer. I think delaying more would cause the audio to be more delayed than a normal receiver. I am not sure about 1-3 codec with 4 packets maximum. Does calculating minimum and maximum latency matter?

The "Audio Transport" section also describes how elastic buffering is supposed to work at a high level. The receiver is supposed to use an elastic buffer so that it outputs exactly 32 audio frames per PDU, even though the incoming PDUs can have anywhere from 24 to 40 audio frames each. This is the buffer that nrsc5 is currently missing, which causes our output audio rate to be variable.

If I know correctly, PDU timing is also variable. I've had a PDU take 1.8 seconds before. Since you are willing for me to touch other areas for this, it looks like input_push would be great time to push audio packets through output.c when it's the best time to do so. That way, I don't need to create another thread like I was testing.

[argilo]

Yes. Should I assume packets are meant to be ordered per stream? If its not reset buffer? It would be nice to support the alignment of many streams.

The API streams out all audio programs at once, so we'd need a separate buffer for each one.

I don't have a recording of any enhanced streams to test alignment.

I've only seen enhanced streams used on AM. But nrsc5 doesn't currently decode them, so I think it's fine to ignore them for now. (Related: #245) If you need a recording, I can send you one.

Should I start playback once it receives full PDU Sequence?

A real receiver would output analog audio (for the HD1 channel, anyway). nrsc5 doesn't currently demodulate the analog, but for now it could output silence, to maintain a constant input / output ratio. The same will need to be done for audio frames that are rejected due to an invalid CRC. Slots in the circular buffer could simply default to silence.

I am not sure about 1-3 codec with 4 packets maximum.

Those are used on logical channels (e.g. P3) that have a "transfer frame modulus" of 8, i.e. 8 transfer frames per block. With an average of 4 audio frames per transfer frame, it still comes out to an average of 32 audio frames per block. The PDU control word may work a bit differently though; it's been a while since I looked at that.

it looks like input_push would be great time to push audio packets through output.c when it's the best time to do so.

Yeah, as long as it happens before the digital demodulation steps (which can take a variable amount of time) we should be good.

[argilo]

That way, I don't need to create another thread like I was testing.

Right, I do believe it should be possible to do this without adding a thread.

[TheDaChicken]

I don't have a recording of any enhanced streams to test alignment.

I've only seen enhanced streams used on AM. But nrsc5 doesn't currently decode them, so I think it's fine to ignore them for now. (Related: #245) If you need a recording, I can send you one.

The reason I wanted a recording is because I am not sure how much delayed the enhanced stream compared to the core stream. Figured out i don't need to worry about it.

[TheDaChicken]

@argilo (Pinging for redundancy)

Hey! I implemented a version of a circular buffer for outputting at a fixed ratio. This buffer is at the bleeding edge!

I store packets in a buffer based on delay instead of 64 audio frames unlike one of your WIPs:

• Sequence numbers don't equal audio frames because each packet represents N+1 frames due to the AAC decoder.
• Enhanced streams can be aligned easily based on delay (since Enhanced hdc #245 uses a hack)

The buffer is meant to look like this:
||latency|| + ||64|| + ||latency||

Such that:
The reader position is placed at 0. The write position is placed after delay (or latency). Allows the reader to have around 32 + latency left in the buffer all the time unless not clocked properly based on the average. if it's not properly clocked, it will miss audio samples. I coded a quick realigner if somehow this happened. These buffers can fill silence due to missing packets yippee! Currently, it seems to be working great.

Dynamically allows any delay into account. For example, enhanced stream delay can be added. When reading the whitepapers, it does read like latency is in a different buffer. There wasn't an easier way to properly take that account without the code being worse.

There is an "output buffer" for each program. This is meant to compensate for the FFT delay & provide IQ samples equal to audio frames. This is very useful for huge IQ buffer sizes. When HD is first received, the program will always start in the middle of a huge IQ sample. There would be no audio available to fill in the gaps before the program starts after the next IQ sample is received from the SDR + There's always a small number of IQ samples left due to FFT.

This output buffer is useful for analog blending for later / do all sorts of buffering techniques.

Little tidbits:

• I made HDC packets be reported through the API delayed to the HDRadio decode (based on average).

I tried the buffer on a semi-bad signal. I thought it was cool how well it handled it so I recorded it: https://youtu.be/pJEh8rs55fs

Please let me know what you think

[ferrellsl]

I compiled this branch (https://github.com/TheDaChicken/nrsc5/tree/elastic-buffering) under MSYS2 to do a little testing. The BER climbs to very high rates after several minutes of playback and loses sync for several seconds causing large gaps in playback even on very strong stations where there's no interference or weak signal.

[TheDaChicken]

The BER climbs to very high rates after several minutes of playback and loses sync for several seconds causing large gaps in playback even on very strong stations where there's no interference or weak signal.

This is either: bad gain / low signal / bad performance

I ran a profiler just in case this new code degrades performance. The new code is very low. acquire_process takes up most CPU which is normal.

If audio is dropping out with a good **MER (BER doesn't matter) then its a new code issue and would be great to see a recording.

[ferrellsl]

The BER climbs to very high rates after several minutes of playback and loses sync for several seconds causing large gaps in playback even on very strong stations where there's no interference or weak signal.

This is either: bad gain / low signal / bad performance

I ran a profiler just in case this new degrades performance. The new code is very low. acquire_process takes up most CPU which is normal.

If audio is dropping out with a good BER then its a new code issue and would be great to see a recording.

The bad gain / low signal /bad performance just doesn't make sense because I'm running an i9-11980H CPU at 3.6Ghz on stations with a receive signal strength of 30-50db. And yes, I used -DUSE_SSE=ON at build time. The average BER stays in the normal range of around .02-.04% for about 10 minutes and then climbs to anywhere from 10% to 40% with the binary compiled from your branch. The audio then drops out for 5 seconds or so and then the error rate drops back into the normal range for another minute or two with the audio track resuming for several minutes before the cycle continues a few minutes later.

This problem doesn't not occur at all when running the binary compiled from the original theori-oi/nrsc5 repo.

Here's a screen shot immediately after the audio drops out showing the bit rates and error rate info as well as signal strength using the binary compiled from your branch.

This problem occurs whether I use the binary from the command line or in conjunction with a python gui. I've verified the same problem on another PC (i7 CPU) using a completely different SDR dongle thinking it might be a system specific problem but the problem persists across both systems.

[TheDaChicken]

This problem occurs whether I use the binary from the command line or in conjunction with a python gui. I've verified the same problem on another PC (i7 CPU) using a completely different SDR dongle thinking it might be a system specific problem but the problem persists across both systems.

I think you are seeing this issue here except its worse:

The last problem with this method is the availability of the packets. They are only properly available after 32 packet time. Any audio driver needs audio before the audio frame is finished. Callback outputs at constant packet time = constant rate which causes lags at the start of playback. I will work on fixing that unless you have an idea yourself.

I will see what I can do.

[ferrellsl]

This problem occurs whether I use the binary from the command line or in conjunction with a python gui. I've verified the same problem on another PC (i7 CPU) using a completely different SDR dongle thinking it might be a system specific problem but the problem persists across both systems.

I think you are seeing this issue here except its worse:

The last problem with this method is the availability of the packets. They are only properly available after 32 packet time. Any audio driver needs audio before the audio frame is finished. Callback outputs at constant packet time = constant rate which causes lags at the start of playback. I will work on fixing that unless you have an idea yourself.

I will see what I can do.

I way out of my depth when it comes to this type of coding. I'm creating a recording at the moment if you think it may hold some clues.

[ferrellsl]

Here's a 7 minute sample. The first audio drop occurs at 3:12 and the second at 6:25 so it appears to be happening at a very regular interval. I can make a longer recording to confirm this if you'd like. https://www.dropbox.com/scl/fi/u9w7bozp02wpigmtq8ob8/sample.wav?rlkey=zmk5y47vlgn7uaxq4zbvn1wl4&st=1f9bg6s0&dl=0

[ferrellsl]

Here's a longer, completely different sample (15 minutes). Curiously, it also experiences the first audio drop at 3:12 and again at 6:25, so there does appear to be a pattern to this. https://www.dropbox.com/scl/fi/ofia7o5fz4m0grm568a35/sample2.wav?rlkey=c063rqif26rl0aj6ej4tz3g2p&st=ue9t9hor&dl=0

[TheDaChicken]

Here's a 7 minute sample. The first audio drop occurs at 3:12 and the second at 6:25 so it appears to be happening at a very regular interval. I can make a longer recording to confirm this if you'd like. https://www.dropbox.com/scl/fi/u9w7bozp02wpigmtq8ob8/sample.wav?rlkey=zmk5y47vlgn7uaxq4zbvn1wl4&st=1f9bg6s0&dl=0

Yep. I figured out the problem - I should have tested subchannels with 4 packet maximum.

Thanks for testing. Try the new latest commit.

[ferrellsl]

I think you solved the problem. There was still a drop at 30 seconds and again at 60 seconds, but after that, everything was just fine and I kept recording for 10 minutes without any further issues.

[ferrellsl]

I ran the binary on my other test system which has an i7-4710Q CPU and the audio drop outs are back. They seem to occur every 3 min and 12 seconds apart. This isn't happening on my system equipped with an i9-11980HK CPU. Strange.

https://www.dropbox.com/scl/fi/iat6xymv8iwl2qcpx8vjh/sample5.wav?rlkey=qwx6fwxsekdgnuuxbzgixd1ql&st=31y1oyf8&dl=0
https://www.dropbox.com/scl/fi/uwc7c61f1iszixwhpv1yi/sample6.wav?rlkey=q1d86nwh71zv331eq2fxt8qk6&st=mt746qfl&dl=0

[TheDaChicken]

I ran the binary on my other test system which has an i7-4710Q CPU and the audio drop outs are back. They seem to occur every 3 min and 12 seconds apart. This isn't happening on my system equipped with an i9-11980HK CPU. Strange.

Could you send me the output of the IQ samples using -w?

[ferrellsl]

I ran the binary on my other test system which has an i7-4710Q CPU and the audio drop outs are back. They seem to occur every 3 min and 12 seconds apart. This isn't happening on my system equipped with an i9-11980HK CPU. Strange.

Could you send me the output of the IQ samples using -w?

Here's an IQ file that's 10 minutes in length. Everything really starts go out of sync badly around the 7 minute mark. https://www.dropbox.com/scl/fi/53p06s6avcq2v88ml44q0/sample.zip?rlkey=j5otk9jv625vx0o03bms1d0aw&st=j0bzgpo0&dl=0

[TheDaChicken]

I ran the binary on my other test system which has an i7-4710Q CPU and the audio drop outs are back. They seem to occur every 3 min and 12 seconds apart. This isn't happening on my system equipped with an i9-11980HK CPU. Strange.

It should be fixed now. The reader clock was going way too fast. Latency was incorrect. The value wasn't multiplied for the delay.

Ignore the continued dumb commits. I deselect things on Clion. It pushes them away. I should use the terminal instead.

[ferrellsl]

Thanks for the update. This appears to have fixed things. I still get the occasional drop out from time to time but I'm not sure if it's code related or if it's RF interference but I don't experience any drops or interference using the theori-io code. Thanks for all the effort you're putting into this. Here's a screenshot of the the occasional audio drops:

And the IQ and WAV files:

https://drive.google.com/file/d/1pR4pjmE8DZ5Ko2L5GnTIume4zF8lVWIq/view?usp=sharing
https://drive.google.com/file/d/13sNWonAVr-Ghm_FH8CvmSTrJk1mTuv1a/view?usp=sharing

[TheDaChicken]

Thanks for the update. This appears to have fixed things. I still get the occasional drop out from time to time but I'm not sure if it's code related or if it's RF interference but I don't experience any drops or interference using the theori-io code. Thanks for all the effort you're putting into this.

It was both RF interference and code-related. I made a silly mistake. Anyway should be fixed.

[ferrellsl]

There's still a code related issue. Here's the output from the latest build. There were only 4 audio drops during this 15 min run, but 3 of them were at regular intervals of appox. 1 min 45 secs apart. Getting much better though.

https://drive.google.com/file/d/1_uewc4lwYNZxejcp2KI4wSAHt11dLcBZ/view?usp=sharing
https://drive.google.com/file/d/1SsVFS9RC1ae8rcvLBDc7rcqa9r5gnD90/view?usp=sharing

[TheDaChicken]

There's still a code related issue. Here's the output from the latest build. There were only 4 audio drops during this 15 min run, but 3 of them were at regular intervals of appox. 1 min 45 secs apart. Getting much better though.

Thanks for the testing yet again even how repetitive this is. I am glad someone else is testing this out. The HDRadio stations here aren't super different.

Unless I couldn't tell from the recording, it looks like RF interference or unrelated to the buffer. From the IQ file, I can see it lost synchronization and resyncs. The buffer properly resyncs tied to that while also keeping playback.

nrsc5 still needs more things to improve reception.

[ferrellsl]

There's still a code related issue. Here's the output from the latest build. There were only 4 audio drops during this 15 min run, but 3 of them were at regular intervals of appox. 1 min 45 secs apart. Getting much better though.

Thanks for the testing yet again even how repetitive this is. I am glad someone else is testing this out. The HDRadio stations here aren't super different.

Unless I couldn't tell from the recording, it looks like RF interference or unrelated to the buffer. From the IQ file, I can see it lost synchronization and resyncs. The buffer properly resyncs tied to that while also keeping playback.

nrsc5 still needs more things to improve reception.

You're welcome. I've enjoyed the testing. Looking forward to any improvements that come about as time progresses.

[FoxxMD]

I compiled and tested with a strong signal inside my city. Everything worked well as far as I could tell!

$ nrsc5 90.1 0 -w samples901.iq -o audio901.wav
Found Rafael Micro R820T tuner
Exact sample rate is: 1488375.071248 Hz
[R82XX] PLL not locked!
09:39:22 Synchronized
09:39:22 Country: US, FCC facility ID: 3538
09:39:22 Station name: WABE
09:39:22 Country: US, FCC facility ID: 3538
09:39:22 Station name: WABE
09:39:22 Audio program 1: public, type: Classical, sound experience 0
09:39:22 Data service: public, type: Emergency, MIME type 444
09:39:22 Country: US, FCC facility ID: 3538
09:39:22 Station name: WABE
09:39:22 Audio program 0: public, type: Information, sound experience 0
09:39:22 Audio program 1: public, type: Classical, sound experience 0
09:39:22 Audio program 2: public, type: News, sound experience 0
09:39:22 Data service: public, type: Emergency, MIME type 444
09:39:23 Country: US, FCC facility ID: 3538
09:39:23 Station name: WABE
09:39:23 Station location: 33.811279, -84.363892, 272m
09:39:23 Audio program 0: public, type: Information, sound experience 0
09:39:23 Audio program 1: public, type: Classical, sound experience 0
09:39:23 Audio program 2: public, type: News, sound experience 0
09:39:23 Data service: public, type: Emergency, MIME type 444
09:39:23 Country: US, FCC facility ID: 3538
09:39:23 Station name: WABE
09:39:23 Slogan: WABE-FM
09:39:23 Station location: 33.811279, -84.363892, 272m
09:39:23 Audio program 0: public, type: Information, sound experience 0
09:39:23 Audio program 1: public, type: Classical, sound experience 0
09:39:23 Audio program 2: public, type: News, sound experience 0
09:39:23 Data service: public, type: Emergency, MIME type 444
09:39:23 MER: 6.1 dB (lower), 5.5 dB (upper)
09:39:23 BER: 0.007637, avg: 0.007637, min: 0.007637, max: 0.007637
09:39:24 Country: US, FCC facility ID: 3538
09:39:24 Station name: WABE
09:39:24 Slogan: WABE-FM
09:39:24 Station location: 33.811279, -84.363892, 272m
09:39:24 Audio program 0: public, type: Information, sound experience 0
09:39:24 Audio program 1: public, type: Classical, sound experience 0
09:39:24 Audio program 2: public, type: News, sound experience 0
09:39:24 Data service: public, type: Emergency, MIME type 444
09:39:25 MER: 10.4 dB (lower), 12.1 dB (upper)
09:39:25 BER: 0.000107, avg: 0.003872, min: 0.000107, max: 0.007637
09:39:25 Title: Juneteenth Mostly Sunny HI 91 Tonight Partly Cloudy Low 70
09:39:25 Title: Juneteenth Mostly Sunny HI 91 Tonight Partly Cloudy Low 70
09:39:25 Audio bit rate: 44.8 kbps
09:39:26 MER: 10.4 dB (lower), 12.1 dB (upper)
09:39:26 BER: 0.000112, avg: 0.002619, min: 0.000107, max: 0.007637
09:39:26 Title: Juneteenth Mostly Sunny HI 91 Tonight Partly Cloudy Low 70
09:39:26 Audio bit rate: 42.9 kbps
09:39:28 MER: 10.2 dB (lower), 11.8 dB (upper)
09:39:28 BER: 0.001231, avg: 0.002272, min: 0.000107, max: 0.007637
09:39:28 Title: Juneteenth Mostly Sunny HI 91 Tonight Partly Cloudy Low 70
09:39:28 Title: Juneteenth Mostly Sunny HI 91 Tonight Partly Cloudy Low 70
09:39:28 Audio bit rate: 43.9 kbps
09:39:28 Country: US, FCC facility ID: 3538
09:39:28 Station name: WABE
09:39:28 Slogan: WABE-FM
09:39:28 Message: Amplifying Atlanta 90.1 WABE
09:39:28 Station location: 33.811279, -84.363892, 272m
09:39:28 Audio program 0: public, type: Information, sound experience 0
09:39:28 Audio program 1: public, type: Classical, sound experience 0
09:39:28 Audio program 2: public, type: News, sound experience 0
09:39:28 Data service: public, type: Emergency, MIME type 444
09:39:29 MER: 7.5 dB (lower), 9.0 dB (upper)
09:39:29 BER: 0.003585, avg: 0.002534, min: 0.000107, max: 0.007637
09:39:29 SIG Service: type=audio number=1 name=WABE-FM
09:39:29   Audio component: id=0 port=0000 type=2 mime=4DC66C5A
09:39:29   Data component: id=1 port=1000 service_data_type=265 type=3 mime=BE4B7536
09:39:29   Data component: id=2 port=1001 service_data_type=265 type=3 mime=D9C72536
09:39:29 SIG Service: type=audio number=2 name=WABE Classical
09:39:29   Audio component: id=0 port=0001 type=15 mime=4DC66C5A
09:39:29   Data component: id=1 port=1002 service_data_type=265 type=3 mime=BE4B7536
09:39:29   Data component: id=2 port=1003 service_data_type=265 type=3 mime=D9C72536
09:39:29 SIG Service: type=audio number=3 name=WABE News
09:39:29   Audio component: id=0 port=0002 type=1 mime=4DC66C5A
09:39:29   Data component: id=1 port=1004 service_data_type=265 type=3 mime=BE4B7536
09:39:29   Data component: id=2 port=1005 service_data_type=265 type=3 mime=D9C72536
09:39:29 Title: Juneteenth Mostly Sunny HI 91 Tonight Partly Cloudy Low 70
09:39:29 Audio bit rate: 43.3 kbps
09:39:31 MER: 10.2 dB (lower), 12.1 dB (upper)
09:39:31 BER: 0.000175, avg: 0.002141, min: 0.000107, max: 0.007637
09:39:31 Title: Juneteenth Mostly Sunny HI 91 Tonight Partly Cloudy Low 70
09:39:31 Audio bit rate: 43.5 kbps
09:39:32 MER: 10.3 dB (lower), 11.9 dB (upper)
09:39:32 BER: 0.000268, avg: 0.001874, min: 0.000107, max: 0.007637
...

I took a 10 minute sample and there only seemed to be one hiccup at 2:12 that was barely noticeable.

https://share.foxxmd.dev/-bnJNyqjYh3 -- .wav audio

The IQ file is very large (1.8GB), I can share if you need it.

https://share.foxxmd.dev/-hUTvtaekFg -- IQ sample file @TheDaChicken

EDIT: Been running for the last 16 hours and the stream is still smooth, no hiccups. 💪

[ferrellsl]

Just wanted to post another update after some further testing. I recompiled after your latest commits and have let the binary run for about 12 hours now without any issues. This is the most stable version of nrsc5 yet on my systems. Whenever I used the binary compiled from theori-io, it would freeze or the audio would get choppy at anywhere from 15 minutes to 4 hours of playback. I'm going to let your binary run for another 24 hours or so just to see what happens. I don't expect any glitches though. Thanks again for the hard work you've put into your fork. I really appreciate it.

[argilo]

Thanks again for your work on this @TheDaChicken. I used your proposed changes here as inspiration, and made a simplified version in #399 which seems to work well. As long as no serious problems come up, I'll get that merged within a few days so we can finally have elastic buffering.

[argilo]

I just merged #399, so I'll close this off.

We may return to the idea of maintaining an (almost) exact ratio of input samples to output samples in a future PR, to support blending (#80).