How to handle real-time sound streams

[attsion]

thank u

[chriswalz]

I'm also interested to know how use this with real-time audio streams

[bmduarte]

I managed to record audio chunks in real time using NAudio and iterate them as they were available.
It seems to slow to handle them. Is there any way to speed things up?

[VRCWizard]

https://github.com/AwesomeYuer/whisper.NAudio.NET is an example but it does seem too slow for real time with this repo.

https://github.com/Const-me/Whisper/tree/master/Examples/MicrophoneCS This other implementation is much faster but it doesn't use naudio loopback (doesn't detect silence... it seems to basically chuck audio depending on what you set the capture max and min duration to be)

[sandrohanea]

Nice options are provided here,
I am thinking that the best solution will be to have something like:

Having 2 configurable values:
IntervalTime = The time needed for one processing (e.g. default to 5 seconds)
OverlapTime= The time which will be processed twice in order to have continuity. (Default to 1 second)

1. Wait for the initial {IntervalTime} and process it.
2. Wait for additional {IntervalTime} for the second interval and process the time from {IntervalTime} - ({OverlapTime} / 2) to 2 * {IntervalTime}
3. Identify some common segment at the end of Result1 and Begginging of Result 2 and merge them.
4. Repeat the process.

During merge we need to keep in mind that exact the end of the segment might be gibberish, the same as the begging of the new segment (as it can be a word which is cut in half in any part).

This way, the context will be maintained as if you just cut randomly and process everything it can end up in the middle of the word and that cannot be recognized.

On the other hand, if we're always processing everything from 0 to CurrentTime, that will become too slow.

In order to improve quality, we can increase the overlap time.

Ideally, there would be this capability directly in this package, and anyone would be able to use NAudio or other stream provider to call some library (e.g. some PushStream similar to Azure Cognitive Service's PushStreams: https://github.com/Azure-Samples/cognitive-services-speech-sdk/blob/7e61fcb022f5dd75cfaf579703f8c92ad83317b0/samples/csharp/sharedcontent/console/speech_recognition_samples.cs#LL352C26-L352C26 )

[dfengpo]

+1

[sandrohanea]

For anyone exploring near-real-time audio stream processing, I’d like to invite you to check out the new library: EchoSharp (currently in its early stages).

EchoSharp is designed to leverage Whisper.net, along with other Speech-to-Text components and VAD modules, to enable near-real-time audio processing.

Your early feedback would be incredibly valuable, so if you have some time to try it out, I’d love to hear your thoughts!