Any chance of keeping the model loaded

[StuartIanNaylor]

Rather than streaming could the model load and just watch a folder and move audio or delete when processed.
So you don't get any load time from processing file inputs than the current ./main ?

[tomheno]

Hi there, looking to achieve the same thing, maybe simply by loading the model into memory and accessing it on the next ./main run or so...
Our goal is to run on audio chunks of 3/4s, and to avoid the ~500ms of loading, once it's already been loaded.
Any suggestion on a way to achieve that would be very appreciated, thanks in advance.

[RndyP]

You call whisper_init() only once, and then start acquiring and processing your chunks. What is your platform/environment? Are you writing and app, or trying to shell out to main in a loop???

[tomheno]

Hi @RndyP, thanks for your answer !
Currently using the python implementation, but shifting to cpp implementation to experiment on performances, the final goal is to have either a local port receiving simple POST requests including a single file path (of a .wav file), returning the transcript for it as a response.
The implementation you are describing is what I had in mind, to precise it a bit : the goal is for the process to run continuously and to receive/transcript files on the go, not to run on a predefined batch of files.

Thanks again :)

[RndyP]

Sounds like you need a hybrid of the stream and "WAV" file examples. If you start with the stream example you will need to replace the audio card interface with a thread that scans a folder for new WAV files, and then keeps calling whisper_full() with each chunk read in from the file (and then erasing the file?). You will probably want to tweak the default whisper_parms depending on what you finally decide.

I would guess the nuances of either of these methods may depend on how the "gapping" is done. Are your files separated in real time audio by seconds? Minutes? Continuous?

[ggerganov]

@tomheno @RndyP
You don't need to start from stream example.

Just modify main - instead of looping over the input files:

https://github.com/ggerganov/whisper.cpp/blob/4ef3398e8fe0c426de038b8c3816c6791a34a5ae/examples/main/main.cpp#L504-L505

Scan the folder for any new files and load them instead. The rest of the loop is the same.
Also, remember which files have been processed so you don't re-process them again.

[StuartIanNaylor]

That would be really cool as maybe use inotify to scan a in-folder and move to an out-folder with inference text.

If any does do something please shout.

[silvacarl2]

i have looked at installing it on a RAM disk but have not finished testing that yet. also, trying to see what sort of a wrapper could be used to create an API out of it so it is always in memory.