ScribeAI - a native ios app for Whisper with real-time transcribing

[RRUK01]

ScribeAI.mp4

I've been inspired by the whisper project and @ggerganov and wanted to do something to make whisper more portable. So I've made ScribeAI a native ios app that runs whisper (base, small & medium) all on-device.

You can get here: https://apps.apple.com/app/scribeai/id6450321299

It includes the following:

• Live mode (as above) as well as post-speech mode, which processes speech when there's been a significant pause to save on battery.
• Base, Small & Medium(!) models all on-device. The demo above is using the Base model and the video is a true live demonstration (no editing involved).
• The models are all Q6_K quantised for the decoder and the encoder runs on CoreML. I experimented a lot with this and found this had the highest accuracy. I've open sourced the code for K_Quants for Whisper (Metal Decoder Inference, K-Quants added, Sync'd with latest GGML #1074), I've been having a look at the making the decoder run on Metal as well and will open source this too if I can crack it.
• Searchable recordings, you can search through your recordings and it'll pull up all the recordings with your keyword in the transcript.
• Everything is on-device. No data is collected, as such it even works offline and in airplane mode - true portability.
• On-device translation supported (to English).
• Spoken word highlighting on playback

In the future we're looking to add:

• Speaker diarization taking inspiration from [1] and the https://github.com/akashmjn/tinydiarize library
• More accurate word level timestamps
• More export options
• Fine-tune the models further, currently collecting data to support this and money from any subscriptions will pay towards supporting this effort

Hope you enjoy it and happy to hear any feedback or feature requests at [email protected]!

[1] https://arxiv.org/abs/2305.18747

[boringpapa]

Great effort on making this !

Can I request for a massive feature

Search & Replace library , where we can import words from a CSV file . Of course Whisper gets tonnes of words wrong , currently I run a python script on all the transcription. but If it can be run locally , cherry on top. . If you do decide to implement , please give import/export facility , my list only 2 weeks old is already at 132 substitutions.

[boringpapa]

Looking forward to speaker diarization

[RRUK01]

Hey,

Just released a new version (1.1.4) with more export options (srt / csv / txt / sbv) and editing in-app enabled. I've not introduced the search and replace but it's something I can definitely look to include in a future version

The new version (1.1.4) also includes the following:

• Whisper Large v2 now included - you can run this completely on device
• Whisper Medium (Balanced) and Small (Rapid) now run about 20-30% faster

I'm working on making the models run faster each day, at the moment in particular I'm trying to get Whisper Medium running in near real-time and reduce the RAM requirements for Whisper Large v2 so it can be ran for longer.

Let me know what you think and any other feature requests

[RRUK01]

Update

Have now reduced RAM requirements for the Whisper Large v2 model by about 40% and made further speed improvements in the Small and Medium models. Small basically runs in real-time on newer iphones and medium in near real time.

I've also removed the Base model from the app (the demo in the first post is the Base model) as I've got the small model running just as quick now.

[boringpapa]

Wohoo. This is great to hear . With last update my iPhone 13 used to really heats up during transcription. Will try with this and share.

[RRUK01]

Wohoo. This is great to hear . With last update my iPhone 13 used to really heats up during transcription. Will try with this and share.

Hey, if heat still ends up being an issue I'd recommend using in 'post speech' mode, it's much less compute/gpu intensive.

[albertputinjobs]

Plz make a chrome extension like this It's more helpful