Is beam search not efficient enough? It's slower than Faster Whisper.

[twmht]

When I set the beam size to 1, TensorRT-LLM is about 50% faster than Faster Whisper. However, when I set the beam size to 5, the speeds are roughly the same. TensorRT-LLM's latency increase is significantly greater than Faster Whisper's. Any thoughts?

[lfr-0531]

When setting beam size to 5, we will predict different output tokens for differnet beams. For such a case, in the decode phase, if batch_size=1, the equivalent batch will be changed from 1 to 5. So the latency will increase.

[twmht]

So i need to set batch size to 5 to reduce the increased latency?

[lfr-0531]

Nonono, I mean you need to set beam size to 5 or batch size to 5 to Faster Whisper for fair perf comparison.

[twmht]

yeah I set beam size to 5 for faster whisper, but the latency of TensorRT-LLM increased a lot as compared to faster whisper

[lfr-0531]

Got it. I think you can fill a GitHub issue to report this large perf drop.

[MahmoudAshraf97]

TRT-LLM is 3 times faster than CT2 for whisper in almost all cases, you just need to give it enough inputs to see the difference, if you are using it for personal usage and not production, then stick to faster-whiper.
Source: I'm the maintainer of faster whisper