Giving context to WhisperKit models

[oabdullah3]

a) Is it possible to "prompt" a WhisperKit model. i.e. provide it with a set of "hotwords" or special words that I know it will be encountering quiet often in the audios it is given to transcribe. Essentially, can I provide it with context to better transcribe the conversations i give it, and with more accuracy?

b) Is their any multilingual WhisperKit supported model that I can use, and if yes, which one would be the best to use?

[atiorh]

Yes with disclaimers!

1. All Whisper models technically support prompting and you can trace this CLI argument back to the SDK to see how it all works in the core implementation.

2. In terms of accuracy, only large-v2 and large-v3 are known to work reasonably well with prompting. All other models behave erratically when used with prompts. This is an unreliable way to attempt to get "hotwords" from the model. Furthermore, you are limited to 224 tokens for the prompt which is roughly slightly less than 100 words.

3. To address these shortcomings, we have launched Custom Vocabulary in Argmax Pro SDK which is currently in alpha and scheduled to ship in a stable release in 2 weeks. Feel free to follow us on https://x.com/argmax to get notified when this ships, you will be able to test it on our TestFlight app Argmax Playground.

[oabdullah3]

I see. I digged into the WhisperKit.transcibe() function and saw that you can pass as options prefix text and prompt text. Can you help me understand the difference between the two, and whether or not they will serve my purposes (even if with shortcomings)?

[oabdullah3]

Additionally, if you could help me understand how to tokenize my promptText, to act as input for my transcribe function, that would be really helpful as well