What is the proper way to download the Whisper CoreML models and the tokenizer model?

[Kavi-Gupta]

Hi I am trying to use WhisperKit completely offline, I would like to bundle models with the app, instead of relying on a network connection. I have tried downloading the model files from the HuggingFace repository. But, regardless of which model I download, I always get the error:

MelSpectrogram.mlmodelc/coremldata.bin is not a valid .mlmodelc file because the first word (1936876918) is not recognizable.
Unable to load model: ... Compile the model with Xcode or `MLModel.compileModel(at:)`.

How do I go about fixing this? Also, does each model have a separate tokenizer? And, where do I get the tokenizer model(s) from?

Thanks.

[atiorh]

How do you download the models? It could be that you are not fetching the full git lfs files but only pointers

[Kavi-Gupta]

@atiorh I run

make setup

then run

make download-model MODEL=distil-large-v3

and then copy the

distil-whisper_distil-large-v3

into my Xcode project. Interestingly, in the Models folder, it seems like all models are present, not just the distil-large-v3.

[Kavi-Gupta]

I tried using

MODEL=large-v3

And this one took a while longer but there was no error when running it in Xcode. So, my guess is that distil-large-v3 is not the right name. Also, where do I download the tokenizer for a given model?

[maczg]

I tried using

MODEL=large-v3

And this one took a while longer but there was no error when running it in Xcode. So, my guess is that distil-large-v3 is not the right name. Also, where do I download the tokenizer for a given model?

I suggest downloading all the models by make download-models. It will result in download locally all the models in Model folder.
Next, you have to copy the folder of the model you want to embed inside xcode project. Make sure to copy and not reference the folder, and also select the proper target (your app) on copying the folder.

Also be sure that the model folder (or any other hierarchy you have in mind) is a folder and not a group. This will lead to copying the model folder in the main bundle exactly as it is in your Xcode project hierarchy, and so having the model folder embedded in the main Bundle.

Then something like the example below should work.

let modelPath = Bundle.main.path(forResource: $PATH_TO_MODEL_FOLDER, ofType: nil)!
whisperKit = try await WhisperKit(
                model: "tiny",
                modelFolder: Bundle.main.path(forResource: modelPath, ofType: nil)!,
                //omitted options

                download: false
            )
logger.info("WhisperKit initialized successfully")

I've not fully tested, but it seems to load the model properly. Keep in mind that there are limitations on the bundle size to publish the app, so choose the model to embed also based on this

[Kavi-Gupta]

Using code similar to the one you provided, I was able to get local models to work. However, I am struggling to get the tokenizer to load locally. Here is the code I am using:

guard let modelFolder = Bundle.main.url(forResource: self.modelFolder, withExtension: nil, subdirectory: nil) else {
            whisperKitError = "Could not find model folder"
            return
        }
        
        guard let tokenizerFolder = Bundle.main.url(forResource: self.tokenizerFolder, withExtension: nil, subdirectory: nil) else {
            whisperKitError = "Could not find tokenizer folder"
            return
        }
        
        let computeOptions = ModelComputeOptions(melCompute: .cpuAndGPU, audioEncoderCompute: .cpuAndGPU, textDecoderCompute: .cpuAndGPU, prefillCompute: .cpuAndGPU)
        let whisperKitConfiguration = WhisperKitConfig(model: "tiny.en", modelFolder: modelFolder.absoluteString, tokenizerFolder: tokenizerFolder, /*computeOptions: computeOptions,*/ verbose: true, logLevel: .debug, download: false)
        

        do {
            print("Initializing WhisperKit")
            self.whisperKit = try await WhisperKit(whisperKitConfiguration)
            whisperKitError = ""
            print("WhisperKit Initialized")
            print("Model folder: \(whisperKit?.modelFolder)")
            print("Tokenizer folder: \(whisperKit?.tokenizerFolder)")
        } catch {
            whisperKitError = error.localizedDescription
            print(whisperKitError)
        }

Both folders exist, but it is still loading the tokenizer from the HF Hub, but failing to save it, as shown in this screenshot, which makes me think that tokenizerFolder specifies where to save a downloaded tokenizer. If I don't provide tokenizerFolder, it downloads it from the hub (except when it is already cached).

For reference, here is what my folder structure looks like:

Naturally, I inspected the underlying code and after adding a line of logging to the WhisperKit source code found that it was looking for my tokenizer files in:

/Models/tokenizers/whisper-tiny.en/models/openai/whisper-tiny.en/tokenizer.json

This is from ModelUtilities.swift

let tokenizerName = tokenizerNameForVariant(pretrained)
let hubApi = HubApi(downloadBase: tokenizerFolder, useBackgroundSession: useBackgroundSession)

// Attempt to load tokenizer from local folder if specified
let resolvedTokenizerFolder = hubApi.localRepoLocation(HubApi.Repo(id: tokenizerName))
let tokenizerConfigPath = resolvedTokenizerFolder.appendingPathComponent("tokenizer.json")
      
Logging.debug("Looking for tokenizer.json at \(tokenizerConfigPath.path)")

So, it was adding models/openai/whisper-tiny.en/tokenizer.json to whatever folder I gave it. Which I found counterintuitive since modelFolder checks the exact folder I give it.

This leads me to ask, what is the appropriate folder structure for the tokenizers and models? Is there a way for me to provide a folder with say all the models (ignoring bundle size) and a folder with all the tokenizer models and just tell WhisperKit which model I want and WhisperKit will automatically pull the appropriate files?

I could not find any documentation for any of this behavior and I was just wondering if it would be useful for me to create some documentation for using models completely offline and put it on this repository. Also, I think to help users debug in the future, it is useful to keep Logging.debug("Looking for tokenizer.json at \(tokenizerConfigPath.path)"). Should I create a PR for it? I also just wanted to bring up that .path has been deprecated in favor of .path(percentEncoded: Bool)

[argmaxincbot]

This is good feedback for the library, the tokenizer handling is divergent enough from the model handling that it would be great to have a unified process that respects the config as expected. Would be awesome to see a PR with the changes that would fit your use case!

• Zach

[Kavi-Gupta]

@argmaxincbot
Hi Zach,

Here is what I am thinking. In the code that checks for the existence of the tokenizerFolder after the additional path is added, if it is not found, it will check just the original path (without the modifications) before throwing the error. Also, for documentation, I am thinking of creating two documentation files, one which documents how to use AudioStreamTranscriber and one that explains how to use models completely offline. For these documents, what is the best format? I was thinking of creating two Markdown files and possibly putting them in a docs directory for more formal documentation instead of relying on comments. How does that sound?

[maczg]

Hi!
Thank you all for the efforts. I'm dealing with the tokenizer.
I (erroneously) thought that loading the model folder (e.g. tiny) would resolve the problem, but noticed only after the @Kavi-Gupta comments that the tokenizer was downloaded from Hub.

Is there any documentation about the use of tokenizer? For example, from where to download it to embed it in the model folder?