Skip to content

Respect tokenizer.ggml.add_bos_token value when tokenizing#4040

Merged
KerfuffleV2 merged 4 commits intoggml-org:masterfrom
KerfuffleV2:fix-respect-use-bos-token
Nov 17, 2023
Merged

Respect tokenizer.ggml.add_bos_token value when tokenizing#4040
KerfuffleV2 merged 4 commits intoggml-org:masterfrom
KerfuffleV2:fix-respect-use-bos-token