Fix for loading tokenizers using non-utf8 strings#55

Merged

tqchen merged 3 commits intomlc-ai:mainfrom

ThomasProg:utf8fix

Feb 24, 2025

Commits on Jan 13, 2025

std::str::from_utf8() replaced by String::from_utf8_lossy() to fix non utf8 data error in case of BPE use
ThomasProg
committed
Merge remote-tracking branch 'upstream/main' into utf8fix
ThomasProg
committed
fixed invalid type
ThomasProg
committed