Transformer 从4.57.3 升级到5.3.0 后过滤数据时长变慢十倍以上

[chenjiaoAngel]

System Info

H20

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

过滤函数实现：对应链接 https://github.com/verl-project/verl/blob/main/verl/utils/dataset/rl_dataset.py#L241

          def doc2len(doc) -> int:
                    try:
                        apply_kwargs = dict(**self.apply_chat_template_kwargs)
                        if self.tool_schemas is not None:
                            apply_kwargs["tools"] = self.tool_schemas

                        # Keep explicit tokenization to avoid transformers version default changes.
                        apply_kwargs.pop("tokenize", None)
                        apply_kwargs.pop("return_dict", None)
                        apply_kwargs.pop("return_tensors", None)

                        tokenized_prompt = tokenizer.apply_chat_template(
                            doc[prompt_key], add_generation_prompt=True, tokenize=True, **apply_kwargs
                        )
                        return len(normalize_token_ids(tokenized_prompt))
                    except Exception:
                        print("Error processing one of the samples, skipping...")
                        traceback.print_exc()
                        return self.max_prompt_length + 1

            dataframe = dataframe.filter(
                lambda doc: doc2len(doc) <= self.max_prompt_length,
                num_proc=self.num_workers,
                desc=f"Filtering prompts longer than {self.max_prompt_length} tokens",
            )

使用transformer 4.57.3 运行这段code大概十几分钟，如果升级到5.3.0 需要约两个小时；辛苦帮忙看看 模型=qwen3.5-35b-A22 数据集是aime-2024
4.57.3 结果

5.3.0 结果

Expected behavior

希望transformer 升级后，其tokenizer 的速度没有变化

[sparkingarthur]

你应该检查一下，是不是因为v5 tokenize之后，input id变多了……我发觉v5会自动在所有中文前面加空格。
You should check if the increase in input ids after v5 tokenization. I find that v5 automatically adding spaces before all Chinese characters.

[Rocketknight1]

cc @ArthurZucker @itazap to the above comment since it seems important ^

[chenjiaoAngel]

cc @ArthurZucker @itazap to the above comment since it seems important ^

我现在想到方法，区分文字和多模态任务，如果是纯文本话，走processor.tokenizer()处理，否则走processor() 处理；这样可以大大减少纯文本任务的数据处理；I’ve come up with a solution to distinguish between text-only and multi-modal tasks:
For text-only tasks, process data via processor.tokenizer(* kwargs);
For multi-modal tasks, process data via processor(** kwargs).
This approach will significantly reduce the data processing overhead of text-only tasks.

[chenjiaoAngel]

你应该检查一下，是不是因为v5 tokenize之后，input id变多了……我发觉v5会自动在所有中文前面加空格。 You should check if the increase in input ids after v5 tokenization. I find that v5 automatically adding spaces before all Chinese characters.

我看了应该不是的； It's not

[itazap]

我看了应该不是的； It's not

@chenjiaoAngel hello, just confirming if you're still facing an issue?