Issue with llm judge tokenizer

[pipiPdesu]

In roll/pipeline/rlvr/rewards/llm_judge_reward_worker.py, the reward worker receive tokenizer from

ROLL/roll/pipeline/rlvr/rewards/llm_judge_reward_worker.py

Line 53 in 2b807c7

self.tokenizer = default_tokenizer_provider(model_args=self.worker_config.model_args)

This tokenizer need to decode the actor_infer's rollout

ROLL/roll/pipeline/rlvr/rewards/llm_judge_reward_worker.py

Line 222 in 2b807c7

response_text_list = self.tokenizer.batch_decode(data.batch["responses"], skip_special_tokens=True)

and encode the llm judge query

ROLL/roll/pipeline/rlvr/rewards/llm_judge_reward_worker.py

Line 105 in 2b807c7

tokenized = self.tokenizer(text, return_tensors="pt")

It would be fine if the llm judge model and trained model use the same tokenizer, however, if they differ, there may not be an appropriate tokenizer available to decode the response correctly. To address this, it is reasonable to add a new argument to the reward that allows specifying llm judge tokenizer and trained model tokenizer.

[PanAndy]

Thank you for your interest in ROLL.
Yes, your suggestion is correct.
In a reward model, when the tokenizer of the reward model is different from that of the policy model, it is necessary to convert the input_ids.
I have also marked this as a TODO in the base_worker.RewardWorker, and here is the code:

ROLL/roll/pipeline/base_worker.py

Line 575 in 2b807c7

# TODO: _switch_chat_template, 异构reward model

This feature is planned for future development.
If you have time, feel free to implement this feature — and we would be very welcome to your contribution to ROLL!

[HuangJoJo]

fixed by: 41fe274