peft provides the position of the model input?

[ciaoyizhen]

I went to the official repository of the ChatGLM fine-tuning code, and I asked them why position was not provided for the model. Their reply was that peft provided position, so it is no longer needed. Then I saw the following code in peft's code

         if kwargs.get("position_ids", None) is not None:
             warnings.warn("Position ids are not supported for parameter efficient tuning. Ignoring position ids.")
             kwargs["position_ids"] = None

It seems that it will process the position, but I looked at the code and didn't understand how it generated the position and provided it to the model.
Help, thank you

[Wangmerlyn]

On the question of why position_ids are not explicitly needed and can be internally handled by the model — taking ChatGLM4-6B as an example:

ChatGLM uses Rotary Position Embedding (RoPE) for positional encoding. Since RoPE is a form of relative position embedding, as long as tokens are fed into the model in a continuous manner (i.e., positions follow a natural order like 0, 1, 2, ..., n), the model can compute positional information internally using the RoPE formulation — no need to explicitly supply position_ids.

Regarding how position_ids are generated when needed, you can refer to the prepare_inputs_for_generation function in the official implementation. The position IDs are constructed as a 2D tensor of shape (batch_size, seq_len), where each row is a sequence of increasing integers from 0 to seq_len - 1:

def get_position_ids(self, input_ids, device):
    batch_size, seq_length = input_ids.shape
    position_ids = torch.arange(seq_length, dtype=torch.long, device=device).unsqueeze(0).repeat(batch_size, 1)
    return position_ids

In summary: with RoPE, explicit position IDs are optional as long as token positions are consistent — the model can infer them internally.