Transformers v5 compatibility

[Tcc0403]

This is an issue tracker for transformers v5.

Known issues

Feel free to report if you find any issue that isn't listed below.

Monkey patch

Errors when running

python3 -m pytest test/transformers/test_monkey_patch.py

See details in #960 (comment)

• AttributeError for 'language_model' in transformers v5 Fix missing property access for multimodal models #966
• KeyError: 'mrope' rope parameters for transformers v5 #1013
• Mixtral: 'MixtralDecoderLayer' object has no attribute 'block_sparse_moe'
• 'Llama4VisionConfig' object has no attribute 'rope_theta' rope parameters for transformers v5 #1013
• 'Gemma3ForConditionalGeneration' object has no attribute 'vision_tower'
• MoE breaking change, see also Moe kernel for latest transformers v5 #958
• Failing tests because of pad_token_id Transformers v5 monkey_patch test failures because of pad_token_id #1059

Convergence test

Errors when running

python3 -m pytest test/convergence/*

• Some XXXTokenizerFast are deprecated: No module named transformers.models.gemma.tokenization_gemma_fast Fix/transformers v5 gemma tokenizer #1030
• llava: vision_tower generate outputs.hdiden_states=None Llava vision_tower monkey patch generates hidden_states=None #1011
• qwen2_vl, qwen2_5_vl: 'Qwen2_5_VLConfig' object has no attribute 'hidden_size' Qwen2VLConfig and Qwen2_5_VLConfig have no attribute hidden_size #1012
• AttributeError: object has no attribute 'pad_token_id' AttributeError: <config> object has no attribute 'pad_token_id' in transformers v5 convergence tests #1081

Benchmarking

• qwen2vl_mrope: Qwen2VLRotaryEmbedding expects config argument instead of int(hidden_dim) Benchmark benchmark_qwen2vl_mrope.py fails with new transformers #1015

For Contributors

Since we want to keep backward compatibility for transformers>=4.49.0 as well, tests will focus on 1. transformers==4.49.0 (the oldest version) 2. transformers<5.0.0 (last v4 version) 3. transformers>=5.0.0 (the latest version)

Environment setup

uv pip install transformers==4.49.0

uv pip install transformers<5.0.0

uv pip install transformers>=5.0.0

Pull requests

Open pull requests on main branch directly.

It would be great if you can provide relevant tests on three version listed above in your PR description. Thanks for you contribution!

[Monishver11]

Hello @Tcc0403. I'm interested in helping out with this. I see that for some issues, the work is being made but not yet assigned. Can you let me know any issues that I can work on and help with?

[Egor-Krivov]

Rope benchmark currently fails with transformers 5.0:

Details

(triton) (312) jovyan@jupyter-ekrivov:~/triton/intel-xpu-backend-for-triton-copy1/Liger-Kernel$ python benchmark/scripts/benchmark_rope.py 
**************************************
     BENCHMARKING SPEED for ROPE
**************************************
Traceback (most recent call last):
  File "/home/jovyan/triton/intel-xpu-backend-for-triton-copy1/Liger-Kernel/benchmark/scripts/benchmark_rope.py", line 179, in <module>
    run_benchmarks(
  File "/home/jovyan/triton/intel-xpu-backend-for-triton-copy1/Liger-Kernel/benchmark/scripts/utils.py", line 313, in run_benchmarks
    benchmark_result: SingleBenchmarkRunOutput = bench_test_fn(single_benchmark_run_input)
  File "/home/jovyan/triton/intel-xpu-backend-for-triton-copy1/Liger-Kernel/benchmark/scripts/benchmark_rope.py", line 70, in bench_speed_rope
    ms_50, ms_20, ms_80 = triton.testing.do_bench(
  File "/home/jovyan/triton/intel-xpu-backend-for-triton-copy1/python/triton/testing.py", line 149, in do_bench
    fn()
  File "/home/jovyan/triton/intel-xpu-backend-for-triton-copy1/Liger-Kernel/benchmark/scripts/benchmark_rope.py", line 65, in fwd
    return apply_rotary_pos_emb(q, k, cos, sin, pos_ids)
  File "/home/jovyan/triton/intel-xpu-backend-for-triton-copy1/.venv/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 163, in apply_rotary_pos_emb
    cos = cos.unsqueeze(unsqueeze_dim)
TypeError: unsqueeze(): argument 'dim' (position 1) must be int, not Tensor

Fix

The fix:

(triton) (312) jovyan@jupyter-ekrivov:~/triton/intel-xpu-backend-for-triton-copy1/Liger-Kernel$ git diff benchmark/scripts/benchmark_rope.py 
diff --git a/benchmark/scripts/benchmark_rope.py b/benchmark/scripts/benchmark_rope.py
index ac79288..1951c3c 100644
--- a/benchmark/scripts/benchmark_rope.py
+++ b/benchmark/scripts/benchmark_rope.py
@@ -62,7 +62,7 @@ def bench_speed_rope(input: SingleBenchmarkRunInput) -> SingleBenchmarkRunOutput
         if provider == "liger":
             return liger_rotary_pos_emb(q, k, cos, sin, pos_ids)
         elif provider == "huggingface":
-            return apply_rotary_pos_emb(q, k, cos, sin, pos_ids)
+            return apply_rotary_pos_emb(q, k, cos, sin)
         else:
             raise ValueError(f"Invalid provider: {provider} for RoPE embedding")
 
@@ -143,7 +143,7 @@ def bench_memory_rope(input: SingleBenchmarkRunInput) -> SingleBenchmarkRunOutpu
         if provider == "liger":
             q_out, k_out = liger_rotary_pos_emb(q, k, cos, sin, pos_ids)
         else:
-            q_out, k_out = apply_rotary_pos_emb(q, k, cos, sin, pos_ids)
+            q_out, k_out = apply_rotary_pos_emb(q, k, cos, sin)
         torch.autograd.grad((q_out, k_out), (q, k), (dq, dk), allow_unused=True, retain_graph=True)
 
     mem_50, mem_20, mem_80 = _test_memory(

[Tcc0403]

Hello @Tcc0403. I'm interested in helping out with this. I see that for some issues, the work is being made but not yet assigned. Can you let me know any issues that I can work on and help with?

@Monishver11 #1013, goal is 0 rope_parameters related errors

[Monishver11]

@Tcc0403, thanks for the reply. I'll start checking this.