Repository navigation
Add GraniteForCausalLM mappings for AWQ and SmoothQuant - #2797
brian-dellabetta merged 4 commits into
Conversation
|
👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review. Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed. |
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR registers ChangesGraniteForCausalLM Model Support
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Warning Review ran into problems🔥 ProblemsLinked repositories: Your configuration references 1 linked repositories, but your current plan allows 0. Analyzed ``, skipped Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Merge Protections🟢 Merge protection satisfied — ready to merge. Show 1 satisfied protection🟢 Require two reviewsPRs labelled "two-reviews" must have at least two approving reviews before merging.
|
fc5d3ec to
a703007
Compare
Mirrors the pattern from vllm-project#2800 / vllm-project#2797. Tiny config + torch.device('meta') keeps the test under a unit-test budget; verifies DEFAULT_SMOOTHQUANT_MAPPINGS regex matches StableLM's actual layer names (q/k/v_proj + gate/up/down_proj + input/post_attention_layernorm). Addresses the requires-validate label. Signed-off-by: HumphreySun98 <humphreysun98@gmail.com>
|
Added registry + meta-device validation in 135ae6c to address |
|
This pull request has merge conflicts that must be resolved before it can be |
Mapping Validation ReportPR: #2797 SummaryAWQ: 1/1 PASS | SmoothQuant: 1/1 PASS Both AWQ Results
SmoothQuant Results
Notes
|
brian-dellabetta
left a comment
There was a problem hiding this comment.
Thanks, validation is passing. Please resolve conflicts
|
@coderabbitai resolve merge conflict |
|
❌ An unexpected error occurred while resolving merge conflicts: Not Found - https://docs.github.com/rest/git/refs#get-a-reference |
|
The quality checks have failed. Please run |
|
https://github.com/coderabbitai resolve merge conflict |
Maps the standard IBM Granite dense decoder architecture
('GraniteForCausalLM') to the existing default mappings in both
registries. The architecture uses the Llama-style layer naming
(input_layernorm/post_attention_layernorm RMSNorm + q/k/v/o_proj +
gate/up/down_proj) with no QK-norm, so the existing default mappings
apply directly without a new mapping definition.
Part of vllm-project#1442.
Signed-off-by: HumphreySun98 <humphreysun98@gmail.com>
Addresses the requires-validate label and matches the test pattern established for the MoE batch in vllm-project#2800: - test_granite_for_causal_lm_in_registry: asserts the new entry is registered and resolves to DEFAULT_SMOOTHQUANT_MAPPINGS. - test_granite_default_mapping_regex_matches_real_module_tree: instantiates GraniteForCausalLM on the meta device with a tiny config and asserts every re-prefixed pattern in DEFAULT_SMOOTHQUANT_MAPPINGS matches at least one module. No HF Hub downloads, no weight allocation. Signed-off-by: HumphreySun98 <humphreysun98@gmail.com>
135ae6c to
42c2630
Compare
|
Rebased onto latest The conflict was in
|
|
✅ Action performedReview finished.
|
SUMMARY: Adds \`GraniteForCausalLM\` (IBM Granite dense decoder) to both the AWQ and SmoothQuant layer-mapping registries. Maps to the existing \`default_mappings\` (AWQ) / \`DEFAULT_SMOOTHQUANT_MAPPINGS\` (SmoothQuant), since the architecture uses standard Llama-style layer naming: \`input_layernorm\` / \`post_attention_layernorm\` (parametric RMSNorm) + \`q_proj\`/\`k_proj\`/\`v_proj\`/\`o_proj\` + \`gate_proj\`/\`up_proj\`/\`down_proj\`, with no QK-norm. No new mapping definition is needed. This is part of issue #1442 (the keep-open \"add additional model mappings\" tracker), continuing the pattern established by recent PRs that registered single architectures using existing mapping shapes (#2419 for \`GlmMoeDsa\`, #2609 for Qwen MoE variants, #2639 for Cohere/DeepSeek-V3/Phi3). Verified the architecture matches the \`default\` shape by reading vLLM's \`vllm/model_executor/models/granite.py\`: GraniteDecoderLayer uses \`RMSNorm\` for both \`input_layernorm\` and \`post_attention_layernorm\` (line 228 / 258 in current vLLM main), and \`stacked_params_mapping\` resolves \`q_proj\`/\`k_proj\`/\`v_proj\` and \`gate_proj\`/\`up_proj\` to the standard names, so the existing \`re:.*input_layernorm$\` → q/k/v_proj and \`re:.*post_attention_layernorm$\` → gate/up_proj smoothings apply without modification. \`GraniteMoeForCausalLM\` and \`GraniteMoeSharedForCausalLM\` are deliberately **not** included here — those use the Mixtral-style \`block_sparse_moe.experts.*.w1\`/\`w3\` naming, so they need a custom AWQ mapping (and the standard Mixtral-style attention-only SmoothQuant mapping); happy to file that as a follow-up if useful. TEST PLAN: - \`ruff check\` / \`ruff format --check\` clean on both touched files. - Verified existing tests for the SmoothQuant registry pass shape (single architecture additions in #2419 / #2609 / #2639 didn't require new unit tests beyond the registry entry). - Manual mapping verification against vLLM's Granite implementation confirms the layer-name regex covers every smoothing target. Diff: \`\`\`diff + "GraniteForCausalLM": default_mappings, # awq/mappings.py + "GraniteForCausalLM": DEFAULT_SMOOTHQUANT_MAPPINGS, # smoothquant/utils.py \`\`\` --------- Signed-off-by: HumphreySun98 <humphreysun98@gmail.com> Co-authored-by: Brian Dellabetta <brian-dellabetta@users.noreply.github.com>
SUMMARY:
Adds `GraniteForCausalLM` (IBM Granite dense decoder) to both the AWQ and SmoothQuant layer-mapping registries. Maps to the existing `default_mappings` (AWQ) / `DEFAULT_SMOOTHQUANT_MAPPINGS` (SmoothQuant), since the architecture uses standard Llama-style layer naming: `input_layernorm` / `post_attention_layernorm` (parametric RMSNorm) + `q_proj`/`k_proj`/`v_proj`/`o_proj` + `gate_proj`/`up_proj`/`down_proj`, with no QK-norm. No new mapping definition is needed.
This is part of issue #1442 (the keep-open "add additional model mappings" tracker), continuing the pattern established by recent PRs that registered single architectures using existing mapping shapes (#2419 for `GlmMoeDsa`, #2609 for Qwen MoE variants, #2639 for Cohere/DeepSeek-V3/Phi3).
Verified the architecture matches the `default` shape by reading vLLM's `vllm/model_executor/models/granite.py`: GraniteDecoderLayer uses `RMSNorm` for both `input_layernorm` and `post_attention_layernorm` (line 228 / 258 in current vLLM main), and `stacked_params_mapping` resolves `q_proj`/`k_proj`/`v_proj` and `gate_proj`/`up_proj` to the standard names, so the existing `re:.*input_layernorm$` → q/k/v_proj and `re:.*post_attention_layernorm$` → gate/up_proj smoothings apply without modification.
`GraniteMoeForCausalLM` and `GraniteMoeSharedForCausalLM` are deliberately not included here — those use the Mixtral-style `block_sparse_moe.experts.*.w1`/`w3` naming, so they need a custom AWQ mapping (and the standard Mixtral-style attention-only SmoothQuant mapping); happy to file that as a follow-up if useful.
TEST PLAN:
Diff:
```diff
```