Skip to content

Add GraniteForCausalLM mappings for AWQ and SmoothQuant - #2797

Merged
brian-dellabetta merged 4 commits into
vllm-project:mainfrom
HumphreySun98:add-granite-olmo-olmoe-mappings
Jul 7, 2026
Merged

brian-dellabetta merged 4 commits into
vllm-project:mainfrom
HumphreySun98:add-granite-olmo-olmoe-mappings

Conversation

@HumphreySun98

Copy link
Copy Markdown
Contributor

SUMMARY:
Adds `GraniteForCausalLM` (IBM Granite dense decoder) to both the AWQ and SmoothQuant layer-mapping registries. Maps to the existing `default_mappings` (AWQ) / `DEFAULT_SMOOTHQUANT_MAPPINGS` (SmoothQuant), since the architecture uses standard Llama-style layer naming: `input_layernorm` / `post_attention_layernorm` (parametric RMSNorm) + `q_proj`/`k_proj`/`v_proj`/`o_proj` + `gate_proj`/`up_proj`/`down_proj`, with no QK-norm. No new mapping definition is needed.

This is part of issue #1442 (the keep-open "add additional model mappings" tracker), continuing the pattern established by recent PRs that registered single architectures using existing mapping shapes (#2419 for `GlmMoeDsa`, #2609 for Qwen MoE variants, #2639 for Cohere/DeepSeek-V3/Phi3).

Verified the architecture matches the `default` shape by reading vLLM's `vllm/model_executor/models/granite.py`: GraniteDecoderLayer uses `RMSNorm` for both `input_layernorm` and `post_attention_layernorm` (line 228 / 258 in current vLLM main), and `stacked_params_mapping` resolves `q_proj`/`k_proj`/`v_proj` and `gate_proj`/`up_proj` to the standard names, so the existing `re:.*input_layernorm$` → q/k/v_proj and `re:.*post_attention_layernorm$` → gate/up_proj smoothings apply without modification.

`GraniteMoeForCausalLM` and `GraniteMoeSharedForCausalLM` are deliberately not included here — those use the Mixtral-style `block_sparse_moe.experts.*.w1`/`w3` naming, so they need a custom AWQ mapping (and the standard Mixtral-style attention-only SmoothQuant mapping); happy to file that as a follow-up if useful.

TEST PLAN:

Diff:
```diff

  • "GraniteForCausalLM": default_mappings, # awq/mappings.py
  • "GraniteForCausalLM": DEFAULT_SMOOTHQUANT_MAPPINGS, # smoothquant/utils.py
    ```

@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown

👋 Hi! Thank you for contributing to llm-compressor. Please add the ready label when the PR is ready for review.

Note: This is required to complete the testing suite, please only add the label once the PR is code complete and local testing has been performed.

@coderabbitai

coderabbitai Bot commented Jun 4, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 70422c00-969f-4902-a921-4415724165d1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

This PR registers GraniteForCausalLM in the AWQ and SmoothQuant mapping registries and adds SmoothQuant tests that verify the registry entry and the default regex mappings against a Granite model structure.

Changes

GraniteForCausalLM Model Support

Layer / File(s) Summary
Add GraniteForCausalLM to compression mapping registries
src/llmcompressor/modifiers/transform/awq/mappings.py, src/llmcompressor/modifiers/transform/smoothquant/utils.py
GraniteForCausalLM is added to AWQ_MAPPING_REGISTRY with default_mappings and to MAPPINGS_REGISTRY with DEFAULT_SMOOTHQUANT_MAPPINGS.
Validate GraniteForCausalLM SmoothQuant mappings
tests/llmcompressor/modifiers/transform/smoothquant/test_utils.py
Tests assert the SmoothQuant registry resolves GraniteForCausalLM to DEFAULT_SMOOTHQUANT_MAPPINGS and that the default regex patterns match modules in a Granite model instantiated on meta.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

Suggested reviewers: HDCharles, brian-dellabetta, kylesayrs, dsikka

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: adding GraniteForCausalLM mappings for AWQ and SmoothQuant.
Description check ✅ Passed The description is directly related to the changeset and accurately explains the GraniteForCausalLM mapping updates and rationale.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Warning

Review ran into problems

🔥 Problems

Linked repositories: Your configuration references 1 linked repositories, but your current plan allows 0. Analyzed ``, skipped vllm-project/compressed-tensors.


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@mergify mergify Bot added the two-reviews When a PR requires two reviews label Jun 4, 2026
@mergify

mergify Bot commented Jun 4, 2026 •

Copy link
Copy Markdown
Contributor

Merge Protections

🟢 Merge protection satisfied — ready to merge.

Show 1 satisfied protection

🟢 Require two reviews

PRs labelled "two-reviews" must have at least two approving reviews before merging.

  • #approved-reviews-by >= 2
  • #changes-requested-reviews-by = 0

@coderabbitai coderabbitai Bot added enhancement New feature or request awq For any issue / PR related to AWQ support smoothquant For any issue / PR related to SmoothQuant support transforms Related to transforms-based modifiers like SpinQuant and Quip and removed two-reviews When a PR requires two reviews labels Jun 4, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for the GraniteForCausalLM model by mapping it to default configurations in both the AWQ and SmoothQuant modifier utilities. There are no review comments, so I have no feedback to provide.

@mergify mergify Bot added the two-reviews When a PR requires two reviews label Jun 4, 2026
@HumphreySun98
HumphreySun98 force-pushed the add-granite-olmo-olmoe-mappings branch from fc5d3ec to a703007 Compare June 4, 2026 20:25
@brian-dellabetta brian-dellabetta added the requires-validate Indicates that a PR looks appropriate, but needs to be run before merging label Jun 4, 2026
HumphreySun98 added a commit to HumphreySun98/llm-compressor that referenced this pull request Jun 8, 2026
Mirrors the pattern from vllm-project#2800 / vllm-project#2797. Tiny config + torch.device('meta')
keeps the test under a unit-test budget; verifies DEFAULT_SMOOTHQUANT_MAPPINGS
regex matches StableLM's actual layer names (q/k/v_proj + gate/up/down_proj +
input/post_attention_layernorm). Addresses the requires-validate label.

Signed-off-by: HumphreySun98 <humphreysun98@gmail.com>
@HumphreySun98

Copy link
Copy Markdown
Contributor Author

Added registry + meta-device validation in 135ae6c to address requires-validate. Mirrors the test pattern from #2800: instantiate GraniteForCausalLM on the meta device with a tiny config and assert DEFAULT_SMOOTHQUANT_MAPPINGS' regex matches every expected attention + MLP module. No HF Hub downloads, no weight allocation, runs as @pytest.mark.unit.

@mergify

mergify Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @HumphreySun98.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase label Jun 17, 2026
@brian-dellabetta

Copy link
Copy Markdown
Contributor

Mapping Validation Report

PR: #2797
Validated by: Claude Code (automated validate-pr skill)

Summary

AWQ: 1/1 PASS | SmoothQuant: 1/1 PASS

Both GraniteForCausalLM mapping entries validated successfully on ibm-granite/granite-3.3-2b-instruct (2B params, 40 layers).

AWQ Results

Architecture Model Mapping Result Details
GraniteForCausalLM ibm-granite/granite-3.3-2b-instruct default_mappings ✅ PASS Grid search ran on all 40 layers. best_error improved across iterations (e.g. layer 0 input_layernorm: 4.395e-03 → 2.396e-03). W4A16 quantization applied to 280 parameters. Coherent generation output.

SmoothQuant Results

Architecture Model Mapping Result Details
GraniteForCausalLM ibm-granite/granite-3.3-2b-instruct DEFAULT_SMOOTHQUANT_MAPPINGS ✅ PASS Smoothed both input_layernorm and post_attention_layernorm on all 40 layers (layers 0–39). 41 sequential pipeline steps completed. W8A8 quantization applied to 280 parameters. Coherent generation output.

Notes

  • 40 AWQ mappings were skipped due to incompatible shapes (expected for default_mappings on architectures with GQA where num_key_value_heads < num_attention_heads) — this is normal behavior.
  • Generation quality was good for both quantization methods, producing coherent English text about software engineering topics.

@brian-dellabetta brian-dellabetta left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, validation is passing. Please resolve conflicts

@kylesayrs

Copy link
Copy Markdown
Collaborator

@coderabbitai resolve merge conflict

@coderabbitai

coderabbitai Bot commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

❌ An unexpected error occurred while resolving merge conflicts:

Not Found - https://docs.github.com/rest/git/refs#get-a-reference

@mergify

mergify Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

The quality checks have failed. Please run make style and make quality under
the root directory to adddress the lint failures. You will need to install the
dev optional install to get the required linting packages:
https://github.com/vllm-project/llm-compressor/blob/main/CONTRIBUTING.md

@kylesayrs

Copy link
Copy Markdown
Collaborator

https://github.com/coderabbitai resolve merge conflict

Maps the standard IBM Granite dense decoder architecture
('GraniteForCausalLM') to the existing default mappings in both
registries. The architecture uses the Llama-style layer naming
(input_layernorm/post_attention_layernorm RMSNorm + q/k/v/o_proj +
gate/up/down_proj) with no QK-norm, so the existing default mappings
apply directly without a new mapping definition.

Part of vllm-project#1442.

Signed-off-by: HumphreySun98 <humphreysun98@gmail.com>
Addresses the requires-validate label and matches the test pattern
established for the MoE batch in vllm-project#2800:

- test_granite_for_causal_lm_in_registry: asserts the new entry is
  registered and resolves to DEFAULT_SMOOTHQUANT_MAPPINGS.
- test_granite_default_mapping_regex_matches_real_module_tree:
  instantiates GraniteForCausalLM on the meta device with a tiny
  config and asserts every re-prefixed pattern in
  DEFAULT_SMOOTHQUANT_MAPPINGS matches at least one module. No HF Hub
  downloads, no weight allocation.

Signed-off-by: HumphreySun98 <humphreysun98@gmail.com>
@HumphreySun98
HumphreySun98 force-pushed the add-granite-olmo-olmoe-mappings branch from 135ae6c to 42c2630 Compare July 6, 2026 18:35
@mergify mergify Bot removed the quality-failed label Jul 6, 2026
@HumphreySun98

Copy link
Copy Markdown
Contributor Author

Rebased onto latest main and resolved the conflict manually (the @coderabbitai resolve was erroring with Not Found).

The conflict was in tests/llmcompressor/modifiers/transform/smoothquant/test_utils.py — main added the Qwen3.5 MoE dynamic-mapping tests in the same region where this branch adds the GraniteForCausalLM registry/meta-device tests. Resolved as a union: both test sets are kept.

make style and make quality both pass now (the earlier quality failure was just the unresolved conflict markers). @brian-dellabetta @kylesayrs — should be ready for a second look.

@coderabbitai

coderabbitai Bot commented Jul 6, 2026 •

Copy link
Copy Markdown
Contributor

@HumphreySun98 Thanks for the update — glad the conflict resolution worked out and that make style/make quality are passing. I'll go ahead and review the current state of the branch to confirm the merge was resolved cleanly.

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@brian-dellabetta brian-dellabetta added the ready When a PR is ready for full CI testing before merge label Jul 7, 2026
@brian-dellabetta
brian-dellabetta merged commit 80a1e4a into vllm-project:main Jul 7, 2026
9 checks passed
HDCharles pushed a commit that referenced this pull request Aug 10, 2026
SUMMARY:
Adds \`GraniteForCausalLM\` (IBM Granite dense decoder) to both the AWQ
and SmoothQuant layer-mapping registries. Maps to the existing
\`default_mappings\` (AWQ) / \`DEFAULT_SMOOTHQUANT_MAPPINGS\`
(SmoothQuant), since the architecture uses standard Llama-style layer
naming: \`input_layernorm\` / \`post_attention_layernorm\` (parametric
RMSNorm) + \`q_proj\`/\`k_proj\`/\`v_proj\`/\`o_proj\` +
\`gate_proj\`/\`up_proj\`/\`down_proj\`, with no QK-norm. No new mapping
definition is needed.

This is part of issue #1442 (the keep-open \"add additional model
mappings\" tracker), continuing the pattern established by recent PRs
that registered single architectures using existing mapping shapes
(#2419 for \`GlmMoeDsa\`, #2609 for Qwen MoE variants, #2639 for
Cohere/DeepSeek-V3/Phi3).

Verified the architecture matches the \`default\` shape by reading
vLLM's \`vllm/model_executor/models/granite.py\`: GraniteDecoderLayer
uses \`RMSNorm\` for both \`input_layernorm\` and
\`post_attention_layernorm\` (line 228 / 258 in current vLLM main), and
\`stacked_params_mapping\` resolves \`q_proj\`/\`k_proj\`/\`v_proj\` and
\`gate_proj\`/\`up_proj\` to the standard names, so the existing
\`re:.*input_layernorm$\` → q/k/v_proj and
\`re:.*post_attention_layernorm$\` → gate/up_proj smoothings apply
without modification.

\`GraniteMoeForCausalLM\` and \`GraniteMoeSharedForCausalLM\` are
deliberately **not** included here — those use the Mixtral-style
\`block_sparse_moe.experts.*.w1\`/\`w3\` naming, so they need a custom
AWQ mapping (and the standard Mixtral-style attention-only SmoothQuant
mapping); happy to file that as a follow-up if useful.

TEST PLAN:
- \`ruff check\` / \`ruff format --check\` clean on both touched files.
- Verified existing tests for the SmoothQuant registry pass shape
(single architecture additions in #2419 / #2609 / #2639 didn't require
new unit tests beyond the registry entry).
- Manual mapping verification against vLLM's Granite implementation
confirms the layer-name regex covers every smoothing target.

Diff:
\`\`\`diff
+ "GraniteForCausalLM": default_mappings, # awq/mappings.py
+ "GraniteForCausalLM": DEFAULT_SMOOTHQUANT_MAPPINGS, #
smoothquant/utils.py
\`\`\`

---------

Signed-off-by: HumphreySun98 <humphreysun98@gmail.com>
Co-authored-by: Brian Dellabetta <brian-dellabetta@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

awq For any issue / PR related to AWQ support enhancement New feature or request ready When a PR is ready for full CI testing before merge requires-validate Indicates that a PR looks appropriate, but needs to be run before merging smoothquant For any issue / PR related to SmoothQuant support transforms Related to transforms-based modifiers like SpinQuant and Quip two-reviews When a PR requires two reviews

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants