Skip to content

feat: improve llava next pooling for granite vision#3255

Open
drbh wants to merge 2 commits intomainfrom
support-granite-vision
Open

feat: improve llava next pooling for granite vision#3255
drbh wants to merge 2 commits intomainfrom
support-granite-vision

Conversation

@drbh
Copy link
Collaborator

@drbh drbh commented Jun 4, 2025

This PR improve LlavaNextForConditionalGeneration to handle models that specify pooled vision_feature_layer.

more concretely the ibm-granite/granite-vision-3.2-2b model specifies

    "vision_feature_layer": [
        -24,
        -20,
        -12,
        -1
    ],

and requires that those layers are pooled. This change allows vision_feature_layer to be a list an iterates over the layers.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant