Transfer from pg-team/pg-vl-2b-hf

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,100 @@
+<p align="center">
+  <img src="assets/logo.png" width="160" />
+</p>
+
+<h2 align="center">PenguinVL</h2>
+<h4 align="center">
+Exploring the Efficiency Limits of VLM with LLM-based Vision Encoders
+</h4>
+
+---
+
+## 📰 News
+
+* **2025.03** — PenguinVL-Encoder now available for general use.
+* **2025.03** — Released PenguinVL-2B, PenguinVL-8B.
+
+---
+
+## 🌟 Model Overview
+
+PenguinVL is a compact Vision-Language Model, designed to explore the efficiency limits of small-scale VLMs.
+
+Unlike most existing VLMs that rely on contrastive-pretrained vision encoders (e.g., CLIP/SigLIP), PG-VL initializes its vision encoder directly from a **text-only LLM**. This design avoids the objective mismatch between contrastive learning and autoregressive language modeling, enabling tighter alignment between visual representations and the language backbone.
+
+### Key Characteristics
+
+- 🧠 **LLM-based Vision Encoder**  
+  The vision encoder is adapted from a pretrained text LLM (Qwen3-0.6B), modified with bidirectional attention and 2D-RoPE for spatial modeling.  
+  This provides strong semantic priors and native compatibility with the downstream LLM.
+
+- 🎥 **Efficient Video Understanding**  
+  A Temporal Redundancy-Aware (TRA) token compression strategy dynamically allocates token budgets across frames, enabling long-video reasoning within a limited context window.
+
+- 🏗 Unified Architecture  
+  The model consists of:
+  1. LLM-initialized vision encoder  
+  2. Lightweight MLP projector  
+  3. Qwen3 language backbone  
+
+- 📊 Compact but Strong  
+  At 2B scale, PG-VL achieves competitive performance across image, document, OCR, math, and video benchmarks while remaining deployment-friendly.
+
+---
+
+## 🧪 Quick Start — Transformers Inference
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoProcessor
+
+model_name = "pg-team/pg-vl-2b-hf"
+
+model = AutoModelForCausalLM.from_pretrained(
+    model_name,
+    trust_remote_code=True,
+    device_map="auto",
+    torch_dtype=torch.bfloat16,
+)
+
+processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)
+
+# Example: Image + Text
+inputs = processor(
+    conversation=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {
+            "role": "user",
+            "content": [
+                {"type": "image", "image": {"image_path": "assets/example.jpg"}},
+                {"type": "text", "text": "Describe this image."}
+            ],
+        },
+    ],
+    return_tensors="pt",
+)
+
+inputs = {k: v.to("cuda") for k, v in inputs.items() if isinstance(v, torch.Tensor)}
+
+output_ids = model.generate(**inputs, max_new_tokens=128)
+response = processor.decode(output_ids[0], skip_special_tokens=True)
+
+print(response)
+```
+
+## 🌎 Model Zoo
+| Model                | Base Model   | HF Link                                                      |
+| -------------------- | ------------ | ------------------------------------------------------------ |
+| PenguinVL-8B         | Qwen3-8B     | [pg-team/pg-vl-8b-hf](https://huggingface.co/pg-team/pg-vl-8b-hf) |
+| PenguinVL-2B         | Qwen3-1.7B   | [pg-team/pg-vl-2b-hf](https://huggingface.co/pg-team/pg-vl-2b-hf) |
+| PenguinVL-Encoder    | Qwen3-0.6B   | [pg-team/pg-vision-encoder](https://huggingface.co/pg-team/pg-vision-encoder) |
+
+## 🚀 Main Results
+xxx
+
+## Citation
+
+If you find PenguinVL useful for your research and applications, please cite using this BibTeX:
+```bibtex
+...
+```

added_tokens.json

@@ -0,0 +1,34 @@
+{
+  "</think>": 151668,
+  "</tool_call>": 151658,
+  "</tool_response>": 151666,
+  "<image>": 151669,
+  "<think>": 151667,
+  "<tool_call>": 151657,
+  "<tool_response>": 151665,
+  "<|audio_end|>": 151674,
+  "<|audio_start|>": 151673,
+  "<|audio|>": 151672,
+  "<|box_end|>": 151649,
+  "<|box_start|>": 151648,
+  "<|endoftext|>": 151643,
+  "<|file_sep|>": 151664,
+  "<|fim_middle|>": 151660,
+  "<|fim_pad|>": 151662,
+  "<|fim_prefix|>": 151659,
+  "<|fim_suffix|>": 151661,
+  "<|im_end|>": 151645,
+  "<|im_start|>": 151644,
+  "<|image_pad|>": 151655,
+  "<|object_ref_end|>": 151647,
+  "<|object_ref_start|>": 151646,
+  "<|quad_end|>": 151651,
+  "<|quad_start|>": 151650,
+  "<|repo_name|>": 151663,
+  "<|stream_end|>": 151671,
+  "<|stream_start|>": 151670,
+  "<|video_pad|>": 151656,
+  "<|vision_end|>": 151653,
+  "<|vision_pad|>": 151654,
+  "<|vision_start|>": 151652
+}

chat_template.json

@@ -0,0 +1,3 @@
+{
+  "chat_template": "\n{%- set identifier = 'im' %}\n{% for message in messages %}\n    {% if message['role'] == 'stream' %}\n        {% set identifier = 'stream' %}\n    {% else %}\n        {% set identifier = 'im' %}\n    {% endif %}\n    {% if message['role'] is not none %}\n        {{- '<|' + identifier + '_start|>' + message['role'] + '\n' -}}\n    {% endif %}\n    {% if message['content'] is string %}\n        {{- message['content'] + '<|' + identifier + '_end|>\n' -}}\n    {% else %}\n        {% for content in message['content'] %}\n            {% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}\n                {% if 'time' in content %}\n                    {{- 'Time ' + content['time'] | round(1) | string + 's: ' -}}\n                {% endif %}\n                {{- image_token + '\n' -}}\n            {% elif content['type'] == 'video' or 'video' in content or 'video_url' in content %}\n                {% for i in range(content['num_frames']) %}\n                    {% if 'timestamps' in content and content['timestamps']|length > 0 %}\n                        {{- 'Time ' + content['timestamps'][i] | round(1) | string + 's:' -}}\n                    {% endif %}\n                    {% if i < content['num_frames'] - 1 %}\n                        {{- image_token + ',' -}}\n                    {% else %}\n                        {{- image_token + '\n' -}}\n                    {% endif %}\n                {% endfor %}\n            {% elif content['type'] == 'text' or 'text' in content %}\n                {{- content['text'] -}}\n            {% endif %}\n        {% endfor %}\n        {% if message['role'] is not none %}\n            {{- '<|' + identifier + '_end|>\n' -}}\n        {% endif %}\n    {% endif %}\n{% endfor %}\n{% if add_generation_prompt %}\n    {{- '<|im_start|>assistant\n' -}}\n    {% if not add_think_prompt %}\n        {{- '<think>\n\n</think>\n\n' -}}\n    {% endif %}\n{% endif %}\n"
+}

config.json

@@ -0,0 +1,62 @@
+{
+  "architectures": [
+    "PenguinVLQwen3ForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_penguinvl.PenguinVLQwen3Config",
+    "AutoModelForCausalLM": "modeling_penguinvl_qwen3.PenguinVLQwen3ForCausalLM"
+  },
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 151643,
+  "eos_token_id": 151645,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "image_aspect_ratio": "square",
+  "image_token_index": 151669,
+  "initializer_range": 0.02,
+  "intermediate_size": 6144,
+  "loss_reduction_scope": "batch",
+  "max_frames": 180,
+  "max_position_embeddings": 40960,
+  "max_window_layers": 28,
+  "model_type": "penguinvl_qwen3",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "num_key_value_heads": 8,
+  "rms_norm_eps": 1e-06,
+  "rope_scaling": null,
+  "rope_theta": 1000000,
+  "sliding_window": null,
+  "tie_word_embeddings": true,
+  "tokenizer_model_max_length": 32768,
+  "tokenizer_padding_side": "right",
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.51.3",
+  "use_cache": true,
+  "use_sliding_window": false,
+  "vision_encoder": "pg-team/pg-vision-encoder",
+  "vision_hidden_size": 1024,
+  "vision_projector_type": "mlp2x_gelu",
+  "vocab_size": 151936,
+  "vision_encoder_config": {
+    "head_dim": 128,
+    "hidden_act": "silu",
+    "hidden_size": 1024,
+    "initializer_range": 0.02,
+    "intermediate_size": 3072,
+    "layer_norm_eps": 1e-06,
+    "max_window_layers": 28,
+    "num_attention_heads": 16,
+    "num_channels": 3,
+    "num_hidden_layers": 28,
+    "num_key_value_heads": 8,
+    "patch_size": 14,
+    "rms_norm_eps": 1e-06,
+    "rope_scaling": null,
+    "rope_theta": 1000000,
+    "sliding_window": null,
+    "torch_dtype": "bfloat16"
+  }
+}

configuration_penguinvl.py

@@ -0,0 +1,48 @@
+"""PenguinVL model configuration."""
+
+import importlib.util
+import os.path as osp
+from typing import Optional, Dict, Any
+
+from transformers import PretrainedConfig, Qwen3Config
+
+try:
+    from .configuration_penguinvl_encoder import PenguinVLVisionEncoderConfig
+except ModuleNotFoundError:
+    spec = importlib.util.spec_from_file_location(
+        "configuration_penguinvl_encoder",
+        osp.join(osp.dirname(__file__), "configuration_penguinvl_encoder.py"),
+    )
+    configuration_penguinvl_encoder = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(configuration_penguinvl_encoder)
+    PenguinVLVisionEncoderConfig = getattr(
+        configuration_penguinvl_encoder,
+        "PenguinVLVisionEncoderConfig",
+    )
+
+
+class PenguinVLQwen3Config(Qwen3Config):
+
+    model_type = "penguinvl_qwen3"
+    sub_configs = {"vision_encoder_config": PenguinVLVisionEncoderConfig}
+
+    def __init__(
+        self,
+        vision_encoder: Optional[str] = None,
+        vision_encoder_config: Dict[str, Any] = {},
+        vision_projector_type: str = "mlp2x_gelu",
+        use_token_compression: bool = True,
+        image_token_index: int = -1,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+        self.model_type = "penguinvl_qwen3"
+
+        self.vision_encoder = vision_encoder
+        if vision_encoder_config is not None and not isinstance(vision_encoder_config, PretrainedConfig):
+            vision_encoder_config = PenguinVLVisionEncoderConfig(**vision_encoder_config)
+        self.vision_encoder_config = vision_encoder_config
+
+        self.vision_projector_type = vision_projector_type
+        self.use_token_compression = use_token_compression
+        self.image_token_index = image_token_index

configuration_penguinvl_encoder.py

@@ -0,0 +1,33 @@
+"""PenguinVL vision encoder model configuration."""
+
+from transformers import Qwen3Config
+
+
+class PenguinVLVisionEncoderConfig(Qwen3Config):
+
+    model_type = "penguinvl_vision_encoder"
+
+    def __init__(
+        self,
+        hidden_size=1536,
+        intermediate_size=8960,
+        num_hidden_layers=12,
+        num_attention_heads=12,
+        num_channels=3,
+        patch_size=14,
+        layer_norm_eps=1e-6,
+        attention_dropout=0.0,
+        num_key_value_heads=2,
+        **kwargs,
+    ):
+        super().__init__(**kwargs)
+
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.num_channels = num_channels
+        self.patch_size = patch_size
+        self.attention_dropout = attention_dropout
+        self.num_key_value_heads = num_key_value_heads
+        self.layer_norm_eps = layer_norm_eps

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "bos_token_id": 151643,
+  "do_sample": true,
+  "eos_token_id": [
+    151645,
+    151643
+  ],
+  "pad_token_id": 151643,
+  "temperature": 0.6,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "4.51.3"
+}

image_processing_penguinvl.py

@@ -0,0 +1,548 @@
+# Adopted from https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_vl/image_processing_qwen2_vl.py.
+# Below is the original copyright:
+# Copyright 2024 The Qwen team, Alibaba Group and the HuggingFace Inc. team. All rights reserved.
+#
+# This code is based on EleutherAI's GPT-NeoX library and the GPT-NeoX
+# and OPT implementations in this library. It has been modified from its
+# original forms to accommodate minor architectural differences compared
+# to GPT-NeoX and OPT used by the Meta AI team that trained the model.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Image processor class for PenguinVL."""
+
+import math
+from typing import Dict, List, Optional, Union
+
+import numpy as np
+
+import torch
+from transformers.image_processing_utils import BaseImageProcessor, BatchFeature
+from transformers.image_utils import ImageInput
+from transformers.image_transforms import (
+    convert_to_rgb,
+    resize,
+    to_channel_dimension_format,
+)
+from transformers.image_utils import (
+    OPENAI_CLIP_MEAN,
+    OPENAI_CLIP_STD,
+    ChannelDimension,
+    ImageInput,
+    PILImageResampling,
+    get_image_size,
+    infer_channel_dimension_format,
+    is_scaled_image,
+    is_valid_image,
+    make_list_of_images,
+    to_numpy_array,
+)
+try:
+    from transformers.image_utils import VideoInput
+except:
+    from transformers.video_utils import VideoInput
+from transformers.utils import TensorType, is_vision_available, logging
+
+
+logger = logging.get_logger(__name__)
+
+
+if is_vision_available():
+    from PIL import Image
+
+
+def is_valid_video(video) -> bool:
+    if isinstance(video, (list, tuple)):
+        return all(is_valid_image(frame) for frame in video)
+    elif isinstance(video, np.ndarray):
+        return video.ndim == 4
+    elif isinstance(video, torch.Tensor):
+        return video.ndim == 4
+    return False
+
+
+def make_batched_images(images) -> List[List[ImageInput]]:
+    """
+    Normalize visual inputs to ``List[List[ImageInput]]`` – a list of *clips*,
+    where each clip is a list of frames.
+
+    Supported input formats::
+
+        Nested clips : [[image], [f1, f2, ...], ...]   → returned as-is
+        Flat frames  : [f1, f2, ...]                    → [[f1, f2, ...]]
+        Single image : image                            → [[image]]
+
+    Returns:
+        List of clips, where each clip is a list of valid images / frames.
+    """
+    if isinstance(images, (list, tuple)) and len(images) > 0:
+        if isinstance(images[0], (list, tuple)):
+            return [list(clip) for clip in images]
+        if all(is_valid_image(f) for f in images):
+            return [list(images)]
+    if is_valid_image(images):
+        return [[images]]
+    raise ValueError(f"Could not make batched images from {images}")
+
+
+def simple_batched_resize(
+    images, 
+    factor: int = 28, 
+    min_tokens: int = 4 * 4, 
+    max_tokens: int = 16384, 
+    input_data_format: str = None, 
+    frame_types=None
+):
+    """
+    Compute per-frame target (h, w) for a video frame list under a token budget (key/intermediate may differ).
+
+    Uses the Temporal Redundancy-Aware (TRA) token compression strategy: key and intermediate frames
+    can have different target areas (e.g. 1:16 ratio when compressing) to stay within max_tokens.
+
+    Args:
+        images: List of video frames (each PIL Image or ndarray).
+        factor: Alignment granularity (height and width are multiples of factor), default 28.
+        min_tokens: Minimum tokens per frame (used to derive min_pixels), default 16.
+        max_tokens: Token cap for total pixel budget, default 16384.
+        input_data_format: Channel format when not PIL, e.g. "channels_first".
+        frame_types: Per-frame type list, 0=key, 1=intermediate; None means all key.
+
+    Returns:
+        image_sizes: List of (h, w) per frame, one-to-one with images.
+    """
+    min_pixels = min_tokens * factor * factor * 1.5
+    max_pixels = max_tokens * factor * factor * 0.95
+
+    # --- Base info ---
+    first_image = images[0]
+    if isinstance(first_image, Image.Image):
+        width, height = first_image.size
+    else:
+        height, width = get_image_size(first_image, channel_dim=input_data_format)
+
+    aspect_ratio = height / width
+    raw_area = height * width
+
+    num_frames = len(images)
+    if frame_types is not None:
+        ft_list = frame_types.tolist() if hasattr(frame_types, 'tolist') else frame_types
+        num_intermediate = ft_list.count(1)
+        num_key = ft_list.count(0)
+    else:
+        num_key = num_frames
+        num_intermediate = 0
+        ft_list = [0] * num_frames
+
+    def get_dims_from_area(target_area, ar, fac):
+        """Compute aligned (h, w) from target area and aspect ratio; area = w²·ar => w = sqrt(area/ar)."""
+        w_new = math.sqrt(target_area / ar)
+        h_new = w_new * ar
+
+        h_bar = round(h_new / fac) * fac
+        w_bar = round(w_new / fac) * fac
+        h_bar = max(h_bar, fac)
+        w_bar = max(w_bar, fac)
+
+        return h_bar, w_bar
+
+    # --- Stage 1: No-downscale check ---
+    # If total pixels within budget, keep original size for both key and intermediate frames.
+    total_raw_pixels = num_frames * raw_area
+    target_key_area = raw_area
+    target_intermediate_area = raw_area
+
+    if total_raw_pixels > max_pixels:
+        # --- Stage 2: Sync compression ---
+        # Over budget: compress with 1:16 area ratio, intermediate_area = key_area / 16.
+        # Constraint: N_key·A_key + N_intermediate·(A_key/16) = max_pixels => A_key = max_pixels / (N_key + N_intermediate/16).
+        effective_count = num_key + (num_intermediate / 16.0)
+        calc_key_area = max_pixels / effective_count
+        calc_intermediate_area = calc_key_area / 16.0
+
+        # --- Stage 3: Intermediate-frame floor ---
+        # If computed intermediate area is below min_pixels, pin intermediate to min_pixels and give remaining budget to key.
+        if calc_intermediate_area >= min_pixels:
+            target_key_area = calc_key_area
+            target_intermediate_area = calc_intermediate_area
+        else:
+            target_intermediate_area = min_pixels
+            pixels_taken_by_intermediate = num_intermediate * min_pixels
+            remaining_for_key = max_pixels - pixels_taken_by_intermediate
+            target_key_area = remaining_for_key / num_key
+
+        # --- Stage 4: Key-frame hard floor ---
+        if target_key_area < min_pixels:
+            target_key_area = min_pixels
+
+    # --- Area to aligned dimensions ---
+    k_h, k_w = get_dims_from_area(target_key_area, aspect_ratio, factor)
+    if num_intermediate > 0:
+        i_h, i_w = get_dims_from_area(target_intermediate_area, aspect_ratio, factor)
+    else:
+        i_h, i_w = 0, 0
+
+    def ensure_min_hw(h, w, min_p, raw_ar):
+        """If area still below min_pixels after alignment (rounding), recompute from min area and align upward."""
+        if h * w < min_p:
+            w = math.sqrt(min_p / raw_ar)
+            h = w * raw_ar
+            h = math.ceil(h / factor) * factor
+            w = math.ceil(w / factor) * factor
+        return h, w
+
+    k_h, k_w = ensure_min_hw(k_h, k_w, min_pixels, aspect_ratio)
+    if num_intermediate > 0:
+        i_h, i_w = ensure_min_hw(i_h, i_w, min_pixels, aspect_ratio)
+
+    image_sizes = [
+        (i_h, i_w) if ft_list[i] == 1 else (k_h, k_w)
+        for i in range(num_frames)
+    ]
+    return image_sizes
+
+
+class PenguinVLImageProcessor(BaseImageProcessor):
+    r"""
+    Constructs a PenguinVL image processor that dynamically resizes images based on the original images.
+
+    Args:
+        do_resize (`bool`, *optional*, defaults to `True`):
+            Whether to resize the image's (height, width) dimensions.
+        resample (`PILImageResampling`, *optional*, defaults to `Resampling.BICUBIC`):
+            Resampling filter to use when resizing the image.
+        do_rescale (`bool`, *optional*, defaults to `True`):
+            Whether to rescale the image by the specified scale `rescale_factor`.
+        rescale_factor (`int` or `float`, *optional*, defaults to `1/255`):
+            Scale factor to use if rescaling the image.
+        do_normalize (`bool`, *optional*, defaults to `True`):
+            Whether to normalize the image.
+        image_mean (`float` or `List[float]`, *optional*, defaults to `[0.48145466, 0.4578275, 0.40821073]`):
+            Mean to use if normalizing the image. This is a float or list of floats for each channel in the image.
+        image_std (`float` or `List[float]`, *optional*, defaults to `[0.26862954, 0.26130258, 0.27577711]`):
+            Standard deviation to use if normalizing the image. This is a float or list of floats for each channel in the image.
+        do_convert_rgb (`bool`, *optional*, defaults to `True`):
+            Whether to convert the image to RGB.
+        min_pixels (`int`, *optional*, defaults to `56 * 56`):
+            The min pixels of the image to resize the image.
+        max_pixels (`int`, *optional*, defaults to `28 * 28 * 1280`):
+            The max pixels of the image to resize the image.
+        patch_size (`int`, *optional*, defaults to 14):
+            The spacial patch size of the vision encoder.
+    """
+
+    model_input_names = ["pixel_values", "grid_sizes", "merge_sizes"]
+
+    def __init__(
+        self,
+        do_resize: bool = True,
+        resample: PILImageResampling = PILImageResampling.BICUBIC,
+        do_rescale: bool = True,
+        rescale_factor: Union[int, float] = 1 / 255,
+        do_normalize: bool = True,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        do_convert_rgb: bool = True,
+        min_tokens: int = 4 * 4,
+        max_tokens: int = 16384,
+        patch_size: int = 14,
+        **kwargs,
+    ) -> None:
+        super().__init__(**kwargs)
+        self.do_resize = do_resize
+        self.resample = resample
+        self.do_rescale = do_rescale
+        self.rescale_factor = rescale_factor
+        self.do_normalize = do_normalize
+        self.image_mean = image_mean if image_mean is not None else OPENAI_CLIP_MEAN
+        self.image_std = image_std if image_std is not None else OPENAI_CLIP_STD
+        self.min_tokens = min_tokens
+        self.max_tokens = max_tokens
+        self.patch_size = patch_size
+        self.do_convert_rgb = do_convert_rgb
+
+    def _allocate_token_budget(self, clips, clip_merge_sizes, input_data_format):
+        """Distribute self.max_tokens across clips proportionally to their raw token counts."""
+        clip_raw_tokens = []
+        for clip, ms in zip(clips, clip_merge_sizes):
+            first_frame = clip[0]
+            if isinstance(first_frame, Image.Image):
+                w, h = first_frame.size
+            else:
+                h, w = get_image_size(first_frame, channel_dim=input_data_format)
+            factor = self.patch_size * ms
+            clip_raw_tokens.append(len(clip) * h * w / (factor * factor))
+
+        total_raw_tokens = sum(clip_raw_tokens)
+        if total_raw_tokens <= self.max_tokens:
+            return [self.max_tokens] * len(clips)
+
+        return [
+            max(self.min_tokens * len(clip), raw * self.max_tokens / total_raw_tokens)
+            for clip, raw in zip(clips, clip_raw_tokens)
+        ]
+
+    def _preprocess(
+        self,
+        images: Union[ImageInput, VideoInput],
+        target_size: List[int],
+        merge_size: int = 1,
+        do_resize: bool = None,
+        resample: PILImageResampling = None,
+        do_rescale: bool = None,
+        rescale_factor: float = None,
+        do_normalize: bool = None,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        do_convert_rgb: bool = None,
+        data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
+        input_data_format: Optional[Union[str, ChannelDimension]] = None,
+    ):
+        """
+        Preprocess an image or batch of images. Copy of the `preprocess` method from `CLIPImageProcessor`.
+
+        Args:
+            images (`ImageInput`):
+                Image or batch of images to preprocess. Expects pixel values ranging from 0 to 255. If pixel values range from 0 to 1, set `do_rescale=False`.
+            target_size (`List[int]`):
+                The target size to resize the image to. Should be a list of two integers: [target_height, target_width].
+            merge_size (`int`, *optional*, defaults to `1`):
+                The merge size after the vision encoder.
+            do_resize (`bool`, *optional*, defaults to `self.do_resize`):
+                Whether to resize the image.
+            resample (`PILImageResampling`, *optional*, defaults to `self.resample`):
+                Resampling filter to use if resizing the image. This can be one of the `PILImageResampling` enums.
+            do_rescale (`bool`, *optional*, defaults to `self.do_rescale`):
+                Whether to rescale the image.
+            rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`):
+                Scale factor to use if rescaling the image.
+            do_normalize (`bool`, *optional*, defaults to `self.do_normalize`):
+                Whether to normalize the image.
+            image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`):
+                Mean to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image.
+            image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`):
+                Standard deviation to use if normalizing the image. Can be a float or a list of floats corresponding to the number of channels in the image.
+            do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`):
+                Whether to convert the image to RGB.
+            data_format (`ChannelDimension`, *optional*, defaults to `ChannelDimension.FIRST`):
+                The channel dimension format for the output image. Can be one of:
+                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
+                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
+                - Unset: Use the channel dimension format of the input image.
+            input_data_format (`ChannelDimension` or `str`, *optional*):
+                The channel dimension format for the input image. Can be one of:
+                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
+                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
+                - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.   - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
+        """
+        images = make_list_of_images(images)
+
+        if do_convert_rgb:
+            images = [convert_to_rgb(image) for image in images]
+
+        # All transformations expect numpy arrays.
+        images = [to_numpy_array(image) for image in images]
+
+        if is_scaled_image(images[0]) and do_rescale:
+            logger.warning_once(
+                "It looks like you are trying to rescale already rescaled images. If the input"
+                " images have pixel values between 0 and 1, set `do_rescale=False` to avoid rescaling them again."
+            )
+        if input_data_format is None:
+            # We assume that all images have the same channel dimension format.
+            input_data_format = infer_channel_dimension_format(images[0])
+
+        height, width = get_image_size(images[0], channel_dim=input_data_format)
+        resized_height, resized_width = height, width
+        processed_images = []
+        for image in images:
+            if do_resize:
+                resized_height, resized_width = target_size
+                image = resize(
+                    image, size=(resized_height, resized_width), resample=resample, input_data_format=input_data_format
+                )
+
+            if do_rescale:
+                image = self.rescale(image, scale=rescale_factor, input_data_format=input_data_format)
+
+            if do_normalize:
+                image = self.normalize(
+                    image=image, mean=image_mean, std=image_std, input_data_format=input_data_format
+                )
+
+            image = to_channel_dimension_format(image, data_format, input_channel_dim=input_data_format)
+            processed_images.append(image)
+
+        patches = np.array(processed_images)
+        if data_format == ChannelDimension.LAST:
+            patches = patches.transpose(0, 3, 1, 2)
+        t = patches.shape[0]
+        channel = patches.shape[1]
+        grid_h, grid_w = resized_height // self.patch_size, resized_width // self.patch_size
+        patches = patches.reshape(
+            t,
+            channel,
+            grid_h // merge_size,
+            merge_size,
+            self.patch_size,
+            grid_w // merge_size,
+            merge_size,
+            self.patch_size,
+        )
+        patches = patches.transpose(0, 2, 5, 3, 6, 1, 4, 7)
+        flatten_patches = patches.reshape(
+            t * grid_h * grid_w, channel * self.patch_size * self.patch_size
+        )
+
+        return flatten_patches, (t, grid_h, grid_w)
+
+    def preprocess(
+        self,
+        images: ImageInput,
+        do_resize: bool = None,
+        resample: PILImageResampling = None,
+        do_rescale: bool = None,
+        rescale_factor: float = None,
+        do_normalize: bool = None,
+        image_mean: Optional[Union[float, List[float]]] = None,
+        image_std: Optional[Union[float, List[float]]] = None,
+        do_convert_rgb: bool = None,
+        merge_size: Optional[Union[int, List[int]]] = None,
+        frame_types: Optional[Union[int, List[int]]] = None,
+        return_tensors: Optional[Union[str, TensorType]] = None,
+        data_format: Optional[ChannelDimension] = ChannelDimension.FIRST,
+        input_data_format: Optional[Union[str, ChannelDimension]] = None,
+    ):
+        """
+        Args:
+            images (`ImageInput`):
+                Image to preprocess. Expects a single or batch of images with pixel values ranging from 0 to 255. If
+                passing in images with pixel values between 0 and 1, set `do_rescale=False`.
+            do_resize (`bool`, *optional*, defaults to `self.do_resize`):
+                Whether to resize the image.
+            resample (`int`, *optional*, defaults to `self.resample`):
+                Resampling filter to use if resizing the image. This can be one of the enum `PILImageResampling`. Only
+                has an effect if `do_resize` is set to `True`.
+            do_rescale (`bool`, *optional*, defaults to `self.do_rescale`):
+                Whether to rescale the image.
+            rescale_factor (`float`, *optional*, defaults to `self.rescale_factor`):
+                Rescale factor to rescale the image by if `do_rescale` is set to `True`.
+            do_normalize (`bool`, *optional*, defaults to `self.do_normalize`):
+                Whether to normalize the image.
+            image_mean (`float` or `List[float]`, *optional*, defaults to `self.image_mean`):
+                Image mean to use for normalization. Only has an effect if `do_normalize` is set to `True`.
+            image_std (`float` or `List[float]`, *optional*, defaults to `self.image_std`):
+                Image standard deviation to use for normalization. Only has an effect if `do_normalize` is set to
+                `True`.
+            do_convert_rgb (`bool`, *optional*, defaults to `self.do_convert_rgb`):
+                Whether to convert the image to RGB.
+            return_tensors (`str` or `TensorType`, *optional*):
+                The type of tensors to return. Can be one of:
+                - Unset: Return a list of `np.ndarray`.
+                - `TensorType.TENSORFLOW` or `'tf'`: Return a batch of type `tf.Tensor`.
+                - `TensorType.PYTORCH` or `'pt'`: Return a batch of type `torch.Tensor`.
+                - `TensorType.NUMPY` or `'np'`: Return a batch of type `np.ndarray`.
+                - `TensorType.JAX` or `'jax'`: Return a batch of type `jax.numpy.ndarray`.
+            data_format (`ChannelDimension` or `str`, *optional*, defaults to `ChannelDimension.FIRST`):
+                The channel dimension format for the output image. Can be one of:
+                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
+                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
+                - Unset: Use the channel dimension format of the input image.
+            input_data_format (`ChannelDimension` or `str`, *optional*):
+                The channel dimension format for the input image. If unset, the channel dimension format is inferred
+                from the input image. Can be one of:
+                - `"channels_first"` or `ChannelDimension.FIRST`: image in (num_channels, height, width) format.
+                - `"channels_last"` or `ChannelDimension.LAST`: image in (height, width, num_channels) format.
+                - `"none"` or `ChannelDimension.NONE`: image in (height, width) format.
+
+        """
+        do_resize = do_resize if do_resize is not None else self.do_resize
+        resample = resample if resample is not None else self.resample
+        do_rescale = do_rescale if do_rescale is not None else self.do_rescale
+        rescale_factor = rescale_factor if rescale_factor is not None else self.rescale_factor
+        do_normalize = do_normalize if do_normalize is not None else self.do_normalize
+        image_mean = image_mean if image_mean is not None else self.image_mean
+        image_std = image_std if image_std is not None else self.image_std
+        merge_size = merge_size if merge_size is not None else self.merge_size
+        do_convert_rgb = do_convert_rgb if do_convert_rgb is not None else self.do_convert_rgb
+
+        clips = make_batched_images(images)
+        num_clips = len(clips)
+
+        if isinstance(merge_size, (list, tuple)):
+            assert len(merge_size) == num_clips, (
+                f"merge_size length ({len(merge_size)}) must match number of clips ({num_clips})"
+            )
+            clip_merge_sizes = list(merge_size)
+        else:
+            clip_merge_sizes = [merge_size] * num_clips
+
+        if frame_types is None:
+            clip_frame_types = [None] * num_clips
+        elif isinstance(frame_types, (list, tuple)) and len(frame_types) > 0:
+            if isinstance(frame_types[0], (list, tuple)) or frame_types[0] is None:
+                assert len(frame_types) == num_clips, (
+                    f"frame_types length ({len(frame_types)}) must match number of clips ({num_clips})"
+                )
+                clip_frame_types = list(frame_types)
+            else:
+                assert num_clips == 1, "Flat frame_types is only supported for a single clip"
+                clip_frame_types = [frame_types]
+        else:
+            clip_frame_types = [None] * num_clips
+
+        pixel_values, grid_sizes, per_frame_merge_sizes = [], [], []
+
+        clip_max_tokens_list = self._allocate_token_budget(
+            clips, clip_merge_sizes, input_data_format,
+        )
+
+        for clip, ms, ft, clip_max_tokens in zip(clips, clip_merge_sizes, clip_frame_types, clip_max_tokens_list):
+            target_sizes = simple_batched_resize(
+                clip,
+                factor=self.patch_size * ms,
+                min_tokens=self.min_tokens,
+                max_tokens=clip_max_tokens,
+                input_data_format=input_data_format,
+                frame_types=ft,
+            )
+
+            for frame, target_size in zip(clip, target_sizes):
+                patches, grid_size = self._preprocess(
+                    frame,
+                    target_size=target_size,
+                    merge_size=ms,
+                    do_resize=do_resize,
+                    resample=resample,
+                    do_rescale=do_rescale,
+                    rescale_factor=rescale_factor,
+                    do_normalize=do_normalize,
+                    image_mean=image_mean,
+                    image_std=image_std,
+                    data_format=data_format,
+                    do_convert_rgb=do_convert_rgb,
+                    input_data_format=input_data_format,
+                )
+                pixel_values.append(patches)
+                grid_sizes.append(grid_size)
+                per_frame_merge_sizes.append(ms)
+
+        pixel_values = np.concatenate(pixel_values, axis=0)
+        grid_sizes = np.array(grid_sizes)
+        merge_sizes = np.array(per_frame_merge_sizes)
+
+        data = {
+            "pixel_values": pixel_values,
+            "grid_sizes": grid_sizes,
+            "merge_sizes": merge_sizes,
+        }
+
+        return BatchFeature(data=data, tensor_type=return_tensors)

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b13dc884fa52b873d861b32f3bcfa406b04964ec3412e69a1e0424950d5832d3
+size 4335965984

modeling_penguinvl_encoder.py

@@ -0,0 +1,549 @@
+from torch import nn
+import torch
+import math
+import warnings
+from functools import partial
+from .configuration_penguinvl_encoder import PenguinVLVisionEncoderConfig
+from transformers.modeling_utils import PreTrainedModel
+from transformers.models.qwen3.modeling_qwen3 import Qwen3Model, Qwen3Attention, rotate_half, Qwen3DecoderLayer
+from typing import List, Optional, Tuple, Union
+from transformers.modeling_outputs import BaseModelOutputWithPast
+from transformers.processing_utils import Unpack
+from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.utils import logging, is_flash_attn_greater_or_equal_2_10, is_flash_attn_2_available
+from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS
+from torch.nn.init import _calculate_fan_in_and_fan_out
+import torch.nn.functional as F
+if is_flash_attn_2_available():
+    from transformers.modeling_flash_attention_utils import _flash_attention_forward
+    from flash_attn import flash_attn_varlen_func
+
+logger = logging.get_logger(__name__)
+
+class PenguinVLVisionEncoderEmbeddings(nn.Module):
+
+    def __init__(self, config: PenguinVLVisionEncoderConfig):
+        super().__init__()
+        self.config = config
+        self.embed_dim = config.hidden_size
+        self.patch_size = config.patch_size
+
+        self.patch_embedding = nn.Conv2d(
+            in_channels=config.num_channels,
+            out_channels=self.embed_dim,
+            kernel_size=self.patch_size,
+            stride=self.patch_size,
+            padding="valid",
+        )
+
+    def forward(self, hidden_states: torch.Tensor) -> torch.Tensor:
+        hidden_states = hidden_states.view(
+            -1, self.config.num_channels, self.patch_size, self.patch_size
+        )
+        patch_embeds = self.patch_embedding(hidden_states)
+        embeddings = patch_embeds.view(-1, self.embed_dim)
+
+        return embeddings
+    
+
+# Adapted from Qwen2VLRotaryEmbedding in transformers/models/qwen2/modeling_qwen2.py
+class VisualRotaryEmbedding(nn.Module):
+    def __init__(
+        self,
+        dim=None,
+        max_position_embeddings=2048,
+        base=10000,
+        device=None,
+        scaling_factor=1.0,
+        rope_type="default",
+        config = None,
+    ):
+        super().__init__()
+        # TODO (joao): remove the `if` below, only used for BC
+        self.rope_kwargs = {}
+        if config is None:
+            logger.warning_once(
+                "`Qwen2VLRotaryEmbedding` can now be fully parameterized by passing the model config through the "
+                "`config` argument. All other arguments will be removed in v4.46"
+            )
+            self.rope_kwargs = {
+                "rope_type": rope_type,
+                "factor": scaling_factor,
+                "dim": dim,
+                "base": base,
+                "max_position_embeddings": max_position_embeddings,
+            }
+            self.rope_type = rope_type
+            self.max_seq_len_cached = max_position_embeddings
+            self.original_max_seq_len = max_position_embeddings
+        else:
+            # BC: "rope_type" was originally "type"
+            if config.rope_scaling is not None:
+                self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
+            else:
+                self.rope_type = "default"
+            self.max_seq_len_cached = config.max_position_embeddings
+            self.original_max_seq_len = config.max_position_embeddings
+
+        self.config = config
+        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+
+        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device, **self.rope_kwargs)
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self.original_inv_freq = self.inv_freq
+
+    def _dynamic_frequency_update(self, position_ids, device):
+        """
+        dynamic RoPE layers should recompute `inv_freq` in the following situations:
+        1 - growing beyond the cached sequence length (allow scaling)
+        2 - the current sequence length is in the original scale (avoid losing precision with small sequences)
+        """
+        seq_len = torch.max(position_ids) + 1
+        if seq_len > self.max_seq_len_cached:  # growth
+            inv_freq, self.attention_scaling = self.rope_init_fn(
+                self.config, device, seq_len=seq_len, **self.rope_kwargs
+            )
+            self.register_buffer("inv_freq", inv_freq, persistent=False)  # TODO joao: may break with compilation
+            self.max_seq_len_cached = seq_len
+
+        if seq_len < self.original_max_seq_len and self.max_seq_len_cached > self.original_max_seq_len:  # reset
+            self.register_buffer("inv_freq", self.original_inv_freq, persistent=False)
+            self.max_seq_len_cached = self.original_max_seq_len
+
+    @torch.no_grad()
+    def forward(self, x, position_ids):
+        if "dynamic" in self.rope_type:
+            self._dynamic_frequency_update(position_ids, device=x.device)
+
+        inv_freq_expanded = self.inv_freq[None, None, :, None].float().expand(2, position_ids.shape[1], -1, 1)
+        position_ids_expanded = position_ids[:, :, None, :].float()  # shape (2, bs, 1, positions)
+        # Force float32 (see https://github.com/huggingface/transformers/pull/29285)
+        device_type = x.device.type
+        device_type = device_type if isinstance(device_type, str) and device_type != "mps" else "cpu"
+        with torch.autocast(device_type=device_type, enabled=False):
+            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(2, 3)
+            emb = torch.cat((freqs, freqs), dim=-1)
+            cos = emb.cos()
+            sin = emb.sin()
+
+        # Advanced RoPE types (e.g. yarn) apply a post-processing scaling factor, equivalent to scaling attention
+        cos = cos * self.attention_scaling
+        sin = sin * self.attention_scaling
+
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+
+def apply_multimodal_rotary_pos_emb(q, k, cos, sin, unsqueeze_dim=1):
+    rope_section = [cos.shape[-1] // 2, cos.shape[-1] // 2]
+    cos = torch.cat([m[i % 2] for i, m in enumerate(cos.split(rope_section, dim=-1))], dim=-1).unsqueeze(unsqueeze_dim)
+    sin = torch.cat([m[i % 2] for i, m in enumerate(sin.split(rope_section, dim=-1))], dim=-1).unsqueeze(unsqueeze_dim)
+
+    q_embed = (q * cos) + (rotate_half(q) * sin)
+    k_embed = (k * cos) + (rotate_half(k) * sin)
+    return q_embed, k_embed
+
+
+class PenguinVLAttention(Qwen3Attention):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    # Copied from transformers.models.llama.modeling_llama.LlamaFlashAttention2.__init__
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.is_causal = False
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: Tuple[torch.Tensor, torch.Tensor],
+        attention_mask: Optional[torch.Tensor],
+        past_key_value: Optional[Cache] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        cu_seqlens: Optional[torch.Tensor] = None,
+        **kwargs: Unpack[FlashAttentionKwargs],
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+
+        query_states = self.q_norm(self.q_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
+        key_states = self.k_norm(self.k_proj(hidden_states).view(hidden_shape)).transpose(1, 2)
+        value_states = self.v_proj(hidden_states).view(hidden_shape).transpose(1, 2)
+
+        cos, sin = position_embeddings
+        query_states, key_states = apply_multimodal_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_value is not None:
+            # sin and cos are specific to RoPE models; cache_position needed for the static cache
+            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
+            key_states, value_states = past_key_value.update(key_states, value_states, self.layer_idx, cache_kwargs)
+
+        # This is before the transpose
+        seq_len = query_states.shape[2]
+
+        # In PEFT, usually we cast the layer norms in float32 for training stability reasons
+        # therefore the input hidden states gets silently casted in float32. Hence, we need
+        # cast them back in the correct dtype just to be sure everything works as expected.
+        # This might slowdown training & inference so it is recommended to not cast the LayerNorms
+        # in fp32. (usually our RMSNorm modules handle it correctly)
+        target_dtype = None
+        if query_states.dtype == torch.float32:
+            if torch.is_autocast_enabled():
+                target_dtype = torch.get_autocast_gpu_dtype()
+            # Handle the case where the model is quantized
+            elif hasattr(self.config, "_pre_quantization_dtype"):
+                target_dtype = self.config._pre_quantization_dtype
+            else:
+                target_dtype = next(layer for layer in self.modules() if isinstance(layer, torch.nn.Linear)).weight.dtype
+
+        # FA2 always relies on the value set in the module, so remove it if present in kwargs to avoid passing it twice
+        kwargs.pop("is_causal", None)
+
+        # Reashape to the expected shape for Flash Attention
+        query_states = query_states.transpose(1, 2).squeeze(0)
+        key_states = key_states.transpose(1, 2).squeeze(0)
+        value_states = value_states.transpose(1, 2).squeeze(0)
+
+        max_seqlen = (cu_seqlens[1:] - cu_seqlens[:-1]).max().item()
+        attn_output = flash_attn_varlen_func(
+            query_states,
+            key_states,
+            value_states,
+            cu_seqlens_q=cu_seqlens,
+            cu_seqlens_k=cu_seqlens,
+            max_seqlen_q=max_seqlen,
+            max_seqlen_k=max_seqlen,
+            dropout_p=0.0 if not self.training else self.attention_dropout,
+            causal=self.is_causal
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+        return attn_output, None
+    
+
+class PenguinVLDecoderLayer(Qwen3DecoderLayer):
+    def __init__(self, config: PenguinVLVisionEncoderConfig, layer_idx: int):
+        super(PenguinVLDecoderLayer, self).__init__(config, layer_idx)
+        self.self_attn = PenguinVLAttention(config, layer_idx)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_value: Optional[Cache] = None,
+        output_attentions: Optional[bool] = False,
+        use_cache: Optional[bool] = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        position_embeddings: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC
+        cu_seqlens: Optional[torch.Tensor] = None,
+        **kwargs: Unpack[FlashAttentionKwargs],
+    ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:
+        residual = hidden_states
+
+        hidden_states = self.input_layernorm(hidden_states)
+
+        # Self Attention
+        hidden_states, self_attn_weights = self.self_attn(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_value=past_key_value,
+            output_attentions=output_attentions,
+            use_cache=use_cache,
+            cache_position=cache_position,
+            position_embeddings=position_embeddings,
+            cu_seqlens=cu_seqlens,
+            **kwargs,
+        )
+        hidden_states = residual + hidden_states
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+
+        outputs = (hidden_states,)
+        if output_attentions:
+            outputs += (self_attn_weights,)
+
+        return outputs
+    
+    
+class PenguinVLVisionEncoderFromQwen3Model(Qwen3Model):
+    config_class = PenguinVLVisionEncoderConfig
+    def __init__(self, config: PenguinVLVisionEncoderConfig):
+        super().__init__(config)
+        self.layers = nn.ModuleList(
+            [PenguinVLDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        self.rotary_emb = VisualRotaryEmbedding(config=config)
+        del self.embed_tokens
+
+    @staticmethod
+    def _prepare_4d_causal_attention_mask_with_cache_position(
+        attention_mask: torch.Tensor,
+        sequence_length: int,
+        target_length: int,
+        dtype: torch.dtype,
+        device: torch.device,
+        cache_position: torch.Tensor,
+        batch_size: int,
+        config: PenguinVLVisionEncoderConfig,
+        past_key_values: Cache,
+    ):
+        """
+        Override the original causal mask method to create full attention mask instead.
+        Creates a full attention 4D mask of shape `(batch_size, 1, query_length, key_value_length)` 
+        from a 2D mask of shape `(batch_size, key_value_length)`.
+        
+        For vision encoding, we want full attention between all patches, not causal attention.
+        """
+        if attention_mask is not None and attention_mask.dim() == 4:
+            # In this case we assume that the mask comes already in inverted form and requires no inversion or slicing.
+            full_attention_mask = attention_mask
+        else:
+            # Create full attention mask (all zeros, meaning attend to all positions)
+            # We only mask based on the provided attention_mask for padding
+            if attention_mask is not None:
+                # Use the provided attention_mask to handle padding
+                min_dtype = torch.finfo(dtype).min
+                full_attention_mask = torch.zeros(
+                    (sequence_length, target_length), dtype=dtype, device=device
+                )
+                # Expand to 4D
+                full_attention_mask = full_attention_mask[None, None, :, :].expand(batch_size, 1, -1, -1)
+                
+                # Apply padding mask if provided
+                full_attention_mask = full_attention_mask.clone()  # copy to contiguous memory for in-place edit
+                if attention_mask.shape[-1] > target_length:
+                    attention_mask = attention_mask[:, :target_length]
+                mask_length = attention_mask.shape[-1]
+                padding_mask = attention_mask[:, None, None, :] == 0
+                full_attention_mask[:, :, :, :mask_length] = full_attention_mask[:, :, :, :mask_length].masked_fill(
+                    padding_mask, min_dtype
+                )
+            else:
+                # No attention mask provided, create all-zeros mask (full attention)
+                full_attention_mask = torch.zeros(
+                    (batch_size, 1, sequence_length, target_length), dtype=dtype, device=device
+                )
+        return full_attention_mask
+
+    def get_rope_index(self, grid_sizes, merge_sizes, position_ids):
+        position_ids = position_ids.contiguous()
+        batch_size = grid_sizes.shape[0]
+
+        # Vision Part: Generate 2D position indices for vision tokens
+        vision_pos_ids = []
+        for (t, h, w), merge_size in zip(grid_sizes, merge_sizes):
+            # Generate height position indices
+            hpos_ids = torch.arange(h).unsqueeze(1).expand(-1, w).to(position_ids.device)
+            hpos_ids = hpos_ids.reshape(
+                h // merge_size,
+                merge_size,
+                w // merge_size,
+                merge_size,
+            )
+            hpos_ids = hpos_ids.permute(0, 2, 1, 3)
+            hpos_ids = hpos_ids.flatten()
+
+            # Generate width position indices  
+            wpos_ids = torch.arange(w).unsqueeze(0).expand(h, -1).to(position_ids.device)
+            wpos_ids = wpos_ids.reshape(
+                h // merge_size,
+                merge_size,
+                w // merge_size,
+                merge_size,
+            )
+            wpos_ids = wpos_ids.permute(0, 2, 1, 3)
+            wpos_ids = wpos_ids.flatten()
+            
+            # Stack height and width to create 2D positions
+            vision_pos_ids.append(torch.stack([hpos_ids, wpos_ids], dim=-1).repeat(t, 1))
+
+        num_start_idx = 0
+        for batch_idx in range(batch_size):
+            pos_len = vision_pos_ids[batch_idx].shape[0]
+            position_ids[:, 0, num_start_idx: num_start_idx+pos_len] = vision_pos_ids[batch_idx].permute(1, 0)
+            num_start_idx += pos_len
+        
+        return position_ids
+
+
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        grid_sizes: Optional[torch.Tensor] = None,
+        merge_sizes: Optional[torch.Tensor] = None,
+        **flash_attn_kwargs: Unpack[FlashAttentionKwargs],
+    ) -> BaseModelOutputWithPast:
+        output_attentions = output_attentions if output_attentions is not None else self.config.output_attentions
+        output_hidden_states = (
+            output_hidden_states if output_hidden_states is not None else self.config.output_hidden_states
+        )
+        use_cache = use_cache if use_cache is not None else self.config.use_cache
+
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if self.gradient_checkpointing and self.training and use_cache:
+            logger.warning_once(
+                "`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`."
+            )
+            use_cache = False
+
+        # TODO (joao): remove this exception in v4.56 -- it exists for users that try to pass a legacy cache
+        if not isinstance(past_key_values, (type(None), Cache)):
+            raise ValueError("The `past_key_values` should be either a `Cache` object or `None`.")
+
+        if inputs_embeds is None:
+            inputs_embeds = self.embed_tokens(input_ids)
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache()
+
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position = torch.arange(
+                past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
+            )
+
+        # the hard coded `2` is for temporal, height and width.
+        if position_ids is None:
+            position_ids = cache_position.view(1, 1, -1).expand(2, inputs_embeds.shape[0], -1)
+        elif position_ids.dim() == 2:
+            position_ids = position_ids[None, ...].expand(2, position_ids.shape[0], -1)
+        position_ids = self.get_rope_index(grid_sizes, merge_sizes, position_ids)
+
+        causal_mask = None
+
+        hidden_states = inputs_embeds
+
+        # create position embeddings to be shared across the decoder layers
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        # decoder layers
+        all_hidden_states = () if output_hidden_states else None
+        all_self_attns = () if output_attentions else None
+
+        # Calculate cumulative sequence lengths for the grid sizes
+        cu_seqlens = torch.repeat_interleave(grid_sizes[:, 1] * grid_sizes[:, 2], grid_sizes[:, 0]).cumsum(dim=0, dtype=torch.int32)
+        cu_seqlens = F.pad(cu_seqlens, (1, 0), value=0)
+
+        for decoder_layer in self.layers[: self.config.num_hidden_layers]:
+            if output_hidden_states:
+                all_hidden_states += (hidden_states,)
+
+            if self.gradient_checkpointing and self.training:
+                layer_outputs = self._gradient_checkpointing_func(
+                    partial(decoder_layer.__call__, **flash_attn_kwargs),
+                    hidden_states,
+                    causal_mask,
+                    position_ids,
+                    past_key_values,
+                    output_attentions,
+                    use_cache,
+                    cache_position,
+                    position_embeddings,
+                    cu_seqlens,
+                )
+            else:
+                layer_outputs = decoder_layer(
+                    hidden_states,
+                    attention_mask=causal_mask,
+                    position_ids=position_ids,
+                    past_key_value=past_key_values,
+                    output_attentions=output_attentions,
+                    use_cache=use_cache,
+                    cache_position=cache_position,
+                    position_embeddings=position_embeddings,
+                    cu_seqlens=cu_seqlens,
+                    **flash_attn_kwargs,
+                )
+
+            hidden_states = layer_outputs[0]
+
+            if output_attentions:
+                all_self_attns += (layer_outputs[1],)
+
+        hidden_states = self.norm(hidden_states)
+
+        # add hidden states from the last decoder layer
+        if output_hidden_states:
+            all_hidden_states += (hidden_states,)
+
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values if use_cache else None,
+            hidden_states=all_hidden_states,
+            attentions=all_self_attns,
+        )
+
+
+class PenguinVLVisionEncoderModel(PreTrainedModel):
+
+    config_class = PenguinVLVisionEncoderConfig
+    base_model_prefix = "penguinvl_vision_encoder"
+    main_input_name = "pixel_values"
+    supports_gradient_checkpointing = True
+    _no_split_modules = [
+        "PenguinVLVisionEncoderEmbeddings",
+    ]
+    _supports_flash_attn_2 = True
+    _supports_sdpa = True
+
+    def __init__(self, config: PenguinVLVisionEncoderConfig):
+        super().__init__(config=config)
+        self.embeddings = PenguinVLVisionEncoderEmbeddings(config)
+        self.encoder = PenguinVLVisionEncoderFromQwen3Model(config)
+
+        self.post_init()
+
+
+    def forward(self, pixel_values, grid_sizes, merge_sizes=None) -> torch.Tensor:
+        hidden_states = self.embeddings(pixel_values)
+        encoder_output = self.encoder(
+            inputs_embeds=hidden_states[None, ...],
+            grid_sizes=grid_sizes,
+            merge_sizes=merge_sizes,
+            output_hidden_states=True,
+        )
+        hidden_states = encoder_output.hidden_states
+        hidden_states = hidden_states[-1].squeeze(0)
+
+        hidden_states_chunks = hidden_states.split(grid_sizes.prod(dim=1).tolist(), dim=0)
+        outputs = []
+
+        for hidden_states, grid_size, merge_size in zip(hidden_states_chunks, grid_sizes, merge_sizes):
+            # NOTE: previous implementation, which supports downsampling with any factor
+            c = hidden_states.shape[-1]
+            hidden_states = hidden_states.view(
+                grid_size[0], grid_size[1] // merge_size, grid_size[2] // merge_size, merge_size, merge_size,  c
+            ).permute(0, 1, 3, 2, 4, 5)
+            hidden_states = hidden_states.reshape(
+                grid_size[0], grid_size[1], grid_size[2], c
+            ).permute(0, 3, 1, 2)
+            hidden_states = torch.nn.functional.interpolate(
+                hidden_states,
+                size=(grid_size[1] // merge_size, grid_size[2] // merge_size),
+                mode='bilinear'
+            )
+            hidden_states = hidden_states.permute(0, 2, 3, 1).view(-1, c)
+
+            # NOTE: simplified implementation, which only supports downsampling with integer factor
+            # NOTE: this implementation is mathematically equivalent to the previous one when merge_size is 1 or 2 but may cause slightly different results
+            # hidden_states = hidden_states.view(-1, merge_size * merge_size, hidden_states.size(-1))
+            # hidden_states = hidden_states.mean(dim=1)
+
+            outputs.append(hidden_states)
+        return torch.cat(outputs, dim=0)

modeling_penguinvl_qwen3.py

@@ -0,0 +1,475 @@
+# Adopted from https://github.com/haotian-liu/LLaVA.
+# Below is the original copyright:
+# Copyright 2023 Haotian Liu
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""PyTorch PenguinVL model."""
+
+import importlib.util
+import os.path as osp
+import re
+from abc import ABC, abstractmethod
+from typing import List, Optional, Tuple, Union
+
+import torch
+import torch.nn as nn
+import torch.utils.checkpoint
+import math
+
+from transformers import Qwen3ForCausalLM, Qwen3Model
+from transformers.generation.utils import GenerateOutput
+from transformers.modeling_outputs import CausalLMOutputWithPast
+
+try:
+    from .configuration_penguinvl import PenguinVLQwen3Config
+except ModuleNotFoundError:
+    spec = importlib.util.spec_from_file_location(
+        "configuration_penguinvl",
+        osp.join(osp.dirname(__file__), "configuration_penguinvl.py"),
+    )
+    configuration_penguinvl = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(configuration_penguinvl)
+    PenguinVLQwen3Config = getattr(
+        configuration_penguinvl,
+        "PenguinVLQwen3Config",
+    )
+
+try:
+    from .configuration_penguinvl_encoder import PenguinVLVisionEncoderConfig
+    from .modeling_penguinvl_encoder import PenguinVLVisionEncoderModel
+except ModuleNotFoundError:
+    enc_spec = importlib.util.spec_from_file_location(
+        "configuration_penguinvl_encoder",
+        osp.join(osp.dirname(__file__), "configuration_penguinvl_encoder.py"),
+    )
+    configuration_penguinvl_encoder = importlib.util.module_from_spec(enc_spec)
+    enc_spec.loader.exec_module(configuration_penguinvl_encoder)
+    PenguinVLVisionEncoderConfig = getattr(
+        configuration_penguinvl_encoder,
+        "PenguinVLVisionEncoderConfig",
+    )
+    enc_model_spec = importlib.util.spec_from_file_location(
+        "modeling_penguinvl_encoder",
+        osp.join(osp.dirname(__file__), "modeling_penguinvl_encoder.py"),
+    )
+    modeling_penguinvl_encoder = importlib.util.module_from_spec(enc_model_spec)
+    enc_model_spec.loader.exec_module(modeling_penguinvl_encoder)
+    PenguinVLVisionEncoderModel = getattr(
+        modeling_penguinvl_encoder,
+        "PenguinVLVisionEncoderModel",
+    )
+
+
+def build_mlp(depth, hidden_size, output_hidden_size):
+    modules = [nn.Linear(hidden_size, output_hidden_size)]
+    for _ in range(1, depth):
+        modules.append(nn.GELU())
+        modules.append(nn.Linear(output_hidden_size, output_hidden_size))
+    return nn.Sequential(*modules)
+
+
+def build_vision_projector(config, **kwargs):
+    projector_type = getattr(config, 'vision_projector_type', 'linear')
+    if projector_type == "linear":
+        return nn.Linear(config.mm_hidden_size, config.hidden_size)
+    elif projector_type.startswith("mlp"):
+        return MlpGeluProjector(config.vision_encoder_config.hidden_size, config.hidden_size, projector_type)
+    else:
+        raise ValueError(f'Unknown projector type: {projector_type}')
+
+
+class MlpGeluProjector(nn.Module):
+
+    def __init__(self, mm_hidden_size, hidden_size, projector_type):
+        super().__init__()
+
+        mlp_gelu_match = re.match(r"^mlp(\d+)x_gelu$", projector_type)
+        mlp_depth = int(mlp_gelu_match.group(1))
+
+        self.readout = build_mlp(mlp_depth, mm_hidden_size, hidden_size)
+
+    def forward(self, x):
+        x = self.readout(x)
+        return x
+    
+
+class MlpGeluDownsampleProjector(nn.Module):
+    def __init__(self, mm_hidden_size, hidden_size, projector_type):
+        super().__init__()
+        self.downsample = nn.Linear(mm_hidden_size*8, mm_hidden_size)
+
+        mlp_gelu_match = re.match(r"^dmlp(\d+)x_gelu$", projector_type)
+        mlp_depth = int(mlp_gelu_match.group(1))
+
+        self.readout = build_mlp(mlp_depth, mm_hidden_size, hidden_size)
+
+    def forward(self, x):
+        B, S, D = x.shape
+
+        group = 8
+        S8 = (S // group) * group
+        x = x[:, :S8, :]
+        x = x.reshape(B, S8 // group, group * D)
+        x = self.downsample(x)
+        x = self.readout(x)
+        return x
+
+
+class VLMMetaModel:
+
+    def __init__(self, config):
+        super(VLMMetaModel, self).__init__(config)
+        if config.vision_encoder is not None:
+            # Load with custom config/model so transformers doesn't need to know "penguinvl_vision_encoder"
+            encoder_config = PenguinVLVisionEncoderConfig.from_pretrained(config.vision_encoder)
+            self.vision_encoder = PenguinVLVisionEncoderModel.from_pretrained(
+                config.vision_encoder,
+                config=encoder_config,
+                attn_implementation=self.config._attn_implementation,
+                torch_dtype=self.dtype,
+            )
+            self.config.vision_encoder_config = self.vision_encoder.config
+            self.config.vision_encoder = None
+        elif config.vision_encoder_config is not None:
+            self.vision_encoder = PenguinVLVisionEncoderModel.from_config(
+                self.config.vision_encoder_config,
+                attn_implementation=self.config._attn_implementation,
+                torch_dtype=self.dtype,
+            )
+        else:
+            raise ValueError("Vision encoder is not provided in config")
+
+        self.vision_projector = build_vision_projector(config)
+
+    def get_vision_encoder(self):
+        return self.vision_encoder
+
+    def get_vision_projector(self):
+        return self.vision_projector
+
+
+class PenguinVLQwen3Model(VLMMetaModel, Qwen3Model):
+
+    config_class = PenguinVLQwen3Config
+
+    def __init__(self, config: PenguinVLQwen3Config):
+        super(PenguinVLQwen3Model, self).__init__(config)
+
+
+class VLMMetaForCausalLM(ABC):
+
+    @abstractmethod
+    def get_model(self):
+        pass
+
+    def get_vision_encoder(self):
+        return self.get_model().get_vision_encoder()
+
+    def get_vision_projector(self):
+        return self.get_model().get_vision_projector()
+
+    def encode_images(
+        self,
+        pixel_values: torch.FloatTensor,
+        grid_sizes: torch.LongTensor,
+        merge_sizes: torch.LongTensor,
+    ) -> torch.FloatTensor:
+        mm_features = self.get_model().get_vision_encoder()(
+            pixel_values=pixel_values,
+            grid_sizes=grid_sizes,
+            merge_sizes=merge_sizes,
+        )
+        mm_features = self.get_model().vision_projector(mm_features)
+        return mm_features
+
+    def _get_valid_visual_tokens(
+        self,
+        mm_features: torch.FloatTensor,
+        batched_num_patches: torch.LongTensor,
+        modals: List[str],
+    ):
+        valid_masks = []
+        for num_patches, modal in zip(batched_num_patches, modals):
+            valid_mask = torch.full((num_patches, ), modal != "text", dtype=torch.bool, device=mm_features.device)
+            valid_masks.append(valid_mask)
+        mm_features = mm_features[torch.cat(valid_masks)]
+        return mm_features
+
+    def prepare_inputs_labels_for_multimodal(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        labels: Optional[torch.LongTensor] = None,
+        pixel_values: Optional[torch.FloatTensor] = None,
+        grid_sizes: Optional[torch.LongTensor] = None,
+        merge_sizes: Optional[torch.LongTensor] = None,
+        modals: Optional[List[str]] = None,
+    ):
+        vision_encoder = self.get_vision_encoder()
+        # NOTE: text-only situation
+        if vision_encoder is None or pixel_values is None or input_ids.shape[1] == 1:
+            return input_ids, attention_mask, position_ids, past_key_values, None, labels
+
+        # 1. flatten text inputs
+        B, N = input_ids.shape
+        input_ids = input_ids.view(B * N)
+        if attention_mask is not None:
+            attention_mask = attention_mask.view(B * N)
+        if position_ids is not None:
+            position_ids = position_ids.view(B * N)
+        if labels is not None:
+            labels = labels.view(B * N)
+
+        # 2. embed visual tokens
+        image_selected, mm_features_teacher = None, None
+        if pixel_values is not None:
+            # 2.1 encode images
+            batched_num_patches = grid_sizes.prod(dim=1).div(merge_sizes ** 2).long()
+            mm_features = self.encode_images(pixel_values, grid_sizes, merge_sizes)
+            mm_features = mm_features.to(input_ids.device)
+            mm_features = self._get_valid_visual_tokens(mm_features, batched_num_patches, modals)
+
+            # 2.2 get image selected
+            image_selected = (input_ids == self.config.image_token_index)
+            input_ids[image_selected] = 0
+
+            num_vision_tokens = image_selected.sum()
+            if mm_features.size(0) != num_vision_tokens:
+                print(f"Number of vision_features ({mm_features.size(0)}) does not match the number of image tokens ({num_vision_tokens}). Please check the inputs.")
+                mm_features = mm_features[:num_vision_tokens]
+
+        # 3. replace multimodal tokens with features
+        inputs_embeds = self.get_model().embed_tokens(input_ids).clone()
+        if image_selected is not None:
+            inputs_embeds[image_selected] = inputs_embeds[image_selected] * 0.0 + mm_features
+
+        # 4. reshape back to batched format
+        C = inputs_embeds.shape[-1]
+        inputs_embeds = inputs_embeds.reshape(B, -1, C)
+        if attention_mask is not None:
+            attention_mask = attention_mask.view(B, -1)
+        if labels is not None:
+            labels = labels.view(B, -1)
+        if position_ids is not None:
+            position_ids = position_ids.view(B, -1)
+
+        return None, attention_mask, position_ids, past_key_values, inputs_embeds, labels
+
+
+class PenguinVLQwen3ForCausalLM(Qwen3ForCausalLM, VLMMetaForCausalLM):
+
+    config_class = PenguinVLQwen3Config
+
+    def __init__(self, config, **kwargs):
+        super(Qwen3ForCausalLM, self).__init__(config)
+        self.model = PenguinVLQwen3Model(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    def get_model(self):
+        return self.model
+    
+    @classmethod
+    def _load_pretrained_model(
+        cls,
+        model,
+        state_dict,
+        checkpoint_files,
+        pretrained_model_name_or_path,
+        ignore_mismatched_sizes=False,
+        sharded_metadata=None,
+        device_map=None,
+        disk_offload_folder=None,
+        offload_state_dict=None,
+        dtype=None,
+        hf_quantizer=None,
+        keep_in_fp32_regex=None,
+        device_mesh=None,
+        key_mapping=None,
+        weights_only=True,
+    ):
+        """
+        Override to handle nested vision_encoder keys before calling parent's load method.
+        Remaps keys from 'model.vision_encoder.vision_encoder.*' to 'model.vision_encoder.*'
+        """
+        # If state_dict is provided and needs remapping, do it here
+        if state_dict is not None:
+            needs_remapping = any(k.startswith('model.vision_encoder.vision_encoder.') for k in state_dict.keys())
+            if needs_remapping:
+                print("Detected nested encoder keys, remapping 'model.vision_encoder.vision_encoder.*' -> 'model.vision_encoder.*'")
+                new_state_dict = {}
+                for k, v in state_dict.items():
+                    if k.startswith('model.vision_encoder.vision_encoder.'):
+                        # Remap: model.vision_encoder.vision_encoder.xxx -> model.vision_encoder.xxx
+                        new_key = k.replace('model.vision_encoder.vision_encoder.', 'model.vision_encoder.')
+                        new_state_dict[new_key] = v
+                    else:
+                        new_state_dict[k] = v
+                state_dict = new_state_dict
+        
+        # For checkpoint files, we need to add key_mapping to remap the keys during loading
+        if checkpoint_files is not None and key_mapping is None:
+            # Check if we need remapping by loading the first checkpoint
+            from transformers.modeling_utils import load_state_dict
+            checkpoint = {}
+            checkpoint_files_list = checkpoint_files if isinstance(checkpoint_files, list) else [checkpoint_files]
+            for ckpt_file in checkpoint_files_list:
+                ckpt = load_state_dict(ckpt_file, map_location="cpu", weights_only=weights_only)
+                checkpoint.update(ckpt)
+            needs_remapping = any(k.startswith('model.vision_encoder.vision_encoder.') for k in checkpoint.keys())
+
+            if needs_remapping:
+                print("Detected nested encoder keys in checkpoint, adding key mapping for vision_encoder")
+                key_mapping = {}
+                for k in checkpoint.keys():
+                    if k.startswith('model.vision_encoder.vision_encoder.'):
+                        new_key = k.replace('model.vision_encoder.vision_encoder.', 'model.vision_encoder.')
+                        key_mapping[k] = new_key
+            del checkpoint
+        
+        return super()._load_pretrained_model(
+            model=model,
+            state_dict=state_dict,
+            checkpoint_files=checkpoint_files,
+            pretrained_model_name_or_path=pretrained_model_name_or_path,
+            ignore_mismatched_sizes=ignore_mismatched_sizes,
+            sharded_metadata=sharded_metadata,
+            device_map=device_map,
+            disk_offload_folder=disk_offload_folder,
+            offload_state_dict=offload_state_dict,
+            dtype=dtype,
+            hf_quantizer=hf_quantizer,
+            keep_in_fp32_regex=keep_in_fp32_regex,
+            device_mesh=device_mesh,
+            key_mapping=key_mapping,
+            weights_only=weights_only,
+        )
+
+    # NOTE: arguments are copied from transformers==4.51.3
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[List[torch.FloatTensor]] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        output_attentions: Optional[bool] = None,
+        output_hidden_states: Optional[bool] = None,
+        return_dict: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        num_logits_to_keep: int = 0,
+        # multimodal inputs
+        pixel_values: Optional[torch.FloatTensor] = None,
+        grid_sizes: Optional[torch.LongTensor] = None,
+        merge_sizes: Optional[torch.LongTensor] = None,
+        modals: Optional[List[str]] = None,
+        **loss_kwargs,
+    ) -> Union[Tuple, CausalLMOutputWithPast]:
+        if inputs_embeds is None:
+            (
+                input_ids,
+                attention_mask,
+                position_ids,
+                past_key_values,
+                inputs_embeds,
+                labels,
+            ) = self.prepare_inputs_labels_for_multimodal(
+                input_ids=input_ids,
+                attention_mask=attention_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                labels=labels,
+                pixel_values=pixel_values,
+                grid_sizes=grid_sizes,
+                merge_sizes=merge_sizes,
+                modals=modals,
+            )
+
+        return super().forward(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            labels=labels,
+            use_cache=use_cache,
+            output_attentions=output_attentions,
+            output_hidden_states=output_hidden_states,
+            return_dict=return_dict,
+            cache_position=cache_position,
+            num_logits_to_keep=num_logits_to_keep,
+            **loss_kwargs,
+        )
+
+    @torch.no_grad()
+    def generate(
+        self,
+        # multimodal inputs
+        pixel_values: Optional[torch.FloatTensor] = None,
+        grid_sizes: Optional[torch.LongTensor] = None,
+        merge_sizes: Optional[torch.LongTensor] = None,
+        modals: Optional[List[str]] = None,
+        **kwargs,
+    ) -> Union[GenerateOutput, torch.LongTensor]:
+        input_ids = kwargs.pop("input_ids", None)
+        attention_mask = kwargs.pop("attention_mask", None)
+        position_ids = kwargs.pop("position_ids", None)
+        past_key_values = kwargs.pop("past_key_values", None)
+
+        if "inputs_embeds" in kwargs:
+            raise NotImplementedError("`inputs_embeds` is not supported")
+
+        if pixel_values is not None:
+            (
+                input_ids,
+                attention_mask,
+                position_ids,
+                past_key_values,
+                inputs_embeds,
+                labels,
+            ) = self.prepare_inputs_labels_for_multimodal(
+                input_ids=input_ids,
+                attention_mask=attention_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                labels=None,
+                pixel_values=pixel_values,
+                grid_sizes=grid_sizes,
+                merge_sizes=merge_sizes,
+                modals=modals,
+            )
+        else:
+            inputs_embeds = self.get_model().embed_tokens(input_ids)
+
+        return super().generate(
+            position_ids=position_ids,
+            attention_mask=attention_mask,
+            inputs_embeds=inputs_embeds,
+            **kwargs
+        )
+
+    def prepare_inputs_for_generation(self, input_ids, past_key_values=None, inputs_embeds=None, **kwargs):
+        images = kwargs.pop("images", None)
+        _inputs = super().prepare_inputs_for_generation(
+            input_ids, past_key_values=past_key_values, inputs_embeds=inputs_embeds, **kwargs
+        )
+        if images is not None:
+            _inputs['images'] = images
+        return _inputs

preprocessor_config.json

@@ -0,0 +1,27 @@
+{
+  "auto_map": {
+    "AutoImageProcessor": "image_processing_penguinvl.PenguinVLImageProcessor",
+    "AutoProcessor": "processing_penguinvl.PenguinVLQwen3Processor"
+  },
+  "do_convert_rgb": true,
+  "do_normalize": true,
+  "do_rescale": true,
+  "do_resize": true,
+  "image_mean": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "image_processor_type": "PenguinVLImageProcessor",
+  "image_std": [
+    0.5,
+    0.5,
+    0.5
+  ],
+  "max_tokens": 16384,
+  "min_tokens": 16,
+  "patch_size": 14,
+  "processor_class": "PenguinVLQwen3Processor",
+  "resample": 3,
+  "rescale_factor": 0.00392156862745098
+}

processing_penguinvl.py

@@ -0,0 +1,1520 @@
+"""Processor class for PenguinVL."""
+
+import copy
+import importlib.util
+import os
+import os.path as osp
+import warnings
+from collections import defaultdict
+from typing import Any, List, Union, Dict, Optional, Tuple, TypedDict
+
+import cv2
+import ffmpeg
+import imageio
+import json
+import math
+import numpy as np
+import torch
+import transformers
+from decord import VideoReader, cpu
+from einops import rearrange
+from torch import nn
+from PIL import Image
+from transformers.feature_extraction_utils import BatchFeature
+from transformers.image_utils import ImageInput
+from transformers.processing_utils import ProcessingKwargs, ProcessorMixin, Unpack
+from transformers.tokenization_utils_base import PreTokenizedInput, TextInput
+
+try:
+    from . import image_processing_penguinvl
+    from .image_processing_penguinvl import (
+        is_valid_image, is_valid_video,
+    )
+except ModuleNotFoundError:
+    spec = importlib.util.spec_from_file_location(
+        "image_processing_penguinvl",
+        osp.join(osp.dirname(__file__), "image_processing_penguinvl.py"),
+    )
+    image_processing_penguinvl = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(image_processing_penguinvl)
+    is_valid_image = getattr(image_processing_penguinvl, "is_valid_image")
+    is_valid_video = getattr(image_processing_penguinvl, "is_valid_video")
+
+# constants
+DEFAULT_IMAGE_TOKEN = "<image>"
+IGNORE_INDEX = -100
+
+# Type aliases
+Conversation = List[Dict[str, Any]]
+SingleImage = Union[Image.Image, np.ndarray, torch.Tensor]
+SingleVideo = Union[List[SingleImage], np.ndarray, torch.Tensor]
+BatchedImage = List[Union[SingleImage, SingleVideo]]
+BatchedNamedImage = List[Tuple[str, Union[SingleImage, SingleVideo]]]
+
+
+def _custom_import(class_name: str):
+    try:
+        attribute_class = getattr(transformers, class_name)
+    except AttributeError:
+        if "image" in class_name.lower():
+            attribute_class = getattr(image_processing_penguinvl, class_name)
+    return attribute_class
+
+
+def is_named_image(image) -> bool:
+    return isinstance(image, (list, tuple)) and \
+        len(image) == 2 and \
+        isinstance(image[0], str) and \
+        image[0] in ["image", "video"] and \
+        (is_valid_image(image[1]) or is_valid_video(image[1]))
+
+
+def make_batched_images(images) -> List[List[ImageInput]]:
+    if isinstance(images, (list, tuple)) and all(is_named_image(image) for image in images):
+        # list of named images
+        return [image[0] for image in images], [image[1] for image in images]
+    elif isinstance(images, (list, tuple)) and all(is_valid_image(image) or is_valid_video(image) for image in images):
+        # list of images/videos
+        batch = []
+        for image in images:
+            if is_valid_video(image):
+                batch.append(("video", image))
+            elif is_valid_image(image):
+                batch.append(("image", image))
+            else:
+                raise ValueError(f"Could not make batched images from {images}")
+        return [x[0] for x in batch], [x[1] for x in batch]
+    elif is_named_image(images):
+        # named images
+        return [images[0]], [image[1]]
+    elif is_valid_video(images):
+        # single video
+        return ["video"], [images]
+    elif is_valid_image(images):
+        # single image
+        return ["image"], [images]
+
+    raise ValueError(f"Could not make batched images from {images}")
+
+
+def frame_sample(duration, mode='uniform', num_frames=None, vid_fps=None, fps=None):
+    if mode == 'uniform':
+        assert num_frames is not None, "Number of frames must be provided for uniform sampling."
+        if duration <= num_frames:
+            return np.arange(duration).astype(int)
+        # NOTE: v1 version
+        # Calculate the size of each segment from which a frame will be extracted
+        # if duration <= num_frames:
+        #     return np.arange(duration).astype(int)
+        # seg_size = float(duration - 1) / num_frames
+
+        # frame_ids = []
+        # for i in range(num_frames):
+        #     # Calculate the start and end indices of each segment
+        #     start = seg_size * i
+        #     end   = seg_size * (i + 1)
+        #     # Append the middle index of the segment to the list
+        #     frame_ids.append((start + end) / 2)
+
+        # return np.round(np.array(frame_ids) + 1e-6).astype(int)
+        # NOTE: v0 version
+        return np.linspace(0, duration-1, num_frames, dtype=int)
+    elif mode == 'fps':
+        assert vid_fps is not None, "FPS must be provided for FPS sampling."
+        assert fps is not None, "FPS must be provided for FPS sampling."
+        segment_len = min(vid_fps // fps, duration)
+        return np.arange(segment_len // 2, duration, segment_len, dtype=int)
+    else:
+        raise ImportError(f'Unsupported frame sampling mode: {mode}')
+
+
+def load_video_from_ids(video_path, s=None, e=None, fps=None, max_frames=128, temporal_factor=1):
+    if s is not None and e is not None:
+        s = s if s >= 0. else 0.
+        e = e if e >= 0. else 0.
+        if s > e:
+            s, e = e, s
+        elif s == e:
+            e = s + 1
+
+    # 1. Loading Video
+    if os.path.isdir(video_path):
+        frame_files = sorted(os.listdir(video_path))
+
+        vid_fps = 3
+        num_frames_of_video = len(frame_files)
+    elif video_path.endswith('.gif'):
+        gif_reader = imageio.get_reader(video_path)
+
+        vid_fps = 25
+        num_frames_of_video = len(gif_reader)
+    else:
+        vreader = VideoReader(video_path, ctx=cpu(0), num_threads=2)
+        # vreader = VideoReader(video_path, ctx=cpu(0), num_threads=1)
+
+        vid_fps = vreader.get_avg_fps()
+        num_frames_of_video = len(vreader)
+
+    # 2. Determine frame range & Calculate frame indices
+    f_start = 0                       if s is None else max(int(s * vid_fps) - 1, 0)
+    f_end   = num_frames_of_video - 1 if e is None else min(int(e * vid_fps) - 1, num_frames_of_video - 1)
+    frame_indices = list(range(f_start, f_end + 1))
+
+    duration = len(frame_indices)
+    # 3. Sampling frame indices
+    if fps is not None and duration / vid_fps < max_frames:
+        sampled_frame_indices = [frame_indices[i] for i in frame_sample(duration, mode='fps', vid_fps=vid_fps, fps=fps)]
+    else:
+        sampled_frame_indices = [frame_indices[i] for i in frame_sample(duration, mode='uniform', num_frames=max_frames)]
+
+    # 4. Acquire frame data
+    if os.path.isdir(video_path):
+        frames = np.array([cv2.cvtColor(cv2.imread(os.path.join(video_path, frame_files[frame_idx])), cv2.COLOR_BGR2RGB) for frame_idx in sampled_frame_indices])
+    elif video_path.endswith('.gif'):
+        frames = np.array([cv2.cvtColor(frame, cv2.COLOR_RGBA2RGB) for idx, frame in enumerate(gif_reader) if idx in sampled_frame_indices])
+    else:
+        frames = vreader.get_batch(sampled_frame_indices).asnumpy()
+
+    frames = frames.transpose(0, 3, 1, 2)
+    timestamps = [x / vid_fps for x in sampled_frame_indices]
+
+    if temporal_factor > 1:
+        pad_length = temporal_factor - len(frames) % temporal_factor
+        frames = np.concatenate([frames, frames[-1:].repeat(pad_length, axis=0)])
+        [timestamps.append(timestamps[-1] + 1 / fps) for _ in range(pad_length)]
+
+    frames = [frame for frame in frames]
+
+    return frames, timestamps
+
+
+
+def round_by_factor(number: int, factor: int) -> int:
+    """Returns the closest integer to 'number' that is divisible by 'factor'."""
+    return round(number / factor) * factor
+
+
+def ceil_by_factor(number: int, factor: int) -> int:
+    """Returns the smallest integer greater than or equal to 'number' that is divisible by 'factor'."""
+    return math.ceil(number / factor) * factor
+
+
+def floor_by_factor(number: int, factor: int) -> int:
+    """Returns the largest integer less than or equal to 'number' that is divisible by 'factor'."""
+    return math.floor(number / factor) * factor
+
+def smart_resize(
+        height: int, width: int,
+        factor: int = 14,
+        min_pixels: int = 0,
+        max_pixels: int = 16384):
+    """
+    Rescales the image so that the following conditions are met:
+
+    1. Both dimensions (height and width) are divisible by 'factor'.
+
+    2. The total number of pixels is within the range ['min_pixels', 'max_pixels'].
+
+    3. The aspect ratio of the image is maintained as closely as possible.
+    """
+
+    if max(height, width) / min(height, width) > 200:
+        raise ValueError(
+            f"absolute aspect ratio must be smaller than {200}, got {max(height, width) / min(height, width)}"
+        )
+    h_bar = max(factor, round_by_factor(height, factor))
+    w_bar = max(factor, round_by_factor(width, factor))
+    if h_bar * w_bar > max_pixels:
+        beta = math.sqrt((height * width) / max_pixels)
+        h_bar = floor_by_factor(height / beta, factor)
+        w_bar = floor_by_factor(width / beta, factor)
+    elif h_bar * w_bar < min_pixels:
+        beta = math.sqrt(min_pixels / (height * width))
+        h_bar = ceil_by_factor(height * beta, factor)
+        w_bar = ceil_by_factor(width * beta, factor)
+    return max(h_bar, factor), max(w_bar, factor)
+
+def get_frame_sim(frame1, frame2,
+                  patch_size: int=14,
+                  threshold: float = 0.7,
+                  epsilon: float=1e-8):
+    assert frame1.dim() == 3 and frame2.dim() == 3, "输入必须是3D张量 [C, H, W]"
+    
+    # 将PyTorch张量转换为OpenCV格式的numpy数组
+    def to_numpy_cvt(tensor):
+        # 确保张量在CPU上并转换为HWC格式
+        tensor = tensor.cpu().permute(1, 2, 0).numpy()
+        if tensor.dtype == np.float32 or tensor.dtype == np.float64:
+            tensor = (tensor).astype(np.uint8)
+        # 转换为HSV颜色空间
+        return cv2.cvtColor(tensor, cv2.COLOR_RGB2HSV)
+
+    # 转换颜色空间
+    frame1_hsv = to_numpy_cvt(frame1)
+    frame2_hsv = to_numpy_cvt(frame2)
+
+    # 将HSV图像转回PyTorch张量
+    frame1_tensor = torch.from_numpy(frame1_hsv).permute(2, 0, 1).to(frame1.device).float()
+    frame2_tensor = torch.from_numpy(frame2_hsv).permute(2, 0, 1).to(frame2.device).float()
+
+    # 分块处理
+    patch1 = rearrange(
+        frame1_tensor, "c (h p1) (w p2) -> h w (c p1 p2)", p1=patch_size, p2=patch_size).float()
+    patch2 = rearrange(
+        frame2_tensor, "c (h p1) (w p2) -> h w (c p1 p2)", p1=patch_size, p2=patch_size).float()
+
+    norm1 = torch.norm(patch1, p=2, dim=-1, keepdim=True) + epsilon
+    norm2 = torch.norm(patch2, p=2, dim=-1, keepdim=True) + epsilon
+    
+    normalized1 = patch1 / norm1
+    normalized2 = patch2 / norm2
+    cos_sim = (normalized1 * normalized2).sum(dim=-1)
+    
+    zero_vector_mask = (norm1.squeeze() < 0.01) & (norm2.squeeze() < 0.01)  # 全黑图
+    
+    similar = torch.ones_like(cos_sim)  # 默认全部相似
+    
+    non_zero_mask = ~zero_vector_mask
+    similar[non_zero_mask] = (cos_sim[non_zero_mask] > threshold).float()
+
+    return similar[non_zero_mask].float().mean().item()
+
+def extract_slow_fast_frames(frames, threshold = 0.95):
+    def _extract_slow_indices(frames):
+        assert frames.dim() == 4, "输入必须是4D张量 [N, C, H, W]"
+
+        # 首帧一定是Slow
+        slow_indices = [0]
+        # 定位这里，检查和image[0]报错是不是同一视频
+        last_key_frame = frames[0]
+        for i in range(1, frames.size(0)):
+            current_frame = frames[i]
+            sim = get_frame_sim(last_key_frame, current_frame)
+
+            if sim < threshold:
+                slow_indices.append(i)
+                last_key_frame = current_frame  # 更新关键帧
+        
+        return slow_indices
+
+    _, _, height, width = frames.shape
+    resized_height, resized_width = smart_resize(
+        height,
+        width,
+        factor=14,
+        min_pixels=10 * 14 * 14,
+        max_pixels=10240 * 14 * 14,
+    )
+
+    resized_frames = nn.functional.interpolate(
+        frames,
+        [resized_height, resized_width],
+        mode="bilinear",
+        antialias=True,
+    ).float()
+
+    slow_indices = _extract_slow_indices(resized_frames)
+    frame_types = torch.ones(size=(frames.size(0), ), dtype=torch.int32)
+    frame_types[slow_indices] = 0
+
+    return list(frame_types)
+
+
+class ChatTemplateKwargs(TypedDict, total=False):
+
+    chat_template: Optional[str]
+    add_system_prompt: Optional[bool]
+    add_generation_prompt: Optional[bool]
+
+
+class PenguinVLQwen3ProcessorKwargs(ProcessingKwargs, ChatTemplateKwargs, total=False):
+
+    chat_template_kwargs: ChatTemplateKwargs = {
+        **ChatTemplateKwargs.__annotations__,
+    }
+
+    _defaults = {
+        "text_kwargs": {
+            "padding": False,
+        },
+        "image_kwargs": {
+            "merge_size": None,
+        },
+        "chat_template_kwargs": {
+            "chat_template": None,
+            "add_system_prompt": False,
+            "add_generation_prompt": False,
+        },
+    }
+
+
+class PenguinVLQwen3Processor(ProcessorMixin):
+
+    attributes = ["image_processor", "tokenizer"]
+    image_processor_class = "PenguinVLImageProcessor"
+    tokenizer_class = ("Qwen2Tokenizer", "Qwen2TokenizerFast")
+    valid_kwargs = ["chat_template", "image_merge_size", "video_merge_size", "fps", "max_frames"]
+
+    def __init__(
+        self,
+        image_processor=None,
+        tokenizer=None,
+        chat_template: str = None,
+        image_merge_size: int = 1,
+        video_merge_size: int = 2,
+        fps: Optional[int] = 1,
+        max_frames: Optional[int] = 128,
+        use_codec = False,
+    ):
+        self.image_processor = image_processor
+        self.tokenizer = tokenizer
+        if chat_template is None:
+            chat_template = self.tokenizer.chat_template
+        self.chat_template = chat_template
+
+        self.image_merge_size = image_merge_size
+        self.video_merge_size = video_merge_size
+        self.fps = fps
+        self.max_frames = max_frames
+        self.use_codec = use_codec
+        self.generation_prompt = self._infer_generation_prompt()
+        self.generation_prompt_ids = self.tokenizer.encode(self.generation_prompt, return_tensors="pt")
+        self.generation_prompt_length = len(self.generation_prompt_ids[0])
+        self.image_token_id = self.tokenizer.convert_tokens_to_ids(DEFAULT_IMAGE_TOKEN)
+        self.eos_token_id = self.tokenizer.eos_token_id
+
+    @classmethod
+    def _get_arguments_from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
+        args = []
+        for attribute_name in cls.attributes:
+            class_name = getattr(cls, f"{attribute_name}_class")
+            if isinstance(class_name, tuple):
+                classes = tuple(_custom_import(n) if n is not None else None for n in class_name)
+                use_fast = kwargs.get("use_fast", True)
+                if use_fast and classes[1] is not None:
+                    attribute_class = classes[1]
+                else:
+                    attribute_class = classes[0]
+            else:
+                attribute_class = _custom_import(class_name)
+
+            args.append(attribute_class.from_pretrained(pretrained_model_name_or_path, **kwargs))
+        return args
+
+    def get_generation_prompt(self):
+        return self.generation_prompt
+
+    def get_generation_prompt_ids(self):
+        return self.generation_prompt_ids
+
+    def _infer_generation_prompt(self):
+        pseudo_message = [{"role": "user", "content": ""}]
+        instruction = self.apply_chat_template(pseudo_message, tokenize=False, add_generation_prompt=True)
+        conversation = self.apply_chat_template(pseudo_message, tokenize=False, add_generation_prompt=False)
+        return instruction.replace(conversation, "")
+
+    def _get_downsampled_grid_sizes(self, image_inputs: Dict[str, Any]):
+        grid_sizes = []
+        for grid_size, merge_size in zip(image_inputs.get("grid_sizes", []), image_inputs.get("merge_sizes", [])):
+            if not torch.all(grid_size[1:] % merge_size == 0):
+                warnings.warn(f"Grid size {grid_size} is not divisible by merge size. Some undesired errors may occur.")
+            if grid_size[0] == 1:
+                grid_sizes.append(grid_size[1:] / merge_size)
+            elif grid_size[0] > 1:
+                grid_sizes.extend([grid_size[1:] / merge_size] * grid_size[0])
+        return grid_sizes
+
+    def _get_visual_seq_len(self, grid_size: torch.Tensor):
+        num_tokens = int(grid_size.prod().item())
+        return num_tokens
+
+    def load_images(self, image_path: Union[str, List[str], Image.Image, List[Image.Image]]):
+        if isinstance(image_path, str) and os.path.isfile(image_path):
+            # images = [cv2.cvtColor(cv2.imread(image_path), cv2.COLOR_BGR2RGB)]
+            images = [Image.open(image_path).convert('RGB')]
+        elif isinstance(image_path, str) and os.path.isdir(image_path):
+            # images = [cv2.cvtColor(cv2.imread(os.path.join(image_path, f)), cv2.COLOR_BGR2RGB) for f in sorted(os.listdir(image_path))]
+            images = [Image.open(os.path.join(image_path, f)).convert('RGB') for f in sorted(os.listdir(image_path))]
+        elif isinstance(image_path, list) and isinstance(image_path[0], str):
+            # images = [cv2.cvtColor(cv2.imread(f), cv2.COLOR_BGR2RGB) for f in image_path]
+            images = [Image.open(f).convert('RGB') for f in image_path]
+        elif isinstance(image_path, list) and isinstance(image_path[0], Image.Image):
+            images = [np.array(x) for x in image_path]
+        elif isinstance(image_path, Image.Image):
+            images = [np.array(image_path)]
+        else:
+            raise ValueError(f"Unsupported image path type: {type(image_path)}")
+        return images
+
+    def load_video(
+        self,
+        video_path: str,
+        start_time: Optional[float] = None,
+        end_time: Optional[float] = None,
+        fps: Optional[float] = None,
+        max_frames: Optional[float] = None,
+        size: Optional[int] = None,
+        size_divisible: int = 1,
+        precise_time: bool = False,
+        verbose: bool = False,
+        temporal_factor: int = 1
+    ):
+        """
+        Load and process a video file and return the frames and the timestamps of each frame.
+
+        Args:
+            video_path (str): Path to the video file.
+            start_time (float, optional): Start time in seconds. Defaults to None.
+            end_time (float, optional): End time in seconds. Defaults to None.
+            fps (float, optional): Frames per second. Defaults to None.
+            num_frames (float, optional): Number of frames to sample. Defaults to None.
+            size (int, optional): Size of the shortest side. Defaults to None.
+            size_divisible (int, optional): Size divisible by this number. Defaults to 1.
+            precise_time (bool, optional): Whether to use precise time. Defaults to False.
+            verbose (bool, optional): Print ffmpeg output. Defaults to False.
+
+        Returns:
+            frames (List[PIL.Image]): List of frames.
+            timestamps (List[float]): List of timestamps.
+        """
+        if self.use_codec:
+            return self.load_video_with_codec(**locals())
+        fps = self.fps if fps is None else fps
+        max_frames = self.max_frames if max_frames is None else max_frames
+
+        if start_time is not None and end_time is not None and end_time - start_time < 1:
+            return load_video_from_ids(video_path, start_time, end_time, fps=fps, max_frames=max_frames)
+        if os.path.isdir(video_path):
+            return load_video_from_ids(video_path, start_time, end_time, fps=fps, max_frames=max_frames)
+        if video_path.endswith('.gif'):
+            return load_video_from_ids(video_path, start_time, end_time, fps=fps, max_frames=max_frames)
+        probe = ffmpeg.probe(video_path)
+        duration = float(probe['format']['duration'])
+        video_stream = next((stream for stream in probe['streams'] if stream['codec_type'] == 'video'), None)
+        w, h = int(video_stream['width']), int(video_stream['height'])
+
+        kwargs, input_kwargs, output_kwargs = {}, {}, {}
+        do_trim = start_time is not None or end_time is not None
+        if start_time is not None:
+            new_start_time = max(float(video_stream['start_time']), start_time)
+            duration -= new_start_time - start_time
+            start_time = new_start_time
+        else:
+            start_time = float(video_stream['start_time'])
+        if end_time is not None:
+            duration = min(duration, end_time - start_time)
+        else:
+            duration = duration
+        if do_trim:
+            kwargs = {'ss': start_time, 't': duration}
+        if precise_time:
+            output_kwargs.update(kwargs)
+        else:
+            input_kwargs.update(kwargs)
+
+        if size is not None:
+            scale_factor = size / min(w, h)
+            new_w, new_h = round(w * scale_factor), round(h * scale_factor)
+        else:
+            new_w, new_h = w, h
+        new_w = new_w // size_divisible * size_divisible
+        new_h = new_h // size_divisible * size_divisible
+
+        # NOTE: It may result in unexpected number of frames in ffmpeg
+        # if calculate the fps directly according to max_frames
+        # if max_frames is not None and (fps is None or duration * fps > 2 * max_frames):
+        #     fps = round(max_frames / duration * 2)
+
+        stream = ffmpeg.input(video_path, **input_kwargs)
+        if fps is not None:
+            stream = ffmpeg.filter(stream, "fps", fps=fps, round="down")
+        if new_w != w or new_h != h:
+            stream = ffmpeg.filter(stream, 'scale', new_w, new_h)
+        stream = ffmpeg.output(stream, "pipe:", format="rawvideo", pix_fmt="rgb24", **output_kwargs)
+        out, _ = ffmpeg.run(stream, capture_stdout=True, quiet=not verbose)
+
+        frames = np.frombuffer(out, np.uint8).reshape([-1, new_h, new_w, 3]).transpose([0, 3, 1, 2])
+
+        if fps is not None:
+            timestamps = np.arange(start_time, start_time + duration + 1 / fps, 1 / fps)[:len(frames)]
+        else:
+            timestamps = np.linspace(start_time, start_time + duration, len(frames))
+
+        if max_frames is not None and len(frames) > max_frames:
+            indices = np.linspace(0, len(frames) - 1, max_frames, dtype=int)
+            frames = frames[indices]
+            timestamps = timestamps[indices]
+
+        if temporal_factor > 1:
+            pad_length = temporal_factor - len(frames) % temporal_factor
+            frames = np.concatenate([frames, frames[-1:].repeat(pad_length, axis=0)])
+            timestamps = np.concatenate([timestamps, timestamps[-1:].repeat(pad_length) + np.arange(1, pad_length + 1) / fps])
+
+        frames_tensor = torch.from_numpy(frames.copy()).float()
+        frame_types = extract_slow_fast_frames(frames_tensor)
+
+        frames = [frame for frame in frames]
+        timestamps = [timestamp for timestamp in timestamps]
+
+        return frames, timestamps, frame_types
+    
+    def load_video_with_codec(
+        self,
+        video_path: str,
+        start_time: Optional[float] = None,
+        end_time: Optional[float] = None,
+        fps: Optional[float] = None,
+        max_frames: Optional[float] = None,
+        size: Optional[int] = None,
+        size_divisible: int = 1,
+        precise_time: bool = False,
+        verbose: bool = False,
+        temporal_factor: int = 1,
+        slow_fast: bool = True
+    ):
+        """
+        Load a video by prioritizing I-frames (keyframes) and dynamically sampling
+        additional frames between adjacent I-frames up to `max_frames`.
+
+        Notes:
+        - Real codec I-frames (keyframes) are always used as-is and do NOT follow `fps`.
+        - If `fps` is provided, it controls how we sample additional non-I frames between
+        adjacent I-frames (and still respects `max_frames`).
+        - This function does NOT call `load_video_from_ids`.
+
+        Returns:
+            frames: List[np.ndarray] where each is CHW (3, H, W) uint8
+            timestamps: List[float] timestamps in seconds for each returned frame
+            frame_types: List[int] where 0 = I-frame (keyframe), 1 = non-I-frame (sampled)
+        """
+        return_frame_types = slow_fast
+        max_frames = int(max_frames if max_frames is not None else self.max_frames)
+        if max_frames <= 0:
+            return ([], [], []) if return_frame_types else ([], [])
+
+        def _coerce_range(s: Optional[float], e: Optional[float]):
+            if s is not None and e is not None:
+                s = s if s >= 0.0 else 0.0
+                e = e if e >= 0.0 else 0.0
+                if s > e:
+                    s, e = e, s
+                elif s == e:
+                    e = s + 1.0
+            return s, e
+
+        # Fallbacks for non-standard "videos"
+        if os.path.isdir(video_path):
+            # Directory input is a sequence of images; there is no keyframe/I-frame concept.
+            # We mimic `load_video_from_ids` semantics: interpret start/end in seconds using a
+            # small assumed FPS, then uniformly sample up to `max_frames` within that range.
+            start_time, end_time = _coerce_range(start_time, end_time)
+            dir_fps = 3.0
+
+            all_entries = sorted(os.listdir(video_path))
+            frame_files = []
+            for name in all_entries:
+                p = os.path.join(video_path, name)
+                if not os.path.isfile(p):
+                    continue
+                if not name.lower().endswith((".jpg", ".jpeg", ".png", ".bmp", ".webp")):
+                    continue
+                frame_files.append(name)
+
+            if len(frame_files) == 0:
+                return ([], [], []) if return_frame_types else ([], [])
+
+            num_frames_of_video = len(frame_files)
+            f_start = 0 if start_time is None else max(int(start_time * dir_fps) - 1, 0)
+            f_end = (num_frames_of_video - 1) if end_time is None else min(int(end_time * dir_fps) - 1, num_frames_of_video - 1)
+            if f_end < f_start:
+                return ([], [], []) if return_frame_types else ([], [])
+
+            frame_indices = list(range(f_start, f_end + 1))
+            duration = len(frame_indices)
+            sampled = frame_sample(duration, mode="uniform", num_frames=max_frames)
+            sampled_frame_indices = [frame_indices[i] for i in sampled.tolist()]
+
+            frames = []
+            timestamps = []
+            for i in sampled_frame_indices:
+                img = cv2.imread(os.path.join(video_path, frame_files[i]))
+                if img is None:
+                    continue
+                frames.append(cv2.cvtColor(img, cv2.COLOR_BGR2RGB).transpose(2, 0, 1))
+                timestamps.append(float(i) / dir_fps)
+
+            # No keyframe concept for image directories; treat all as non-keyframes.
+            frame_types = [1] * len(frames)
+            return (frames, timestamps, frame_types) if return_frame_types else (frames, timestamps)
+
+        if video_path.endswith('.gif'):
+            gif_reader = imageio.get_reader(video_path)
+            num_frames_of_video = len(gif_reader)
+            if num_frames_of_video == 0:
+                return ([], [], []) if return_frame_types else ([], [])
+            n = min(max_frames, num_frames_of_video)
+            idxs = np.linspace(0, num_frames_of_video - 1, n, dtype=int).tolist()
+            frames = [
+                cv2.cvtColor(frame, cv2.COLOR_RGBA2RGB).transpose(2, 0, 1)
+                for idx, frame in enumerate(gif_reader) if idx in set(idxs)
+            ]
+            # crude timestamps for gif; i-frame concept not applicable
+            timestamps = [float(i) for i in range(len(frames))]
+            # GIF frames are intra-coded; treat them as keyframes.
+            frame_types = [0] * len(frames)
+            return (frames, timestamps, frame_types) if return_frame_types else (frames, timestamps)
+
+        def _get_video_stream_info(path: str):
+            probe = ffmpeg.probe(path)
+            fmt_duration = float(probe["format"]["duration"])
+            vstream = next((st for st in probe["streams"] if st.get("codec_type") == "video"), None)
+            if vstream is None:
+                raise ValueError(f"No video stream found in: {path}")
+            w, h = int(vstream["width"]), int(vstream["height"])
+            stream_start = float(vstream.get("start_time") or 0.0)
+            return probe, vstream, fmt_duration, (w, h), stream_start
+
+        def _safe_float(x) -> Optional[float]:
+            if x is None:
+                return None
+            try:
+                return float(x)
+            except Exception:
+                return None
+
+        def _get_iframe_timestamps(path: str, s: float, e: float) -> List[float]:
+            """
+            Return sorted I-frame timestamps within [s, e].
+            Uses ffprobe with skip_frame=nokey to avoid scanning all frames.
+            """
+            try:
+                p = ffmpeg.probe(
+                    path,
+                    select_streams="v:0",
+                    skip_frame="nokey",
+                    show_frames=None,
+                    show_entries="frame=pict_type,pkt_pts_time,best_effort_timestamp_time,key_frame,pkt_size",
+                    of="json",
+                )
+            except ffmpeg.Error as ex:
+                print("ffprobe keyframe scan failed:", ex)
+                return []
+            frames_meta = p.get("frames") or []
+            out_ts = []
+            for fr in frames_meta:
+                # Prefer pict_type == I; fall back to key_frame == 1 if pict_type missing.
+                pict_type = fr.get("pict_type")
+                is_i = (pict_type == "I") or (pict_type is None and str(fr.get("key_frame")) == "1")
+                if not is_i:
+                    continue
+                ts = _safe_float(fr.get("pkt_pts_time"))
+                if ts is None:
+                    ts = _safe_float(fr.get("best_effort_timestamp_time"))
+                if ts is None:
+                    continue
+                if ts < s or ts > e:
+                    continue
+                size_bytes = int(fr.get("pkt_size", 0))
+                out_ts.append((ts, size_bytes))
+
+            out_ts.sort(key=lambda x: x[0])
+            out_sizes = [x[1] for x in out_ts]
+            return [x[0] for x in out_ts], out_sizes
+
+        def _normalize_uint8_nchw(data: torch.Tensor) -> torch.Tensor:
+            """
+            Ensure tensor is NCHW uint8 on CPU with values in [0, 255].
+            torchcodec may return float in [0,1] or [0,255] depending on backend.
+            """
+            if not isinstance(data, torch.Tensor):
+                raise TypeError(f"Expected torch.Tensor, got {type(data)}")
+            if data.ndim != 4:
+                raise ValueError(f"Expected NCHW tensor, got shape {tuple(data.shape)}")
+            if data.device.type != "cpu":
+                data = data.cpu()
+            if data.dtype != torch.uint8:
+                d = data
+                if d.is_floating_point():
+                    mx = float(d.max().item()) if d.numel() > 0 else 0.0
+                    if mx <= 1.0 + 1e-6:
+                        d = d * 255.0
+                    d = d.round()
+                data = d.clamp(0, 255).to(torch.uint8)
+            return data
+
+
+        def _allocate_remaining_floor_ratio(widths: np.ndarray, remaining: int) -> list[int]:
+            """
+            Allocate `remaining` frames across windows proportionally by window width using floor,
+            without redistributing leftover.
+
+            This matches the spec:
+            - prioritize large I-frame windows
+            - use floor so the sum does not exceed `remaining`
+            """
+            nwin = int(widths.shape[0])
+            if nwin == 0 or remaining <= 0:
+                return [0] * nwin
+            widths = np.maximum(widths.astype(float), 0.0)
+            wsum = float(widths.sum())
+            if wsum <= 0.0:
+                return [0] * nwin
+            alloc = np.floor(float(remaining) * (widths / wsum)).astype(int)
+            # Defensive clamp (should already be <= remaining by construction)
+            s = int(alloc.sum())
+            if s > remaining:
+                # remove extras from smallest windows first
+                order = np.argsort(widths)  # ascending
+                i = 0
+                while s > remaining and i < nwin:
+                    j = int(order[i])
+                    if alloc[j] > 0:
+                        alloc[j] -= 1
+                        s -= 1
+                    else:
+                        i += 1
+            return alloc.tolist()
+
+        def _uniform_inside(a: float, b: float, k: int) -> List[float]:
+            """k points uniformly spaced inside (a, b), excluding endpoints."""
+            if k <= 0:
+                return []
+            if b <= a:
+                return []
+            step = (b - a) / (k + 1)
+            return [a + step * (j + 1) for j in range(k)]
+
+        def _sample_inside_fps(a: float, b: float, fps_val: float) -> List[float]:
+            """Sample points at `fps_val` within (a, b), excluding endpoints."""
+            if fps_val is None:
+                return []
+            try:
+                fps_f = float(fps_val)
+            except Exception:
+                return []
+            if not (fps_f > 0.0):
+                return []
+            if b <= a:
+                return []
+            step = 1.0 / fps_f
+            t = a + step
+            out = []
+            # avoid producing a huge list if `fps` is absurd; we'll downsample anyway,
+            # but keep a reasonable cap based on the window size.
+            # (This cap is still safe because we always keep I-frames.)
+            max_points = int(max(0.0, (b - a) * fps_f)) + 2
+            n = 0
+            while t < b and n < max_points:
+                out.append(float(t))
+                t += step
+                n += 1
+            return out
+
+        start_time, end_time = _coerce_range(start_time, end_time)
+        probe, video_stream, fmt_duration, (w, h), stream_start = _get_video_stream_info(video_path)
+
+        # Use absolute timestamps in seconds.
+        if start_time is None:
+            start_time = float(stream_start)
+        else:
+            start_time = max(float(stream_start), float(start_time))
+
+        if end_time is None:
+            end_time = float(stream_start) + float(fmt_duration)
+        else:
+            end_time = float(end_time)
+
+        if end_time <= start_time:
+            end_time = start_time + 1e-3
+
+        # Output scaling (same logic as `load_video`)
+        if size is not None:
+            scale_factor = size / min(w, h)
+            new_w, new_h = round(w * scale_factor), round(h * scale_factor)
+        else:
+            new_w, new_h = w, h
+        new_w = new_w // size_divisible * size_divisible
+        new_h = new_h // size_divisible * size_divisible
+
+        # 1) Extract all I-frames in [start_time, end_time]
+        iframe_ts, iframe_sizes = _get_iframe_timestamps(video_path, start_time, end_time)
+
+        # 2) Decide timestamps to decode, and frame_types aligned to timestamps
+        timestamps: List[float] = []
+        frame_types: List[int] = []
+
+        if len(iframe_ts) == 0:
+            # No I-frames detected by ffprobe (rare / container oddities). Fall back to uniform time sampling.
+            if end_time <= start_time:
+                return ([], [], []) if return_frame_types else ([], [])
+            if fps is None:
+                n = max_frames
+                timestamps = np.linspace(start_time, end_time, n, endpoint=False, dtype=float).tolist()
+            else:
+                try:
+                    fps_f = float(fps)
+                except Exception:
+                    fps_f = 0.0
+                if fps_f > 0.0:
+                    step = 1.0 / fps_f
+                    timestamps = np.arange(start_time, end_time, step, dtype=float).tolist()
+                    if len(timestamps) > max_frames:
+                        idxs = np.linspace(0, len(timestamps) - 1, max_frames, dtype=int).tolist()
+                        idxs = list(dict.fromkeys(idxs))
+                        timestamps = [timestamps[i] for i in idxs][:max_frames]
+                else:
+                    timestamps = np.linspace(start_time, end_time, max_frames, endpoint=False, dtype=float).tolist()
+            # No I-frames detected; treat all as non-keyframes.
+            frame_types = [1] * len(timestamps)
+        elif len(iframe_ts) >= max_frames:
+            # Too many I-frames: uniformly sample among all available keyframes.
+            idxs = np.linspace(0, len(iframe_ts) - 1, max_frames, dtype=int).tolist()
+            idxs = list(dict.fromkeys(idxs))
+            if len(idxs) != max_frames:
+                missing = max_frames - len(idxs)
+                all_idxs = np.arange(len(iframe_ts), dtype=int).tolist()
+                remain = [i for i in all_idxs if i not in set(idxs)]
+                if len(remain) > 0 and missing > 0:
+                    fill = np.linspace(0, len(remain) - 1, missing, dtype=int).tolist()
+                    idxs.extend([remain[i] for i in fill])
+                idxs = sorted(idxs)[:max_frames]
+            timestamps = [iframe_ts[i] for i in idxs]
+            frame_types = [0] * len(timestamps)
+        else:
+            # Use all I-frames, then allocate remaining between adjacent I-frames.
+            timestamps = list(iframe_ts)
+            frame_types = [0] * len(iframe_ts)
+            remaining = max_frames - len(iframe_ts)
+
+            if len(iframe_ts) >= 2 and remaining > 0:
+                left = np.array(iframe_ts[:-1], dtype=float)
+                right = np.array(iframe_ts[1:], dtype=float)
+
+                widths = (right - left).astype(float)
+                extra_ts: List[float] = []
+                if fps is None:
+                    # Spec: allocate remaining frames by window size ratio using floor (no leftover redistribution).
+                    alloc = _allocate_remaining_floor_ratio(widths, remaining)
+                    for a, b, k in zip(left.tolist(), right.tolist(), alloc):
+                        extra_ts.extend(_uniform_inside(float(a), float(b), int(k)))
+                else:
+                    # Spec: prioritize large windows; sample at fixed fps inside each window until `max_frames` is reached
+                    # or all windows are exhausted.
+                    order = np.argsort(-widths).tolist()  # descending widths
+                    rem = int(remaining)
+                    for j in order:
+                        if rem <= 0:
+                            break
+                        a = float(left[j])
+                        b = float(right[j])
+                        cand = _sample_inside_fps(a, b, fps)
+                        if len(cand) == 0:
+                            continue
+                        if len(cand) > rem:
+                            cand = cand[:rem]
+                        extra_ts.extend(cand)
+                        rem -= len(cand)
+
+                # Drop samples too close to any I-frame timestamp to avoid collisions at decode.
+                if len(extra_ts) > 0:
+                    iframe_set = [float(x) for x in iframe_ts]
+                    def _far_from_iframes(t: float) -> bool:
+                        return all(abs(float(t) - it) > 1e-3 for it in iframe_set)
+                    extra_ts = [t for t in extra_ts if _far_from_iframes(t)]
+
+                timestamps.extend(extra_ts)
+                frame_types.extend([1] * len(extra_ts))
+            elif remaining > 0:
+                # Only 1 I-frame: sample the rest uniformly across the range, avoiding exact collision.
+                if end_time > start_time:
+                    it = float(iframe_ts[0])
+                    if fps is None:
+                        extra_ts = np.linspace(start_time, end_time, remaining + 2, endpoint=True, dtype=float)[1:-1].tolist()
+                    else:
+                        extra_ts = _sample_inside_fps(float(start_time), float(end_time), fps)
+                        # Keep at most `remaining` samples.
+                        if len(extra_ts) > remaining and remaining > 0:
+                            idxs = np.linspace(0, len(extra_ts) - 1, remaining, dtype=int).tolist()
+                            idxs = list(dict.fromkeys(idxs))
+                            extra_ts = [extra_ts[i] for i in idxs][:remaining]
+                        elif remaining <= 0:
+                            extra_ts = []
+
+                    # drop timestamps extremely close to the I-frame timestamp
+                    extra_ts = [t for t in extra_ts if abs(float(t) - it) > 1e-3]
+                    # if we dropped some, refill with tiny offsets (to preserve count behavior)
+                    while len(extra_ts) < remaining:
+                        extra_ts.append(min(end_time, max(start_time, it + 1e-3 * (len(extra_ts) + 1))))
+                    timestamps.extend(extra_ts[:remaining])
+                    frame_types.extend([1] * min(remaining, len(extra_ts)))
+
+            # Sort by time and keep types aligned
+            order = np.argsort(np.array(timestamps, dtype=float)).tolist()
+            timestamps = [float(timestamps[i]) for i in order]
+            frame_types = [int(frame_types[i]) for i in order]
+
+        # 3) Decode frames at chosen timestamps with torchcodec (batch decode).
+        #    We keep the same return format: List[np.ndarray] CHW uint8.
+        if len(timestamps) == 0:
+            return ([], [], []) if return_frame_types else ([], [])
+
+        try:
+            from torchcodec.decoders import VideoDecoder  # type: ignore
+        except Exception as ex:
+            raise ImportError(
+                "torchcodec is required for video decoding in mm_utils.load_video. "
+                "Please install torchcodec (https://github.com/pytorch/torchcodec)."
+            ) from ex
+
+        # if precise_time and verbose:
+        #     # torchcodec selects frames at/around the requested playback times; there's no ffmpeg-style
+        #     # input-vs-output seek mode. We keep the flag for API compatibility.
+        #     print("[mm_utils.load_video_dynamic] note: `precise_time=True` has no special effect with torchcodec.")
+        if not os.path.exists(video_path):
+            raise FileNotFoundError(f"Video file not found: {video_path}")
+        data: torch.Tensor
+        decoder = VideoDecoder(video_path, seek_mode="exact" if precise_time else "approximate")
+        stream_end_time = decoder.metadata.end_stream_seconds
+        stream_start_time = decoder.metadata.begin_stream_seconds
+        # torchcodec accepts list[float] or a torch tensor.
+        if start_time != 0:
+            t_req = [max(stream_start_time + 0.001, min(float(t), stream_end_time - 0.001)) for t in timestamps]
+        else:
+            t_req = [min(float(t), stream_end_time - 0.001) for t in timestamps]
+        try:
+            batch = decoder.get_frames_played_at(torch.tensor(t_req, dtype=torch.float32))
+        except Exception:
+            batch = decoder.get_frames_played_at(t_req)
+
+        raw = getattr(batch, "data", None)
+        if raw is None:
+            raise RuntimeError("torchcodec FrameBatch missing `.data` attribute.")
+        if not isinstance(raw, torch.Tensor):
+            raise RuntimeError(f"torchcodec FrameBatch `.data` is not a torch.Tensor (got {type(raw)}).")
+        data = _normalize_uint8_nchw(raw)
+
+        # Optional resize to match existing `size` / `size_divisible` behavior.
+        _, _, H, W = data.shape
+        if int(new_h) != int(H) or int(new_w) != int(W):
+            data_f = data.to(torch.float32)
+            data_f = torch.nn.functional.interpolate(
+                data_f,
+                size=(int(new_h), int(new_w)),
+                mode="bilinear",
+                align_corners=False,
+            )
+            data = data_f.round().clamp(0, 255).to(torch.uint8)
+
+        n_out = int(data.shape[0])
+        # torchcodec should return 1:1 with requested timestamps, but be defensive.
+        n_keep = min(n_out, len(t_req), len(frame_types))
+        data = data[:n_keep]
+        timestamps = t_req[:n_keep]
+        frame_types = frame_types[:n_keep]
+
+        frames: List[np.ndarray] = [data[i].numpy() for i in range(n_keep)]
+
+        # 4) Temporal padding (keep types aligned)
+        if temporal_factor > 1 and len(frames) > 0:
+            pad_length = (temporal_factor - (len(frames) % temporal_factor)) % temporal_factor
+            if pad_length > 0:
+                if len(timestamps) >= 2:
+                    dt = float(timestamps[-1] - timestamps[-2])
+                    dt = dt if dt > 0 else 1e-3
+                else:
+                    dt = 1e-3
+                for _ in range(pad_length):
+                    frames.append(frames[-1].copy())
+                    timestamps.append(float(timestamps[-1] + dt))
+                    frame_types.append(int(frame_types[-1]))
+
+        return (frames, timestamps, frame_types) if return_frame_types else (frames, timestamps)
+
+    def _load_multimodal_data(self, conversation: Conversation):
+        multimodal_info = defaultdict(list)
+        new_conversation = []
+        for message in conversation:
+            new_message = {"role": message["role"]}
+            if not isinstance(message["content"], (list, tuple)):
+                new_message["content"] = message["content"]
+                new_conversation.append(new_message)
+                continue
+
+            new_contents = []
+            for content in message["content"]:
+                if not isinstance(content, dict):
+                    new_contents.append(content)
+                    continue
+                assert "type" in content, "Content must have 'type' field."
+                if content["type"] in ["image", "video"] and content["type"] in content and isinstance(content[content["type"]], dict):
+                    # TODO: support other types which are not compatible with json
+                    load_args = content[content["type"]]
+                    data_id = json.dumps({k: v for k, v in load_args.items() if not k in ["start_time", "end_time"]})
+                    new_content = copy.deepcopy(content)
+                    multimodal_info[data_id].append(new_content)
+                    new_contents.append(new_content)
+                else:
+                    new_contents.append(content)
+
+            new_message["content"] = new_contents
+            new_conversation.append(new_message)
+
+        for data_id, contents in multimodal_info.items():
+            data_type = contents[0]["type"]
+            if data_type == "image":
+                image = self.load_images(contents[0][data_type]["image_path"])[0]
+                for content in contents:
+                    content["image"] = [image.copy()]
+
+            elif data_type == "video":
+                start_times = [content["video"].get("start_time", 0.) for content in contents]
+                end_times = [content["video"].get("end_time", float("inf")) for content in contents]
+
+                load_args = contents[0][data_type]
+                start_time, end_time = min(start_times), max(end_times)
+                if start_time > 0:
+                    load_args["start_time"] = start_time
+                if end_time < float("inf"):
+                    load_args["end_time"] = end_time
+                images, timestamps, frame_types = self.load_video(**load_args)
+
+                for content, start_time, end_time in zip(contents, start_times, end_times):
+                    cur_images, cur_timestamps, cur_frame_types = [], [], []
+                    for image, timestamp, frame_type in zip(images, timestamps, frame_types):
+                        if start_time <= timestamp <= end_time:
+                            cur_images.append(image.copy())
+                            cur_timestamps.append(timestamp)
+                            cur_frame_types.append(frame_type)
+
+                    content[data_type] = cur_images
+                    content["num_frames"] = len(cur_images)
+                    content["timestamps"] = cur_timestamps
+                    content["frame_types"] = cur_frame_types
+
+        return new_conversation
+
+    def _gather_multimodal_data(self, conversation: Conversation):
+        images = []
+        clip_frame_types = []
+        for message in conversation:
+            if not isinstance(message["content"], (list, tuple)):
+                continue
+            for content in message["content"]:
+                if not isinstance(content, dict):
+                    continue
+                if content["type"] == "video":
+                    video = content["video"]
+                    assert is_valid_video(video), f"Invalid video data: {video}."
+                    images.append(("video", video))
+                    clip_frame_types.append(content.get("frame_types", None))
+                elif content["type"] == "image":
+                    image = content["image"]
+                    images.append(("image", image))
+                    clip_frame_types.append(None)
+        if len(images) == 0:
+            return None, None
+        return images, clip_frame_types
+
+    def _process_conversation_with_label(
+        self,
+        conversation: Conversation,
+        image_inputs: Dict[str, Any],
+        **kwargs,
+    ):
+        assert kwargs.pop("return_tensors", "pt") == "pt", "Only PyTorch tensors are supported when return_labels=True."
+        assert not "add_generation_prompt" in kwargs, "'add_generation_prompt' argument is not supported when return_labels=True."
+
+        output_kwargs = self._merge_kwargs(
+            PenguinVLQwen3ProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+        output_kwargs["chat_template_kwargs"].pop("add_generation_prompt")
+
+        grid_sizes = self._get_downsampled_grid_sizes(image_inputs)
+        text_inputs = {"input_ids": [], "labels": []}
+        sample_types_list = []
+        image_idx = 0
+
+        for message_idx, message in enumerate(conversation):
+            prompt = self.apply_chat_template(
+                [message],
+                tokenize=False,
+                add_generation_prompt=False,
+                **output_kwargs["chat_template_kwargs"],
+            )
+            prompt_chunks = prompt.split(DEFAULT_IMAGE_TOKEN)
+            prompt = []
+            for chunk_idx in range(len(prompt_chunks) - 1):
+                prompt.append(prompt_chunks[chunk_idx])
+                num_tokens = self._get_visual_seq_len(grid_sizes[image_idx])
+                prompt.append(DEFAULT_IMAGE_TOKEN * num_tokens)
+                image_idx += 1
+            prompt.append(prompt_chunks[-1])
+            prompt = "".join(prompt)
+
+            # TODO: support attention_mask, position_ids, etc.
+            input_ids = self.tokenizer.encode(prompt, return_tensors="pt", **output_kwargs["text_kwargs"])[0]
+            text_inputs["input_ids"].append(input_ids)
+
+            targets = torch.full_like(input_ids, IGNORE_INDEX)
+            sample_types = torch.full_like(input_ids, IGNORE_INDEX)
+            if message["role"] == "assistant":
+                targets[self.generation_prompt_length:-1] = input_ids[self.generation_prompt_length:-1].clone()
+            # elif message["role"] == "stream":
+            #     diff = torch.diff((input_ids == self.image_token_id).float())
+            #     image_end_indices = torch.nonzero(diff < 0)[:, 0]
+            #     targets[image_end_indices + 1] = input_ids[image_end_indices + 1]
+            #     sample_types = targets.clone()
+            #     sample_types[torch.logical_and(sample_types > 0, sample_types != self.eos_token_id)] = 0
+            #     targets[-2] = input_ids[-2]    # <|im_end|>
+
+            if message_idx > 0 and conversation[message_idx - 1]["role"] == "stream":
+                targets[0] = input_ids[0]
+                # TODO: consider non-special tokens
+                sample_types[0] = input_ids[0]
+
+            text_inputs["labels"].append(targets)
+            sample_types_list.append(sample_types)
+
+        # Negative sampling for streaming data
+        text_inputs = {k: torch.cat(v) for k, v in text_inputs.items()}
+        sample_types = torch.cat(sample_types_list)
+        types, counts = torch.unique(sample_types[sample_types > -1], return_counts=True)
+
+        if len(types) > 0:
+            target_num_samples = counts.amin()
+            for type_id, type_count in zip(types, counts):
+                if type_count > target_num_samples:
+                    indices = torch.nonzero(sample_types == type_id)[:, 0]
+                    random_selector = torch.randperm(indices.size(0))[:-target_num_samples]
+                    text_inputs["labels"][indices[random_selector]] = IGNORE_INDEX
+                    # sample_types[indices[random_selector]] = -1
+
+        assert len(grid_sizes) == image_idx, "Number of images does not match the number of image tokens in the text."
+
+        return text_inputs
+
+    def _process_conversation_without_label(
+        self,
+        conversation: Conversation,
+        image_inputs: Dict[str, Any],
+        **kwargs,
+    ):
+        output_kwargs = self._merge_kwargs(
+            PenguinVLQwen3ProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+        prompt = self.apply_chat_template(
+            conversation,
+            tokenize=False,
+            **output_kwargs["chat_template_kwargs"],
+        )
+        return self.process_text(prompt, image_inputs, **output_kwargs["text_kwargs"])
+
+    def _process_conversation(
+        self,
+        conversation: Conversation,
+        images: Optional[Union[BatchedImage, BatchedNamedImage]] = None,
+        return_labels: bool = False,
+        **kwargs: Unpack[PenguinVLQwen3ProcessorKwargs],
+    ) -> BatchFeature:
+        assert isinstance(conversation, list), "Conversation must be a list of messages."
+
+        frame_types = None
+        if images is None:
+            conversation = self._load_multimodal_data(conversation)
+            images, frame_types = self._gather_multimodal_data(conversation)
+
+        output_kwargs = self._merge_kwargs(
+            PenguinVLQwen3ProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+
+        if images is not None:
+            image_kwargs = output_kwargs["images_kwargs"]
+            if frame_types is not None:
+                image_kwargs["frame_types"] = frame_types
+            image_inputs = self.process_images(images, **image_kwargs)
+        else:
+            image_inputs = {}
+
+        if return_labels:
+            text_inputs = self._process_conversation_with_label(conversation, image_inputs, **kwargs)
+        else:
+            text_inputs = self._process_conversation_without_label(conversation, image_inputs, **kwargs)
+
+        return BatchFeature(data={**text_inputs, **image_inputs})
+
+    def _process_plain(
+        self,
+        text: Union[TextInput, PreTokenizedInput] = None,
+        images: Optional[Union[BatchedImage, BatchedNamedImage]] = None,
+        return_labels: bool = False,
+        **kwargs: Unpack[PenguinVLQwen3ProcessorKwargs],
+    ) -> BatchFeature:
+        if text is None:
+            raise ValueError("You must provide 'text' or 'message'.")
+        if return_labels:
+            raise ValueError("return_labels is not supported for plain text processing.")
+
+        output_kwargs = self._merge_kwargs(
+            PenguinVLQwen3ProcessorKwargs,
+            tokenizer_init_kwargs=self.tokenizer.init_kwargs,
+            **kwargs,
+        )
+
+        if images is not None:
+            image_inputs = self.process_images(images, **output_kwargs["images_kwargs"])
+        else:
+            image_inputs = {}
+
+        text_inputs = self.process_text(text, image_inputs, **output_kwargs["text_kwargs"])
+
+        return BatchFeature(data={**text_inputs, **image_inputs})
+
+    def process_images(self, images: Union[BatchedImage, BatchedNamedImage], **kwargs):
+        modals, images = make_batched_images(images)
+        if not "merge_size" in kwargs:
+            kwargs["merge_size"] = [
+                self.image_merge_size if modal == "image" else self.video_merge_size
+                for modal in modals
+            ]
+        image_inputs = self.image_processor(images=images, **kwargs)
+        expanded_modals = []
+        for modal, img in zip(modals, images):
+            num_frames = len(img) if is_valid_video(img) else 1
+            expanded_modals.extend([modal] * num_frames)
+        image_inputs["modals"] = expanded_modals
+        return image_inputs
+
+    def process_text(
+        self,
+        text: TextInput,
+        image_inputs: Dict[str, Any],
+        **kwargs,
+    ):
+        grid_sizes = self._get_downsampled_grid_sizes(image_inputs)
+
+        kwargs.pop("padding")
+        kwargs.pop("padding_side")
+
+        image_idx = 0
+        while DEFAULT_IMAGE_TOKEN in text:
+            num_tokens = self._get_visual_seq_len(grid_sizes[image_idx])
+            text = text.replace(DEFAULT_IMAGE_TOKEN, "<placeholder>" * num_tokens, 1)
+            image_idx += 1
+        text = text.replace("<placeholder>", DEFAULT_IMAGE_TOKEN)
+ 
+        assert len(grid_sizes) == image_idx, "Number of images does not match the number of image tokens in the text."
+
+        text_inputs = self.tokenizer(text, **kwargs)
+        return text_inputs
+
+    def __call__(
+        self,
+        text: Optional[TextInput] = None,
+        conversation: Optional[Conversation] = None,
+        images: Optional[Union[BatchedImage, BatchedNamedImage]] = None,
+        return_labels: bool = False,
+        **kwargs: Unpack[PenguinVLQwen3ProcessorKwargs],
+    ) -> BatchFeature:
+        if conversation is not None:
+            if text is not None:
+                raise ValueError("You cannot provide 'message' with 'text'.")
+            return self._process_conversation(conversation, images, return_labels, **kwargs)
+        return self._process_plain(text, images, return_labels, **kwargs)
+
+    def batch_decode(self, *args, **kwargs):
+        return self.tokenizer.batch_decode(*args, **kwargs)
+
+    def decode(self, *args, **kwargs):
+        return self.tokenizer.decode(*args, **kwargs)
+
+    def apply_chat_template(
+        self,
+        conversation: Conversation,
+        chat_template: Optional[str] = None,
+        tokenize: bool = False,
+        add_system_prompt: bool = False,
+        add_generation_prompt: bool = False,
+        add_think_prompt: bool = False,
+        image_token: Optional[str] = DEFAULT_IMAGE_TOKEN,
+        **kwargs,
+    ) -> str:
+        """
+        Similar to the `apply_chat_template` method on tokenizers, this method applies a Jinja template to input
+        conversations to turn them into a single tokenizable string.
+
+        Args:
+            conversation (`List[Dict, str, str]`):
+                The conversation to format.
+            chat_template (`Optional[str]`, *optional*):
+                The Jinja template to use for formatting the conversation. If not provided, the tokenizer's
+                chat template is used.
+            tokenize (`bool`, *optional*, defaults to `False`):
+                Whether to tokenize the output or not.
+            add_system_prompt (`bool`, *optional*, defaults to `False`):
+                Whether to add the system prompt to the output or not.
+            add_generation_prompt (`bool`, *optional*, defaults to `False`):
+                Whether to add the generation prompt to the output or not.
+            image_token (`Optional[str]`, *optional*, defaults to `<image>`):
+                The token to use for indicating images in the conversation.
+            **kwargs:
+                Additional keyword arguments
+        """
+
+        if chat_template is None:
+            if self.chat_template is not None:
+                chat_template = self.chat_template
+            else:
+                raise ValueError(
+                    "No chat template is set for this processor. Please either set the `chat_template` attribute, "
+                    "or provide a chat template as an argument. See "
+                    "https://huggingface.co/docs/transformers/main/en/chat_templating for more information."
+                )
+        return self.tokenizer.apply_chat_template(
+            conversation,
+            chat_template=chat_template,
+            tokenize=tokenize,
+            add_system_prompt=add_system_prompt,
+            add_generation_prompt=add_generation_prompt,
+            add_think_prompt=add_think_prompt,
+            image_token=image_token,
+            **kwargs
+        )
+
+    @property
+    def model_input_names(self):
+        tokenizer_input_names = self.tokenizer.model_input_names
+        image_processor_input_names = self.image_processor.model_input_names
+        return list(dict.fromkeys(tokenizer_input_names + image_processor_input_names)) + ["modals"]
+
+    # modified from transformers.ProcessorMixin
+    def _merge_kwargs(
+        self,
+        ModelProcessorKwargs: ProcessingKwargs,
+        tokenizer_init_kwargs: Optional[Dict] = None,
+        **kwargs,
+    ) -> Dict[str, Dict]:
+        """
+        Method to merge dictionaries of kwargs cleanly separated by modality within a Processor instance.
+        The order of operations is as follows:
+            1) kwargs passed as before have highest priority to preserve BC.
+                ```python
+                high_priority_kwargs = {"crop_size" = {"height": 222, "width": 222}, "padding" = "max_length"}
+                processor(..., **high_priority_kwargs)
+                ```
+            2) kwargs passed as modality-specific kwargs have second priority. This is the recommended API.
+                ```python
+                processor(..., text_kwargs={"padding": "max_length"}, images_kwargs={"crop_size": {"height": 222, "width": 222}}})
+                ```
+            3) kwargs passed during instantiation of a modality processor have fourth priority.
+                ```python
+                tokenizer = tokenizer_class(..., {"padding": "max_length"})
+                image_processor = image_processor_class(...)
+                processor(tokenizer, image_processor) # will pass max_length unless overriden by kwargs at call
+                ```
+            4) defaults kwargs specified at processor level have lowest priority.
+                ```python
+                class MyProcessingKwargs(ProcessingKwargs, CommonKwargs, TextKwargs, ImagesKwargs, total=False):
+                    _defaults = {
+                        "text_kwargs": {
+                            "padding": "max_length",
+                            "max_length": 64,
+                        },
+                    }
+                ```
+        Args:
+            ModelProcessorKwargs (`ProcessingKwargs`):
+                Typed dictionary of kwargs specifically required by the model passed.
+            tokenizer_init_kwargs (`Dict`, *optional*):
+                Dictionary of kwargs the tokenizer was instantiated with and need to take precedence over defaults.
+
+        Returns:
+            output_kwargs (`Dict`):
+                Dictionary of per-modality kwargs to be passed to each modality-specific processor.
+
+        """
+        # Initialize dictionaries
+        output_kwargs = {
+            "text_kwargs": {},
+            "images_kwargs": {},
+            "audio_kwargs": {},
+            "videos_kwargs": {},
+            "chat_template_kwargs": {},
+            "common_kwargs": {},
+        }
+
+        default_kwargs = {
+            "text_kwargs": {},
+            "images_kwargs": {},
+            "audio_kwargs": {},
+            "videos_kwargs": {},
+            "chat_template_kwargs": {},
+            "common_kwargs": {},
+        }
+
+        used_keys = set()
+
+        # get defaults from set model processor kwargs if they exist
+        for modality in default_kwargs:
+            default_kwargs[modality] = ModelProcessorKwargs._defaults.get(modality, {}).copy()
+            # update defaults with arguments from tokenizer init
+            for modality_key in ModelProcessorKwargs.__annotations__[modality].__annotations__.keys():
+                # init with tokenizer init kwargs if necessary
+                if modality_key in tokenizer_init_kwargs:
+                    value = (
+                        getattr(self.tokenizer, modality_key)
+                        if hasattr(self.tokenizer, modality_key)
+                        else tokenizer_init_kwargs[modality_key]
+                    )
+                    default_kwargs[modality][modality_key] = value
+        # now defaults kwargs are updated with the tokenizers defaults.
+        # pass defaults to output dictionary
+        output_kwargs.update(default_kwargs)
+
+        # update modality kwargs with passed kwargs
+        non_modality_kwargs = set(kwargs) - set(output_kwargs)
+        for modality in output_kwargs:
+            for modality_key in ModelProcessorKwargs.__annotations__[modality].__annotations__.keys():
+                # check if we received a structured kwarg dict or not to handle it correctly
+                if modality in kwargs:
+                    kwarg_value = kwargs[modality].pop(modality_key, "__empty__")
+                    # check if this key was passed as a flat kwarg.
+                    if kwarg_value != "__empty__" and modality_key in non_modality_kwargs:
+                        raise ValueError(
+                            f"Keyword argument {modality_key} was passed two times:\n"
+                            f"in a dictionary for {modality} and as a **kwarg."
+                        )
+                elif modality_key in kwargs:
+                    # we get a modality_key instead of popping it because modality-specific processors
+                    # can have overlapping kwargs
+                    kwarg_value = kwargs.get(modality_key, "__empty__")
+                else:
+                    kwarg_value = "__empty__"
+                if kwarg_value != "__empty__":
+                    output_kwargs[modality][modality_key] = kwarg_value
+                    used_keys.add(modality_key)
+
+        # Determine if kwargs is a flat dictionary or contains nested dictionaries
+        if any(key in default_kwargs for key in kwargs):
+            # kwargs is dictionary-based, and some keys match modality names
+            for modality, subdict in kwargs.items():
+                if modality in default_kwargs:
+                    for subkey, subvalue in subdict.items():
+                        if subkey not in used_keys:
+                            output_kwargs[modality][subkey] = subvalue
+                            used_keys.add(subkey)
+        else:
+            # kwargs is a flat dictionary
+            for key in kwargs:
+                if key not in used_keys:
+                    output_kwargs["common_kwargs"][key] = kwargs[key]
+
+        # all modality-specific kwargs are updated with common kwargs
+        for modality in output_kwargs:
+            output_kwargs[modality].update(output_kwargs["common_kwargs"])
+        return output_kwargs

processor_config.json

@@ -0,0 +1,10 @@
+{
+  "auto_map": {
+    "AutoProcessor": "processing_penguinvl.PenguinVLQwen3Processor"
+  },
+  "fps": 1,
+  "image_merge_size": 1,
+  "max_frames": 180,
+  "processor_class": "PenguinVLQwen3Processor",
+  "video_merge_size": 2
+}

special_tokens_map.json

@@ -0,0 +1,31 @@
+{
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "eos_token": {
+    "content": "<|im_end|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3de4265d6c1499ee2f7f7c2ec71004f59d8676ce0373cd32cbad37d40b945cbd
+size 11423788

tokenizer_config.json

@@ -0,0 +1,290 @@
+{
+  "add_bos_token": false,
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "151643": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151644": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151645": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151646": {
+      "content": "<|object_ref_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151647": {
+      "content": "<|object_ref_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151648": {
+      "content": "<|box_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151649": {
+      "content": "<|box_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151650": {
+      "content": "<|quad_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151651": {
+      "content": "<|quad_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151652": {
+      "content": "<|vision_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151653": {
+      "content": "<|vision_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151654": {
+      "content": "<|vision_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151655": {
+      "content": "<|image_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151656": {
+      "content": "<|video_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151657": {
+      "content": "<tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151658": {
+      "content": "</tool_call>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151659": {
+      "content": "<|fim_prefix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151660": {
+      "content": "<|fim_middle|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151661": {
+      "content": "<|fim_suffix|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151662": {
+      "content": "<|fim_pad|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151663": {
+      "content": "<|repo_name|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151664": {
+      "content": "<|file_sep|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151665": {
+      "content": "<tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151666": {
+      "content": "</tool_response>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151667": {
+      "content": "<think>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151668": {
+      "content": "</think>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "151669": {
+      "content": "<image>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151670": {
+      "content": "<|stream_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151671": {
+      "content": "<|stream_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151672": {
+      "content": "<|audio|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151673": {
+      "content": "<|audio_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "151674": {
+      "content": "<|audio_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>",
+    "<|object_ref_start|>",
+    "<|object_ref_end|>",
+    "<|box_start|>",
+    "<|box_end|>",
+    "<|quad_start|>",
+    "<|quad_end|>",
+    "<|vision_start|>",
+    "<|vision_end|>",
+    "<|vision_pad|>",
+    "<|image_pad|>",
+    "<|video_pad|>"
+  ],
+  "bos_token": null,
+  "chat_template": "\n{%- set identifier = 'im' %}\n{% for message in messages %}\n    {% if message['role'] == 'stream' %}\n        {% set identifier = 'stream' %}\n    {% else %}\n        {% set identifier = 'im' %}\n    {% endif %}\n    {% if message['role'] is not none %}\n        {{- '<|' + identifier + '_start|>' + message['role'] + '\n' -}}\n    {% endif %}\n    {% if message['content'] is string %}\n        {{- message['content'] + '<|' + identifier + '_end|>\n' -}}\n    {% else %}\n        {% for content in message['content'] %}\n            {% if content['type'] == 'image' or 'image' in content or 'image_url' in content %}\n                {% if 'time' in content %}\n                    {{- 'Time ' + content['time'] | round(1) | string + 's: ' -}}\n                {% endif %}\n                {{- image_token + '\n' -}}\n            {% elif content['type'] == 'video' or 'video' in content or 'video_url' in content %}\n                {% for i in range(content['num_frames']) %}\n                    {% if 'timestamps' in content and content['timestamps']|length > 0 %}\n                        {{- 'Time ' + content['timestamps'][i] | round(1) | string + 's:' -}}\n                    {% endif %}\n                    {% if i < content['num_frames'] - 1 %}\n                        {{- image_token + ',' -}}\n                    {% else %}\n                        {{- image_token + '\n' -}}\n                    {% endif %}\n                {% endfor %}\n            {% elif content['type'] == 'text' or 'text' in content %}\n                {{- content['text'] -}}\n            {% endif %}\n        {% endfor %}\n        {% if message['role'] is not none %}\n            {{- '<|' + identifier + '_end|>\n' -}}\n        {% endif %}\n    {% endif %}\n{% endfor %}\n{% if add_generation_prompt %}\n    {{- '<|im_start|>assistant\n' -}}\n    {% if not add_think_prompt %}\n        {{- '<think>\n\n</think>\n\n' -}}\n    {% endif %}\n{% endif %}\n",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": {},
+  "model_max_length": 32768,
+  "pad_token": "<|endoftext|>",
+  "padding_side": "right",
+  "processor_class": "PenguinVLQwen3Processor",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null
+}