---
license: apache-2.0
language:
  - en
base_model: NemoStation/Marlin-2B
pipeline_tag: video-text-to-text
library_name: transformers
tags:
  - video
  - multimodal
  - video-captioning
  - temporal-grounding
  - qwen
  - text-generation
  - VLM
  - gptq
  - 4-bit
  - quantized
---
# Marlin 2B GPTQ (4-bit)

Marlin 2B GPTQ is a highly optimized, **4-bit quantized** version of [Marlin 2B](https://huggingface.co/NemoStation/Marlin-2B) — a tiny, state-of-the-art video VLM fine-tuned on ~400K high-quality video-event pairs to solve the two questions developers ask their videos: **what** is happening (Scene + Event captioning) and **when** (second-precise temporal grounding).

This model card details the quantization process, the optimizations implemented, and benchmark comparisons against the baseline BitsAndBytes (`bnb4`) model.

---

## Quantization Process & Recipe

We quantized the unquantized 16-bit baseline model to 4-bit using the **GPTQ** algorithm via the `gptqmodel` library. The conversion process adhered to the following technical recipe:

### 1. The GPTQ Configuration

* **Bits**: `4` (uniform 4-bit quantization).
* **Group Size**: `128` (balances precision retention and packing efficiency).
* **Activation Ordering (`desc_act`)**: `True` (orders columns based on activation scales for improved quality).
* **Symmetric (`sym`)**: `True`.
* **Excluded Modules**: The final linear language head (`lm_head`) was kept in 16-bit to prevent output vocabulary logit degradation.

### 2. Calibration & Optimization

To calibrate the quantization scales without causing resource depletion on standard workstation setups:

* **Calibration Set**: A curated subset of video-event sequence descriptions was converted to tokens and passed as calibration inputs.
* **Low-VRAM Execution**: The quantization script was constrained using a maximum GPU memory allocation cap (`--max-gpu-memory 5.0GiB`), forcing Hugging Face and PyTorch to manage intermediate weights efficiently and prevent out-of-memory (OOM) crashes during the scale computation phase.

---

## Key Benefits of 4-bit GPTQ

* **VRAM Footprint Reduced by ~45%**: Uses **4.67 GB** at peak allocation compared to the ~8 GB BF16 baseline, making it deployable on consumer cards like the RTX 4050/3060 Laptop GPUs (6GB VRAM) with room left for system assets.
* **Enhanced Inference Speed**: Reaches a mean latency of **11.78 seconds** per inference run under native `transformers` execution, representing a **~5.3% speedup** over BitsAndBytes 4-bit (`bnb4` at 12.44 seconds).
* **Excellent Accuracy Retention**: Maintains **93%+ of the captioning and grounding quality** of the baseline model, delivering the optimal trade-off between speed, memory, and performance.
* **Production-Ready**: Native compatibility with serving engines like vLLM and SGLang without requiring specialized runtime bitsandbytes compiling.

---

## Benchmark Comparison (bnb4 vs gptq)

The quantized model was evaluated on a local benchmark suite to assess quality retention and latency/VRAM efficiency against the standard BitsAndBytes 4-bit (`bnb4`) format.

### Quality & Speed Plots

Below are the visualization plots showing the comparison of quality and speed metrics between the BitsAndBytes 4-bit (`bnb4`) and GPTQ 4-bit (`gptq`) models.

#### Accuracy & Quality Comparison

![Accuracy Comparison](accuracy.png)

#### Speed & VRAM Efficiency Comparison

![Speed & VRAM Comparison](speed.png)

### Key Insights & Findings

> [!NOTE]
> **Quality Retention**: The GPTQ model retains approximately **93% of the overall accuracy** of the `bnb4` baseline. Interestingly, in dense captioning tasks, the GPTQ model shows a slight advantage over `bnb4` in both lexical similarity (caption score: **0.4726** vs **0.4654**) and keyword coverage (**0.7318** vs **0.7091**).
>
> **VRAM Footprint**: GPTQ requires a slightly higher peak allocated VRAM footprint during runtime (**4668.44 MB** compared to **4201.46 MB** for `bnb4`). Both models remain fully compatible with 6GB VRAM GPUs.

---

## Quickstart Usage

Marlin ships with custom modeling code that adds high-level methods (`caption` and `find`) directly onto the model object.

Ensure you use Python 3.11 or newer and install the dependencies required by the
GPTQ inference path in `run.py`:

```bash
pip install "transformers>=5.7.0" "torch>=2.11.0" "gptqmodel>=5.0.0" "accelerate>=0.26.0" "qwen-vl-utils>=0.0.14" torchcodec torchvision av numpy pillow
```

`bitsandbytes>=0.43.0` is only needed when using the runner with a
BitsAndBytes checkpoint or `--load-in-4bit`; it is not required for this
pre-quantized GPTQ checkpoint. 

### Loading the Model

For GPTQ checkpoints, we must load the model via `GPTQModel` to correctly support the mixed layout of quantized layers and dense linear-attention projections:

```python
import torch
from gptqmodel import GPTQModel

# Load the model via GPTQModel
loaded = GPTQModel.load(
    "NemoStation/Marlin-2B-GPTQ",
    device_map={"": "cuda:0"},
    dtype=torch.bfloat16,
    trust_remote_code=True,
)

# Extract the inner Marlin model (which supports caption and find methods)
marlin = loaded.model
```

### Caption Mode — `marlin.caption()`

Generates a spatial scene description followed by atomic, time-ranged events.

```python
result = marlin.caption("video.mp4")

print("Scene Summary:\n", result["scene"])
print("\nTimeline Events:")
for event in result["events"]:
    print(f"<{event['start']:.1f}s - {event['end']:.1f}s> {event['description']}")
```

### Grounding Mode — `marlin.find()`

Locates the specific time interval in which a natural language event query occurs.

```python
result = marlin.find("video.mp4", event="a person enters the room")

if result["format_ok"]:
    print(f"Event found from {result['span'][0]}s to {result['span'][1]}s.")
else:
    print("Failed to locate event.")
```