README.md

---
license: apache-2.0
base_model: Qwen/Qwen3-VL-2B-Instruct
language:
  - en
tags:
  - sign-language
  - sign-language-translation
  - asl
  - american-sign-language
  - video
  - video-text-to-text
  - peft
  - lora
  - rslora
datasets:
  - how2sign
  - openasl
library_name: peft
pipeline_tag: video-text-to-text
---

# sign-language-bridge: Qwen3-VL-2B fine-tuned for ASL to English translation

LoRA / RSLoRA fine-tune of
[`Qwen/Qwen3-VL-2B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct)
for continuous American Sign Language (ASL) to English translation.

Checkpoint corresponds to global optimiser step **4,610** (selected on
validation loss). Source code, full technical report, and training pipeline:
[github.com/mamounyosef/sign-language-bridge](https://github.com/mamounyosef/sign-language-bridge).

## Test-set results (How2Sign, 944 clips)

| Metric        | Value   |
|---------------|--------:|
| Test loss     |  2.7896 |
| Perplexity    |  16.28  |
| BLEU-1        |  19.76  |
| BLEU-2        |   6.95  |
| BLEU-4        |   1.64  |
| chrF          |  17.42  |
| ROUGE-L       |  10.43  |
| METEOR        |   9.71  |
| WER (%)       | 112.51  |
| Distinct-2    |   0.103 |

Numbers are reported on a **custom 90/5/5 stratified split**, not the
official How2Sign / OpenASL splits, and are therefore **not directly
comparable to published results** on those corpora. See the
[GitHub repo](https://github.com/mamounyosef/sign-language-bridge) and the
technical report for the full evaluation protocol and the data-cleaning
passes that drove the custom split.

The model produces fluent English in the register of the target captions
and often captures the meaning of the signed input, but the word-level
overlap with the references is modest.

## Repository contents

```text
adapter/
  adapter_config.json          PEFT / LoRA configuration
  adapter_model.safetensors    LoRA weights + saved embedding & output-head modules
  README.md                    PEFT auto-generated card
training_state.pt              optimizer + scheduler states (per tier),
                               InfoNCE projection-head weights,
                               InfoNCE MoCo queues, RNG snapshots,
                               phase / step / epoch bookkeeping
```

`training_state.pt` is required only for resuming training or for reusing
the InfoNCE alignment. It is **not** needed for inference; loading the
`adapter/` folder on top of the base model is sufficient to generate.

## How to use (inference)

```python
import torch
from transformers import AutoProcessor, AutoModelForImageTextToText
from peft import PeftModel

REPO_ID = "mamounyosef/sign-language-bridge"
BASE    = "Qwen/Qwen3-VL-2B-Instruct"

processor = AutoProcessor.from_pretrained(BASE)
base = AutoModelForImageTextToText.from_pretrained(
    BASE, torch_dtype=torch.bfloat16, device_map="auto",
)
model = PeftModel.from_pretrained(base, REPO_ID, subfolder="adapter")
model.eval()

# `video` should be a tensor / list of frames preprocessed by `processor`.
# For best results, replicate the training-time preprocessing:
#   1) pose-guided signer crop  (MediaPipe pose bbox)
#   2) CLAHE on L-channel in LAB (clip limit 2.0, 8x8 tile grid)
#   3) MediaPipe landmark overlay (21 keypoints/hand + 6 upper-body joints)
# See https://github.com/mamounyosef/sign-language-bridge for the exact code.

messages = [{
    "role": "user",
    "content": [
        {"type": "video", "video": video},
        {"type": "text",  "text":  "Translate the signed sentence to English."},
    ],
}]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", tokenize=True,
).to(model.device)

out = model.generate(
    **inputs,
    max_new_tokens=32,
    num_beams=5,
    length_penalty=0.6,
    no_repeat_ngram_size=4,
    repetition_penalty=1.1,
)
print(processor.batch_decode(out, skip_special_tokens=True)[0])
```

## Training summary

- **Base model:** `Qwen/Qwen3-VL-2B-Instruct` (2B parameters: 24-layer vision
  tower, 28-layer Qwen3 decoder, M-RoPE, DeepStack mergers at vision layers
  5 / 11 / 17).
- **Adaptation:** multi-tier LoRA / RSLoRA, **34,321,920 trainable
  parameters (≈1.59%)** of the combined model.
  - T1 (LM attention + MLP): rank 16
  - T2 (Vision encoder): rank 32
  - T3 (Embeddings + output head): rank 8 (plus `modules_to_save`)
  - T4 (InfoNCE projection heads): full-rank, trained from scratch
- **Auxiliary loss:** symmetric InfoNCE between pooled vision and caption
  embeddings (256-dim, τ = 0.07, λ = 0.3 with 200-step linear warmup,
  MoCo-style negative queue of size 64).
- **Schedule:** OpenASL stage (2 epochs, 2,448 steps) → How2Sign stage
  (6 epochs, 3,540 steps). Within OpenASL, Phase 1 (first 20% of steps)
  trains only T2 and T4; Phase 2 unfreezes all four tiers. Per-tier cosine
  LR schedules with a 5% linear warmup.
- **Preprocessing (always-on):** pose-guided signer crop, CLAHE contrast
  enhancement, and pre-extracted MediaPipe landmark overlays
  (21 keypoints / hand + 6 upper-body joints).
- **Compute:** 1× NVIDIA A100 80GB, effective batch size 24
  (per-device 6 × 4 gradient-accumulation steps), bfloat16,
  FlashAttention 2, gradient checkpointing, 8-bit AdamW, Liger fused
  Triton kernels. **Total wall-clock ≈ 4d 18h.**

For full details, see the technical report and source code in the
[GitHub repository](https://github.com/mamounyosef/sign-language-bridge).

## Generation defaults used for evaluation

| Parameter             | Value |
|-----------------------|------:|
| Beam size             |   5   |
| Length penalty        |   0.6 |
| No-repeat n-gram      |   4   |
| Repetition penalty    |   1.1 |
| Max new tokens        |  32   |

## Datasets

- [How2Sign](https://how2sign.github.io/) — multi-view ASL corpus of
  instructional "How To" videos with manually verified English captions.
- [OpenASL](https://github.com/chevalierNoir/OpenASL) — large open-domain
  ASL corpus collected from online video.

Both datasets are subject to their own upstream terms of use. This
repository does **not** redistribute the raw videos.

## Limitations and intended use

- This is a **research preview**, not a production translation system.
  Word-level accuracy is low (BLEU-4 = 1.64, WER = 112.51% on the How2Sign
  test partition); outputs are fluent and often topically appropriate but
  frequently disagree with the reference at the word level.
- The model was trained on a **custom data split**, so reported numbers are
  not directly comparable to published How2Sign / OpenASL results.
- Outputs may be plausibly fluent but **factually wrong** with respect to
  the signed input. Do **not** use this model in any setting where a
  mistranslation could cause harm (medical, legal, safety-critical,
  emergency, etc.).
- The model has been trained almost exclusively on the signers, framings,
  and lighting conditions present in How2Sign and OpenASL, and may
  generalise poorly to out-of-distribution signing.

## License and attribution

- This adapter is released under the **Apache License 2.0**.
- The base model
  [`Qwen/Qwen3-VL-2B-Instruct`](https://huggingface.co/Qwen/Qwen3-VL-2B-Instruct)
  is also under Apache 2.0 ([upstream LICENSE](https://github.com/QwenLM/Qwen3-VL/blob/main/LICENSE)).
  Use of this adapter, together with the base model, remains subject to
  Qwen's Apache 2.0 terms.
- Built using 🤗 [`peft`](https://github.com/huggingface/peft) and
  🤗 [`transformers`](https://github.com/huggingface/transformers).

## Citation

If you use this model or its results, please cite the project repository:

```bibtex
@misc{yosef2026signbridge,
  author       = {Ma'moun Yosef},
  title        = {sign-language-bridge: Fine-Tuning Qwen3-VL-2B for ASL to
                  English Translation},
  year         = {2026},
  howpublished = {\url{https://github.com/mamounyosef/sign-language-bridge}}
}
```

and the base model:

```bibtex
@article{qwen3vl2025,
  author        = {{Qwen Team}},
  title         = {{Qwen3-VL} Technical Report},
  journal       = {arXiv preprint arXiv:2511.21631},
  year          = {2025}
}
```