rudrakshrakeshzodage/nllb-200-distilled-600M-nf4-4bit-gpu

NLLB-200 Distilled 600M — NF4 4-bit GPU Quantized

🌐 Live Web Demo: Try 100% offline in your browser at huggingface.co/spaces/rudrakshrakeshzodage/nllb-200-onnx-browser-demo

This model is part of a suite of optimized/quantized versions of the base model. Other variants in this direction:

• FP32 (Full Precision / Base): rudrakshrakeshzodage/nllb-200-distilled-600M-fp32
• FP16 (Half Precision): rudrakshrakeshzodage/nllb-200-distilled-600M-fp16
• INT8 (Dynamic Quantization - CPU): rudrakshrakeshzodage/nllb-200-distilled-600M-int8-cpu
• NF4 (4-bit GPU Quantization): rudrakshrakeshzodage/nllb-200-distilled-600M-nf4-4bit-gpu (Current)

Base model: facebook/nllb-200-distilled-600M
Quantization: NF4 4-bit via bitsandbytes
Languages: 200 (all NLLB-200 supported languages)

---

💾 Model Size Reduction Comparison (FP32 vs FP16 vs INT8 vs NF4)

📈 200+ Language Performance Benchmark Chart

Quantization Architecture Details

• Linear Layers: Quantized to NormalFloat4 (NF4) format via bitsandbytes.
• Double Quantization: Enabled (quantizes quantization constants for minimal memory footprint).
• Compute Dtype: torch.float16 for execution on CUDA Tensor Cores.
• Embedding & LM Head: Maintained in FP16 for translation precision.

---

📊 Model Performance & Benchmarks

Metric	NF4 4-bit (GPU)	FP16 Base (GPU)	Speedup / Savings
VRAM Footprint	~1.5 GB	~3.8 GB	60.5% VRAM reduction
Avg Latency (Per Lang)	~806 ms	~3,950 ms	4.9× Faster
Throughput	74 languages / min	~15 languages / min	4.9× Higher
200-Lang Benchmark Run	~2.7 minutes	~13.3 minutes	10.6 min saved
BLEU Score (vs FP16)	~42.0	~42.1	< 0.1 BLEU loss
Loop Anomalies	1 (Central Kanuri)	1 (Central Kanuri)	Identical baseline behavior

---

💻 Hardware Requirements

• GPU: NVIDIA GPU with 3 GB+ VRAM (GTX 1060 6GB minimum)
• RAM: 6 GB System Memory
• Software: CUDA 11.8+, bitsandbytes>=0.39.0, transformers>=4.28.0

---

🚀 Usage

import torch
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForSeq2SeqLM.from_pretrained(
    "YOUR_USERNAME/nllb-200-distilled-600M-nf4-4bit-gpu",
    quantization_config=quant_config,
    device_map="cuda",
)

tokenizer = AutoTokenizer.from_pretrained(
    "YOUR_USERNAME/nllb-200-distilled-600M-nf4-4bit-gpu"
)

# Translate English to Hindi (hin_Deva)
inputs = tokenizer("Hello world, how are you?", return_tensors="pt").to("cuda")
translated_tokens = model.generate(
    **inputs, 
    forced_bos_token_id=tokenizer.lang_code_to_id["hin_Deva"], 
    max_length=100
)
print(tokenizer.batch_decode(translated_tokens, skip_special_tokens=True)[0])

---

📜 Citation & Credits

Quantization research and benchmarking by RudrakshRakeshZodage.