comp2025-qwen3-4b-dpo-merged-fp16-v2

This repository provides a fully merged fp16 model fine-tuned from:

Qwen/Qwen3-4B-Instruct-2507

The model was trained using LoRA-based SFT and DPO, then merged into a standalone fp16 model for stable inference.

This repository contains the merged model weights (fp16).
No adapter loading is required during inference.

---

Base Model

• Qwen/Qwen3-4B-Instruct-2507

The original model license and usage terms apply.

---

Training Pipeline

The model was trained in three stages:

---

1️⃣ Supervised Fine-Tuning (SFT)

Configuration

• Base loading: QLoRA (4-bit) via Unsloth
• Max sequence length: 1024
• Epochs: 2
• Train batch size: 2
• Gradient accumulation: 8
• Learning rate: 5e-6
• Warmup ratio: 0.01
• Weight decay: 0.01
• Scheduler: cosine
• Seed: 3407

LoRA

• r = 64
• alpha = 128
• dropout = 0.0
• target modules:
  • q_proj
  • k_proj
  • v_proj
  • o_proj
  • gate_proj
  • up_proj
  • down_proj

Loss Masking

Loss is applied only to the final assistant output.
Intermediate reasoning (Chain-of-Thought) is masked.

---

2️⃣ Direct Preference Optimization (DPO)

Base Preparation

The SFT adapter is first merged into the base model to create a DPO base model.

Configuration

• Base loading: 4-bit via Unsloth
• Max sequence length: 2048
• Max total length: 1024
• Max prompt length: 512
• Epochs: 1
• Train batch size: 2
• Gradient accumulation: 4
• Learning rate: 3e-7
• Beta: 0.05
• Optimizer: adamw_8bit
• Weight decay: 0.01
• Warmup ratio: 0.1
• Seed: 42

LoRA (DPO stage)

• r = 8
• alpha = 16
• dropout = 0
• Same target modules as SFT

---

3️⃣ Merge

The final DPO LoRA adapter was merged into the base model with:

• torch_dtype = fp16
• device_map = auto
• safe_serialization = True

The merged model is saved in fp16 format.

---

Inference

Execution Environment

• Google Colab (T4 GPU)
• vLLM
• tensor_parallel_size = 1
• enforce_eager = True
• disable_log_stats = True

Retry Strategy

Inference automatically retries with decreasing:

• max_model_len
• max_tokens
• gpu_memory_utilization

until a stable configuration succeeds.

Sampling Parameters

• temperature = 0.0
• max_tokens = up to 4096 (fallback ladder down to 1536)

bf16 inference may fail on T4 (Turing architecture).
fp16 is used for stable inference.

---

Usage (vLLM Example)

from vllm import LLM, SamplingParams

llm = LLM(
    model="TomohitoSaito/comp2025-qwen3-4b-dpo-merged-fp16-v2",
    max_model_len=4096,
    gpu_memory_utilization=0.8,
    tensor_parallel_size=1,
    enforce_eager=True,
)

sampling = SamplingParams(
    temperature=0.0,
    max_tokens=1024,
)

outputs = llm.generate(["Your prompt here"], sampling)
print(outputs[0].outputs[0].text)

---

Intended Use

This model is optimized for:

• Structured output generation
• Deterministic inference (temperature=0.0)
• Competition submission JSON generation

---

License & Compliance

Users must comply with:

• The license of Qwen/Qwen3-4B-Instruct-2507
• Any dataset licenses used during training

This repository contains merged weights only.