license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
tags:
- qwen
- dpo
- lora
- fp16
- competition
language:
- en
comp2025-qwen3-4b-dpo-merged-fp16-v2
This repository provides a fully merged fp16 model fine-tuned from:
Qwen/Qwen3-4B-Instruct-2507
The model was trained using LoRA-based SFT and DPO, then merged into a standalone fp16 model for stable inference.
This repository contains the merged model weights (fp16).
No adapter loading is required during inference.
Base Model
- Qwen/Qwen3-4B-Instruct-2507
The original model license and usage terms apply.
Training Pipeline
The model was trained in three stages:
1️⃣ Supervised Fine-Tuning (SFT)
Configuration
- Base loading: QLoRA (4-bit) via Unsloth
- Max sequence length: 1024
- Epochs: 2
- Train batch size: 2
- Gradient accumulation: 8
- Learning rate: 5e-6
- Warmup ratio: 0.01
- Weight decay: 0.01
- Scheduler: cosine
- Seed: 3407
LoRA
- r = 64
- alpha = 128
- dropout = 0.0
- target modules:
- q_proj
- k_proj
- v_proj
- o_proj
- gate_proj
- up_proj
- down_proj
Loss Masking
Loss is applied only to the final assistant output.
Intermediate reasoning (Chain-of-Thought) is masked.
2️⃣ Direct Preference Optimization (DPO)
Base Preparation
The SFT adapter is first merged into the base model to create a DPO base model.
Configuration
- Base loading: 4-bit via Unsloth
- Max sequence length: 2048
- Max total length: 1024
- Max prompt length: 512
- Epochs: 1
- Train batch size: 2
- Gradient accumulation: 4
- Learning rate: 3e-7
- Beta: 0.05
- Optimizer: adamw_8bit
- Weight decay: 0.01
- Warmup ratio: 0.1
- Seed: 42
LoRA (DPO stage)
- r = 8
- alpha = 16
- dropout = 0
- Same target modules as SFT
3️⃣ Merge
The final DPO LoRA adapter was merged into the base model with:
- torch_dtype = fp16
- device_map = auto
- safe_serialization = True
The merged model is saved in fp16 format.
Inference
Execution Environment
- Google Colab (T4 GPU)
- vLLM
- tensor_parallel_size = 1
- enforce_eager = True
- disable_log_stats = True
Retry Strategy
Inference automatically retries with decreasing:
- max_model_len
- max_tokens
- gpu_memory_utilization
until a stable configuration succeeds.
Sampling Parameters
- temperature = 0.0
- max_tokens = up to 4096 (fallback ladder down to 1536)
bf16 inference may fail on T4 (Turing architecture).
fp16 is used for stable inference.
Usage (vLLM Example)
from vllm import LLM, SamplingParams
llm = LLM(
model="TomohitoSaito/comp2025-qwen3-4b-dpo-merged-fp16-v2",
max_model_len=4096,
gpu_memory_utilization=0.8,
tensor_parallel_size=1,
enforce_eager=True,
)
sampling = SamplingParams(
temperature=0.0,
max_tokens=1024,
)
outputs = llm.generate(["Your prompt here"], sampling)
print(outputs[0].outputs[0].text)
Intended Use
This model is optimized for:
- Structured output generation
- Deterministic inference (temperature=0.0)
- Competition submission JSON generation
License & Compliance
Users must comply with:
- The license of Qwen/Qwen3-4B-Instruct-2507
- Any dataset licenses used during training
This repository contains merged weights only.