---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
tags:
  - qwen
  - dpo
  - lora
  - fp16
  - competition
language:
  - en
---

# comp2025-qwen3-4b-dpo-merged-fp16-v2

This repository provides a **fully merged fp16 model** fine-tuned from:

> **Qwen/Qwen3-4B-Instruct-2507**

The model was trained using LoRA-based SFT and DPO, then merged into a
standalone fp16 model for stable inference.

This repository contains the **merged model weights (fp16)**.\
No adapter loading is required during inference.

------------------------------------------------------------------------

## Base Model

-   Qwen/Qwen3-4B-Instruct-2507

The original model license and usage terms apply.

------------------------------------------------------------------------

# Training Pipeline

The model was trained in three stages:

------------------------------------------------------------------------

## 1️⃣ Supervised Fine-Tuning (SFT)

### Configuration

-   Base loading: QLoRA (4-bit) via Unsloth
-   Max sequence length: 1024
-   Epochs: 2
-   Train batch size: 2
-   Gradient accumulation: 8
-   Learning rate: 5e-6
-   Warmup ratio: 0.01
-   Weight decay: 0.01
-   Scheduler: cosine
-   Seed: 3407

### LoRA

-   r = 64
-   alpha = 128
-   dropout = 0.0
-   target modules:
    -   q_proj
    -   k_proj
    -   v_proj
    -   o_proj
    -   gate_proj
    -   up_proj
    -   down_proj

### Loss Masking

Loss is applied only to the final assistant output.\
Intermediate reasoning (Chain-of-Thought) is masked.

------------------------------------------------------------------------

## 2️⃣ Direct Preference Optimization (DPO)

### Base Preparation

The SFT adapter is first merged into the base model to create a DPO base
model.

### Configuration

-   Base loading: 4-bit via Unsloth
-   Max sequence length: 2048
-   Max total length: 1024
-   Max prompt length: 512
-   Epochs: 1
-   Train batch size: 2
-   Gradient accumulation: 4
-   Learning rate: 3e-7
-   Beta: 0.05
-   Optimizer: adamw_8bit
-   Weight decay: 0.01
-   Warmup ratio: 0.1
-   Seed: 42

### LoRA (DPO stage)

-   r = 8
-   alpha = 16
-   dropout = 0
-   Same target modules as SFT

------------------------------------------------------------------------

## 3️⃣ Merge

The final DPO LoRA adapter was merged into the base model with:

-   torch_dtype = fp16
-   device_map = auto
-   safe_serialization = True

The merged model is saved in fp16 format.

------------------------------------------------------------------------

# Inference

### Execution Environment

-   Google Colab (T4 GPU)
-   vLLM
-   tensor_parallel_size = 1
-   enforce_eager = True
-   disable_log_stats = True

### Retry Strategy

Inference automatically retries with decreasing:

-   max_model_len
-   max_tokens
-   gpu_memory_utilization

until a stable configuration succeeds.

### Sampling Parameters

-   temperature = 0.0
-   max_tokens = up to 4096 (fallback ladder down to 1536)

bf16 inference may fail on T4 (Turing architecture).\
fp16 is used for stable inference.

------------------------------------------------------------------------

# Usage (vLLM Example)

``` python
from vllm import LLM, SamplingParams

llm = LLM(
    model="TomohitoSaito/comp2025-qwen3-4b-dpo-merged-fp16-v2",
    max_model_len=4096,
    gpu_memory_utilization=0.8,
    tensor_parallel_size=1,
    enforce_eager=True,
)

sampling = SamplingParams(
    temperature=0.0,
    max_tokens=1024,
)

outputs = llm.generate(["Your prompt here"], sampling)
print(outputs[0].outputs[0].text)
```

------------------------------------------------------------------------

# Intended Use

This model is optimized for:

-   Structured output generation
-   Deterministic inference (temperature=0.0)
-   Competition submission JSON generation

------------------------------------------------------------------------

# License & Compliance

Users must comply with:

-   The license of Qwen/Qwen3-4B-Instruct-2507
-   Any dataset licenses used during training

This repository contains merged weights only.