Qwen2.5-Coder-1.5B-LORA-DEEP

This model is a fine-tuned version of Qwen/Qwen2.5-Coder-1.5B-Instruct on the Deep dataset using LoRA (Low-Rank Adaptation). It was developed as a NLP Lecture project to evaluate the capabilities Small Language Models on specific coding tasks.

Model Details

• Model Name: Qwen2.5-Coder-1.5B-LORA-DEEP
• Base Model: Qwen/Qwen2.5-Coder-1.5B-Instruct
• Task: Python Code Generation
• Frameworks: peft, trl, transformers, torch
• Hardware: Trained on Google Colab (T4 GPU)

Training Details

Dataset

The model was fine-tuned on the [Naholav/CodeGen-Deep-5K] dataset.

Hyperparameters

The following hyperparameters were used during training (as per project requirements):

• LoRA Rank (r): 16
• LoRA Alpha: 32
• LoRA Dropout: 0.05
• Learning Rate: 2e-4
• Batch Size: 1 (per device)
• Gradient Accumulation Steps: 16
• Optimizer: AdamW (fused)
• LR Scheduler: Cosine
• Weight Decay: 0.01
• Max Sequence Length: 900 (on previous training attempt, it was 1024. Due to OOM problem, it was set to 900)
• Num Epochs: 3

Evaluation Results

The model achieved the following loss values during training, showing no signs of overfitting:

Metric	Value
Final Training Loss	0.139
Final Validation Loss	0.183

How to Use (Inference)

You can load this model using peft and transformers. Ensure you use device_map="cuda" for GPU usage.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# 1. Load Base Model
base_model_name = "Qwen/Qwen2.5-Coder-1.5B-Instruct"
adapter_model_name = "zeynepelif/Qwen2.5-Coder-1.5B-LORA-DEEP"

base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.float16,
    device_map="cuda"
)
tokenizer = AutoTokenizer.from_pretrained(base_model_name)

# 2. Load LoRA Adapter
model = PeftModel.from_pretrained(base_model, adapter_model_name)
model.to("cuda")
model.eval()

# 3. Generate Code
def ask_model(question):
    messages = [
        {"role": "system", "content": "You are an expert Python programmer."},
        {"role": "user", "content": question}
    ]
    inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to("cuda")
    
    with torch.no_grad():
        outputs = model.generate(inputs, max_new_tokens=512, temperature=0.2, do_sample=True)
    
    print(tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True))

ask_model("Write a Python function to check if a number is palindrome.")