---
base_model: Qwen/Qwen2.5-7B-Instruct
datasets:
  - FinLang/investopedia-instruction-tuning-dataset
language:
  - en
library_name: peft
pipeline_tag: text-generation
tags:
  - finance
  - lora
  - qwen2.5
  - instruction-tuning
  - investopedia
license: apache-2.0
model-index:
  - name: Qwen2.5-7B Financial LoRA
    results:
      - task:
          type: text-generation
          name: Text Generation
        dataset:
          name: investopedia-instruction-tuning-dataset
          type: FinLang/investopedia-instruction-tuning-dataset
          split: test
        metrics:
          - type: perplexity
            value: 3.33
            name: Perplexity
          - type: bertscore
            value: 0.8986
            name: BERTScore F1
---

# Qwen2.5-7B Financial LoRA Adapter

A LoRA adapter fine-tuned on top of [Qwen/Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct) for financial question answering and explanation tasks. Trained on the full [Investopedia instruction-tuning dataset](https://huggingface.co/datasets/FinLang/investopedia-instruction-tuning-dataset) (206K examples, 1 epoch).

## Model Details

- **Base model:** Qwen/Qwen2.5-7B-Instruct
- **Fine-tuning method:** LoRA (PEFT)
- **Training dataset:** FinLang/investopedia-instruction-tuning-dataset (206,000 examples)
- **Task:** Financial Q&A and instruction following
- **Language:** English
- **License:** Apache 2.0

### LoRA Configuration

| Parameter | Value |
|-----------|-------|
| Rank (r) | 16 |
| Alpha | 32 |
| Dropout | 0 |
| Target modules | q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj |
| Bias | none |

### Training Hyperparameters

| Parameter | Value |
|-----------|-------|
| Learning rate | 2e-4 |
| Batch size (per device) | 1 |
| Gradient accumulation steps | 8 |
| Effective batch size | 8 |
| Epochs | 1 |
| Optimizer | adamw_8bit |
| LR scheduler | linear |
| Precision | bf16 |
| Quantization | 4-bit (NF4) |
| Hardware | Kaggle T4 GPU (x1) |
| Training time | ~24 hours |

## Usage

```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_id = "xczou/qwen2.5-7b-financial-lora"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

def ask(question, max_new_tokens=300):
    prompt = (
        f"<|im_start|>system\nYou are a financial expert.<|im_end|>\n"
        f"<|im_start|>user\n{question}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.7,
            top_p=0.9,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

print(ask("What is the difference between a Roth IRA and a Traditional IRA?"))
```

## Evaluation Results

Evaluated on the test split of `FinLang/investopedia-instruction-tuning-dataset` (100 held-out examples, never seen during training).

### Perplexity (lower is better)

| Model | Perplexity |
|-------|-----------|
| Qwen2.5-7B-Instruct (base) | 84.80 |
| + Financial LoRA adapter | 3.33 |
| **Improvement** | **96.1%** |

### BERTScore F1 (higher is better)

| Model | BERTScore F1 |
|-------|-------------|
| Qwen2.5-7B-Instruct (base) | 0.8373 |
| + Financial LoRA adapter | 0.8986 |
| **Improvement** | **7.3%** |

The large perplexity improvement reflects strong domain language adaptation. The BERTScore improvement reflects that generated answers are semantically closer to expert Investopedia reference answers. The base Qwen2.5-7B-Instruct model already has a high BERTScore baseline (0.8373) due to its broad pre-training, so the 7.3% gain represents meaningful domain specialization on top of an already capable model.

## Scripts

- **Training script:** [qwenadaptertraining on Kaggle](https://www.kaggle.com/code/xczouxiaocheng/qwenadaptertraining)
- **Evaluation script:** [evaluate-qwen-financial-adapter on Kaggle](https://www.kaggle.com/code/xczouxiaocheng/evaluate-qwen-financial-adapter)

## Training Data

Trained on [FinLang/investopedia-instruction-tuning-dataset](https://huggingface.co/datasets/FinLang/investopedia-instruction-tuning-dataset), a dataset of 206,000 financial Q&A pairs sourced from Investopedia, covering topics including stocks, bonds, ETFs, retirement accounts, derivatives, personal finance, and macroeconomics.

## Limitations

- Responses reflect Investopedia's editorial style and may not cover all financial topics equally
- Not suitable for real-time financial data or market predictions
- Should not be used as a substitute for professional financial advice