xczou/qwen2.5-7b-financial-lora

Qwen2.5-7B Financial LoRA Adapter

A LoRA adapter fine-tuned on top of Qwen/Qwen2.5-7B-Instruct for financial question answering and explanation tasks. Trained on the full Investopedia instruction-tuning dataset (206K examples, 1 epoch).

Model Details

• Base model: Qwen/Qwen2.5-7B-Instruct
• Fine-tuning method: LoRA (PEFT)
• Training dataset: FinLang/investopedia-instruction-tuning-dataset (206,000 examples)
• Task: Financial Q&A and instruction following
• Language: English
• License: Apache 2.0

LoRA Configuration

Parameter	Value
Rank (r)	16
Alpha	32
Dropout	0
Target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Bias	none

Training Hyperparameters

Parameter	Value
Learning rate	2e-4
Batch size (per device)	1
Gradient accumulation steps	8
Effective batch size	8
Epochs	1
Optimizer	adamw_8bit
LR scheduler	linear
Precision	bf16
Quantization	4-bit (NF4)
Hardware	Kaggle T4 GPU (x1)
Training time	~24 hours

Usage

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

base_model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_id = "xczou/qwen2.5-7b-financial-lora"

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

tokenizer = AutoTokenizer.from_pretrained(base_model_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True,
)

model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

def ask(question, max_new_tokens=300):
    prompt = (
        f"<|im_start|>system\nYou are a financial expert.<|im_end|>\n"
        f"<|im_start|>user\n{question}<|im_end|>\n"
        f"<|im_start|>assistant\n"
    )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            temperature=0.7,
            top_p=0.9,
            do_sample=True,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)

print(ask("What is the difference between a Roth IRA and a Traditional IRA?"))

Evaluation Results

Evaluated on the test split of FinLang/investopedia-instruction-tuning-dataset (100 held-out examples, never seen during training).

Perplexity (lower is better)

Model	Perplexity
Qwen2.5-7B-Instruct (base)	84.80
+ Financial LoRA adapter	3.33
Improvement	96.1%

BERTScore F1 (higher is better)

Model	BERTScore F1
Qwen2.5-7B-Instruct (base)	0.8373
+ Financial LoRA adapter	0.8986
Improvement	7.3%

The large perplexity improvement reflects strong domain language adaptation. The BERTScore improvement reflects that generated answers are semantically closer to expert Investopedia reference answers. The base Qwen2.5-7B-Instruct model already has a high BERTScore baseline (0.8373) due to its broad pre-training, so the 7.3% gain represents meaningful domain specialization on top of an already capable model.

Scripts

• Training script: qwenadaptertraining on Kaggle
• Evaluation script: evaluate-qwen-financial-adapter on Kaggle

Training Data

Trained on FinLang/investopedia-instruction-tuning-dataset, a dataset of 206,000 financial Q&A pairs sourced from Investopedia, covering topics including stocks, bonds, ETFs, retirement accounts, derivatives, personal finance, and macroeconomics.

Limitations

• Responses reflect Investopedia's editorial style and may not cover all financial topics equally
• Not suitable for real-time financial data or market predictions
• Should not be used as a substitute for professional financial advice