qwen2.5-0.5b-sft-merged-cnndm-50k

Task: abstractive text summarization — given a news article, generate a short summary (2–3 sentences).

Merged full model on the Hub: Qwen/Qwen2.5-0.5B fine-tuned with LoRA on gold CNN/DailyMail summaries, then fused with merge_and_unload(). Load directly via AutoModelForCausalLM.from_pretrained(this_repo) — no PEFT or separate base model.

Training objective: supervised fine-tuning on human gold summaries (gold_summary from CNN/DailyMail highlights).

Paired KD merged model: Harsha901/qwen2.5-0.5b-kd-merged-cnndm-50k

---

Training (notebook 04 — 50k run)

Item	Value
Task	Abstractive summarization (article → summary)
Base model	Qwen/Qwen2.5-0.5B
Training file	teacher_generations.jsonl (50,000 records)
Label field	gold_summary
Train / val split	49,500 train · 500 val (seed 42)
Source domain	CNN/DailyMail (abisee/cnn_dailymail, config 3.0.0)
Published artifact	LoRA trained → merged into base → pushed as this repo

KD and SFT share the same articles, prompt, and hyperparameters — only the summary target differs.

Hyperparameters

Setting	Value
LoRA	r=32, alpha=64, dropout=0.1 (all attn + MLP projections)
Epochs	3 max · early stopping patience=3 on eval_loss
Batch	4 × 4 grad accum → effective 16
LR	2e-4 cosine, 3% warmup · adamw_torch_fused
Max seq length	1536 · articles left-truncated at 6000 chars
Loss	Completion-only on assistant summary tokens

Summarization prompt

Role	Content
System	Concise news summarizer, 2–3 sentences, no preamble
User	Article:\n{article}\n\nSummary:

---

Evaluation (notebook 06 — merged model)

Greedy vLLM · preamble-stripped ROUGE · BERTScore F1 (rescaled) · 1000 examples per dataset.

ROUGE-1 / ROUGE-2 (clean)

Benchmark	Type	Teacher 7B	Base 0.5B	KD merged	SFT merged (this)
CNN/DailyMail	in-domain	33.41 / 10.54	8.62 / 1.88	31.36 / 8.99	35.83 / 12.64
XSum	cross	27.22 / 7.16	11.43 / 1.07	21.46 / 3.92	20.02 / 2.86
SAMSum	cross	38.11 / 12.76	11.66 / 1.78	26.12 / 5.06	24.31 / 4.77
DialogSum	cross	30.45 / 8.86	15.00 / 3.07	21.50 / 4.18	19.76 / 3.82

BERTScore · preamble (CNN/DM)

	KD merged	SFT merged (this)
CNN/DM BERTScore	19.82	24.07
CNN/DM preamble rate	3.7%	66.7%

Best in-domain student on CNN/DM ROUGE-1 (35.83). For cross-domain dialogue/news, use the KD merged repo.

---

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

MODEL_ID = "Harsha901/qwen2.5-0.5b-sft-merged-cnndm-50k"

SYSTEM = (
    "You are a concise news summarizer. Write a short summary of the article in 2-3 sentences. "
    "Output only the summary itself, with no preamble, headers, or commentary."
)

tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model.eval()

article = "..."
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"Article:\n{article}\n\nSummary:"},
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    output_ids = model.generate(**inputs, max_new_tokens=160, do_sample=False)

print(tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip())

---

Reproducibility

Step	Notebook
LoRA SFT training	kd_vs_sft_50k_cnn_dailymail/notebooks/04_sft_lora_finetune_Qwen2.5-0.5B-student.ipynb
Merge + eval	kd_vs_sft_50k_cnn_dailymail/notebooks/06_eval_with_bertscore_and_genfix_50k.ipynb

Metrics: kd_vs_sft_50k_cnn_dailymail/kd_vs_sft_eval_results/all_results.csv