DistilBERT for AG News — LoRA Adapter (PEFT)
This repository contains LoRA adapter weights trained on AG News for DistilBERT.
Results (reference)
| Model | Test Accuracy | Macro F1 |
|---|---|---|
| LoRA (merged) | 0.9400 | 0.9400 |
Confusion Matrix (Merged, Test)
World Sports Busines Sci/Tec
World 1791 14 49 46
Sports 12 1874 9 5
Busines 42 8 1718 132
Sci/Tec 30 9 100 1761
Usage (attach to base)
from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline
from peft import PeftModel
base = "distilbert-base-uncased"
adapter = "starkdv123/agnews-distilbert-lora"
tok = AutoTokenizer.from_pretrained(base)
base_model = AutoModelForSequenceClassification.from_pretrained(base, num_labels=4)
model = PeftModel.from_pretrained(base_model, adapter)
clf = pipeline("text-classification", model=model, tokenizer=tok, truncation=True)
clf(["New AI chip announced for smartphones."])
Training (summary)
- LoRA: r=8, α=16, dropout=0.05, LR=2e-4, epochs=2, batch=16, max_len=256
- Targets: [q_lin, k_lin, v_lin, out_lin]
Author
Karan D Vasa — https://huggingface.co/starkdv123