luanacarolina/gpt2-samsum-lora

GPT-2 SAMSum LoRA Summarizer

This repository contains a LoRA adapter fine-tuned for dialogue summarization on the SAMSum dataset.

The base model is openai-community/gpt2. This repository stores only the PEFT/LoRA adapter weights, not the full GPT-2 model.

Model Details

• Developed by: Luana Carolina Reis
• Model type: LoRA adapter for a causal language model
• Base model: openai-community/gpt2
• Dataset: knkarthick/samsum
• Language: English
• Task: Dialogue summarization
• Training method: Parameter-Efficient Fine-Tuning with LoRA

Intended Use

This adapter is intended for an academic lab assignment on Transformers, language models, and PEFT. It can be used to generate short summaries from messenger-style dialogues.

Prompt format:

Dialogue:
{dialogue}

Summary:

The model is expected to continue the prompt with the summary.

How to Use

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_id = "openai-community/gpt2"
adapter_id = "luanacarolina/gpt2-samsum-lora"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

base_model = AutoModelForCausalLM.from_pretrained(base_model_id)
model = PeftModel.from_pretrained(base_model, adapter_id)
model = model.merge_and_unload()
model.eval()

dialogue = """Amanda: I baked cookies. Do you want some?
Jerry: Sure!
Amanda: I'll bring you tomorrow :-)"""

prompt = f"Dialogue:\n{dialogue.strip()}\n\nSummary:\n"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=96,
        num_beams=4,
        do_sample=False,
        no_repeat_ngram_size=3,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

generated_ids = outputs[0][inputs["input_ids"].shape[-1]:]
summary = tokenizer.decode(generated_ids, skip_special_tokens=True).strip()
print(summary)

For the lab submission, the repository also includes a separate Python file named lora_summarizer_submission.py with:

• load_model()
• generate_summary(model, tokenizer, dialogue) -> str

Training Data

The adapter was fine-tuned on knkarthick/samsum, a dialogue summarization dataset with messenger-like conversations and human-written summaries.

The dataset contains approximately:

• 14,731 training examples
• 818 validation examples
• 819 test examples

Training Procedure

The task was formatted as causal language modeling. Each example was converted to:

Dialogue:
{dialogue}

Summary:
{summary}

During training, prompt tokens were masked with -100, so the loss was computed only on the summary tokens.

LoRA Configuration

• LoRA rank: 8
• LoRA alpha: 16
• LoRA dropout: 0.05
• Target modules: c_attn, c_proj, c_fc
• Trainable parameters: about 1.18M
• Trainable percentage: about 0.94% of GPT-2 parameters

Main Hyperparameters

• Base model: openai-community/gpt2
• Epochs: 1
• Learning rate: 2e-4
• Batch size: 2
• Gradient accumulation steps: 8
• Optimizer: AdamW
• Max sequence length: 768
• Max target length: 128
• Warmup steps: 50

Evaluation

The training script evaluates on the SAMSum validation split and reports validation loss. No additional benchmark metric is provided in this repository.

For qualitative evaluation, users can compare generated summaries against SAMSum reference summaries.

Limitations

• GPT-2 is a relatively small decoder-only model and was not originally designed as a summarization model.
• The model may hallucinate details or repeat text, especially for long or ambiguous dialogues.
• The adapter was trained for English messenger-style dialogues and may not generalize well to other languages or formal documents.
• Generated summaries should be manually reviewed before being used in any important context.

Out-of-Scope Use

This adapter is not intended for high-stakes summarization, legal/medical summaries, or production systems requiring factual guarantees.

Environmental Impact

The adapter was trained locally on a single NVIDIA GeForce RTX 3060 Laptop GPU. The PEFT approach reduces the number of trainable parameters compared with full fine-tuning.

Citation

LoRA:

@article{hu2021lora,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
  journal={arXiv preprint arXiv:2106.09685},
  year={2021}
}

SAMSum:

@article{gliwa2019samsum,
  title={SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization},
  author={Gliwa, Bogdan and Mochol, Iwona and Biesek, Maciej and Wawer, Aleksander},
  journal={arXiv preprint arXiv:1911.12237},
  year={2019}
}