---
language: en
license: apache-2.0
pipeline_tag: summarization
base_model: nsi319/legal-led-base-16384
tags:
- summarization
- legal
- led
- longformer
- long-document
- billsum
- abstractive-summarization
- finetuned
- legal-nlp
datasets:
- FiscalNote/billsum
metrics:
- rouge
- bertscore
---

# 📘 Legal LED – Long-Document BillSum Summarizer  
**Fine-tuned version of NSI’s Legal LED for summarization of long legal and legislative documents.**

This model fine-tunes **`nsi319/legal-led-base-16384`**, a legally pretrained LED (Longformer-Encoder-Decoder) model with a 16k token context window.  
Legal LED is specifically adapted to legal corpora such as case law, statutes, regulatory materials, and legislative documents — making it more reliable than the vanilla LED for legal NLP.

This fine-tuned version is optimized for summarizing **long and complex legal text** such as US bills, policy documents, and multi-section legislative structures.

---

# 🧠 Base Model

This model extends:

👉 **[nsi319/legal-led-base-16384](https://huggingface.co/nsi319/legal-led-base-16384)**

The base model has:

- Longformer sparse attention for **16,384-token** sequences  
- Legal-domain pretraining on:  
  - court judgments  
  - legislation  
  - legal commentary  
  - regulatory filings  
- Strong domain adaptation prior to fine-tuning

This gives LED excellent performance on structural legal documents.

---

# 📚 Fine-Tuning Dataset

- **BillSum** (US Congress + California bills)  
- Additional cleaned legal-style summaries  
- Documents ranged from **3k to 30k tokens**

---

# ⚙️ Training Configuration

| Setting | Value |
|--------|--------|
| Base model | nsi319/legal-led-base-16384 |
| Epochs | 6 total |
| Batch size | 2 |
| Gradient accumulation | 2 |
| Learning rate | 1e-5 |
| Optimizer | AdamW |
| Weight decay | 0.01 |
| FP16 | Yes |
| Warmup steps | 500 |
| Max input length | 4096 tokens |
| Max output length | 512 tokens |
| Attention | Global attention on first token |
| Scheduler | Linear |

Training was performed on **NVIDIA P100 (16GB VRAM)** via Kaggle.

---

# 🧪 Evaluation Metrics

### **Training Progress**
| Epoch | Training Loss | Validation Loss |
|-------|----------------|------------------|
| 1 | 1.39 | 1.33 |
| 2 | 1.17 | 1.26 |
| 3 | 1.19 | 1.23 |
| 4 | 1.15 | 1.18 |
| 5 | 1.03 | 1.16 |
| 6 | 1.02 | 1.16 |

### **ROUGE (document test set)**  
| Metric | F1 |
|--------|------|
| ROUGE-1 | 0.5179 |
| ROUGE-2 | 0.3432 |
| ROUGE-L | 0.4067 |

### **BERTScore**  
| Metric | Score |
|--------|--------|
| Precision | 0.9015 |
| Recall | 0.8868 |
| F1 | 0.8936 |

---

# 🏗️ Long-Document Summarization Strategy

Legal LED supports long contexts (~16k tokens), but many legal bills exceed that.  
To summarize documents up to **30k tokens**, this pipeline was used:

- Length-adaptive chunking  
- Paragraph grouping  
- Sliding-window segmentation  
- Chunk-wise LED summarization  
- Top-K reranking using BERTScore  
- Final second-pass LED rewriting  

This improves semantic cohesion and section preservation.

---

# 📌 Intended Use

Ideal for:

- Legislative document summarization  
- Legal policy analysis  
- Long-form legal NLP applications  
- AI assistants for lawyers or students  
- Preprocessing for legal research systems  

---

# ⚠️ Limitations

- English only  
- Requires chunking for documents >16k tokens  
- May simplify definitions too aggressively  
- Not suitable for citation extraction or case-law reasoning  
- Not intended for legal decision-making  

---

# 🔧 Usage Example

```python
from transformers import AutoTokenizer, LEDForConditionalGeneration
import torch

model_name = "Anurag33Gaikwad/legal-led-billsum-summarization"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = LEDForConditionalGeneration.from_pretrained(model_name)

text = """Your long legal or legislative document here..."""

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=4096,
)

# LED requires global attention on the first token
global_attention_mask = torch.zeros_like(inputs["input_ids"])
global_attention_mask[:, 0] = 1

summary_ids = model.generate(
    inputs["input_ids"],
    global_attention_mask=global_attention_mask,
    num_beams=5,
    max_length=512,
    early_stopping=True
)

print(tokenizer.decode(summary_ids[0], skip_special_tokens=True))