---
language:
- en
tags:
- text-classification
- financial-sentiment
- knowledge-distillation
- patient-knowledge-distillation
- albert
- finbert
- finance
- nlp
- distillation
license: mit
datasets:
  - financial_phrasebank
base_model: albert-base-v2
model-index:
- name: pkd-albert-student
  results:
    - task:
        type: text-classification
        name: Financial Sentiment Analysis
      dataset:
        name: Financial PhraseBank (100% agreement)
        type: financial_phrasebank
        split: test
      metrics:
      - type: accuracy
        value: 0.9735
        name: Test Accuracy
      - type: f1
        value: 0.9650
        name: Macro F1
---
 
# PKD-ALBERT: Lightweight Financial Sentiment Classifier via Patient Knowledge Distillation
 
[![Demo on Spaces](https://img.shields.io/badge/🤗%20Spaces-Live%20Demo-blue)](https://huggingface.co/spaces/hadangvu/pkd-sentiment-api)
[![API](https://img.shields.io/badge/API-/predict-green)](https://huggingface.co/spaces/hadangvu/pkd-sentiment-api)

 ---
 
## Model Summary
 
**PKD-ALBERT** is a lightweight financial sentiment classifier distilled from
[ProsusAI/finbert](https://huggingface.co/ProsusAI/finbert) using a two-stage
**Patient Knowledge Distillation (PKD)** pipeline.
 
It classifies financial text — headlines, earnings excerpts, news snippets — into
**positive**, **neutral**, or **negative** sentiment, achieving **97.4% accuracy**
and **0.965 Macro-F1** on Financial PhraseBank while using **~10× fewer parameters**
than the teacher model.
 
<table>
  <thead>
    <tr>
      <th></th>
      <th>Teacher (FinBERT)</th>
      <th>Student (PKD-ALBERT)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><strong>Parameters</strong></td>
      <td>109.5M</td>
      <td>11.7M</td>
    </tr>
    <tr>
      <td><strong>Model size</strong></td>
      <td>417.7 MB</td>
      <td>44.6 MB</td>
    </tr>
    <tr>
      <td><strong>Test Accuracy</strong></td>
      <td>97.6%</td>
      <td><strong>97.4%</strong></td>
    </tr>
    <tr>
      <td><strong>Macro F1</strong></td>
      <td>0.9696</td>
      <td><strong>0.9650</strong></td>
    </tr>
    <tr>
      <td><strong>Inference (ms/doc)</strong></td>
      <td>1.49 ms</td>
      <td>2.02 ms</td>
    </tr>
    <tr>
      <td><strong>Accuracy drop</strong></td>
      <td>—</td>
      <td><strong>−0.2%</strong></td>
    </tr>
  </tbody>
</table>
 
> The student retains **99.8% of teacher accuracy** at **89% less disk space**.
 
---


## Try It Live
 
A live Gradio demo is available on Hugging Face Spaces. Paste any financial
headline or sentence and receive a sentiment label with a confidence score.
 
👉 **[Open Live Demo →](https://huggingface.co/spaces/hadangvu/pkd-sentiment-api)**
 
**Example inputs from the held-out test set:**

<table>
  <thead>
    <tr>
      <th>Input</th>
      <th>Prediction</th>
      <th>Confidence</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td><em>"Charles Schwab price target raised to $121 from $119 at JPMorgan."</em></td>
      <td>✅ Positive</td>
      <td>93.1%</td>
    </tr>
    <tr>
      <td><em>"Costco assumed with a Peer Perform at Wolfe Research."</em></td>
      <td>➖ Neutral</td>
      <td>92.2%</td>
    </tr>
    <tr>
      <td><em>"IBM Explains How AI Models Are Making a Familiar Human Mistake."</em></td>
      <td>❌ Negative</td>
      <td>90.1%</td>
    </tr>
  </tbody>
</table>

 ---
 
## API Usage
 
The `/predict` endpoint accepts raw text and returns a label and confidence score.
 
```python
import requests
 
API_URL = "https://hadangvu-pkd-sentiment-api.hf.space/predict"
 
response = requests.post(API_URL, json={
    "text": "Charles Schwab price target raised to $121 from $119 at JPMorgan."
})
 
print(response.json())
# {
#   "label": "positive",
#   "confidence": 0.93,
#   "latency_ms": 79.3
# }
```
 
**Input:** `{ "text": "..." }` — any financial sentence or headline (max 128 tokens)
 
**Output:** `{ "label": str, "confidence": float, "latency_ms": float }`

---

## Distillation Approach
 
This model was trained using a **two-stage Patient Knowledge Distillation** strategy.
 
<h3>Stage 1 — Distillation on Pseudo-Labeled Financial News</h3>
 
The student (ALBERT-base) was first trained on a large corpus of scraped financial
news pseudo-labeled by the FinBERT teacher, using a combined loss of soft KL
divergence targets and intermediate layer alignment.
 
<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Dataset</td>
      <td>Scraped financial news (pseudo-labeled by FinBERT)</td>
    </tr>
    <tr>
      <td>Train / Val / Test split</td>
      <td>5,587 / 1,197 / 1,198</td>
    </tr>
    <tr>
      <td>Epochs</td>
      <td>3</td>
    </tr>
    <tr>
      <td>Batch size</td>
      <td>32</td>
    </tr>
    <tr>
      <td>Optimizer</td>
      <td>AdamW</td>
    </tr>
    <tr>
      <td>Learning rate</td>
      <td>2e-5</td>
    </tr>
    <tr>
      <td>KD temperature sweep</td>
      <td>[2, 5, 9]</td>
    </tr>
    <tr>
      <td>Alpha (KD loss weight)</td>
      <td>0.3</td>
    </tr>
    <tr>
      <td>PKD beta</td>
      <td>0.02</td>
    </tr>
    <tr>
      <td>PKD student layers</td>
      <td>[2, 4, 8, 12]</td>
    </tr>
  </tbody>
</table>
 
<h3>Stage 2 — Fine-tuning on Financial PhraseBank</h3>
 
The distilled student was then fine-tuned on the high-quality
**Financial PhraseBank (100% annotator agreement)** subset using standard
cross-entropy loss to align the student with gold-label financial sentiment.
 
<table>
  <thead>
    <tr>
      <th>Parameter</th>
      <th>Value</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Dataset</td>
      <td>Financial PhraseBank (100% agreement)</td>
    </tr>
    <tr>
      <td>Train / Val / Test split</td>
      <td>1,584 / 340 / 340</td>
    </tr>
    <tr>
      <td>Epochs</td>
      <td>1</td>
    </tr>
    <tr>
      <td>Loss</td>
      <td>Cross-entropy</td>
    </tr>
  </tbody>
</table>
 
<h3>Loss Function</h3>
 
The Stage 1 total loss combines:
 
<ul>
  <li><strong>KL Divergence</strong> between teacher soft targets and student logits (soft label transfer)</li>
  <li><strong>Patient KD</strong> alignment between intermediate ALBERT and FinBERT hidden layers</li>
  <li><strong>Alpha</strong> controls the balance between hard label CE loss and soft KD loss</li>
</ul>
 
---
 
## Full Performance Comparison
 
The table below compares all training strategies evaluated against the same
Financial PhraseBank test set (340 samples):
 
<table>
  <thead>
    <tr>
      <th>Model</th>
      <th>Params</th>
      <th>Size</th>
      <th>Test Acc</th>
      <th>Macro F1</th>
      <th>KL (teacher→student)</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>Teacher FinBERT</td>
      <td>109.5M</td>
      <td>417.7 MB</td>
      <td>97.6%</td>
      <td>0.9696</td>
      <td>—</td>
    </tr>
    <tr>
      <td>Fresh → FP (baseline)</td>
      <td>11.7M</td>
      <td>44.6 MB</td>
      <td>77.1%</td>
      <td>0.6126</td>
      <td>0.359</td>
    </tr>
    <tr>
      <td>CE-scraped → FP</td>
      <td>11.7M</td>
      <td>44.6 MB</td>
      <td>95.9%</td>
      <td>0.9392</td>
      <td>0.230</td>
    </tr>
    <tr>
      <td>KD-scraped → FP</td>
      <td>11.7M</td>
      <td>44.6 MB</td>
      <td>96.5%</td>
      <td>0.9514</td>
      <td>0.156</td>
    </tr>
    <tr>
      <td><strong>PKD-scraped → FP (ours)</strong></td>
      <td><strong>11.7M</strong></td>
      <td><strong>44.6 MB</strong></td>
      <td><strong>97.4%</strong></td>
      <td><strong>0.9650</strong></td>
      <td>0.188</td>
    </tr>
  </tbody>
</table>
 
**Key takeaway:** Patient KD achieves the highest Macro F1 among all student
variants and closes to within 0.5% of the teacher — demonstrating that
intermediate layer alignment significantly improves distillation quality beyond
standard KD.
 
---
 
## Intended Use
 
<h3>Designed for</h3>
 
<ul>
  <li>Classifying sentiment in financial news headlines and short excerpts</li>
  <li>Lightweight inference in resource-constrained environments (edge, serverless)</li>
  <li>Research into compact NLP models for financial NLP tasks</li>
  <li>Downstream integration into financial analytics pipelines</li>
</ul>
 
<h3>Not designed for</h3>
 
<ul>
  <li>Long-form financial documents (more than 128 tokens per segment — chunk first)</li>
  <li>Non-financial general text (model is domain-specialized)</li>
  <li>High-stakes trading decisions without additional validation</li>
</ul>
 
---
 
## Limitations
 
<ul>
  <li><strong>Domain-specific:</strong> Trained exclusively on financial text. Performance on general-domain sentiment will degrade.</li>
  <li><strong>Sequence length cap:</strong> Maximum 128 tokens per input. Longer documents should be chunked by sentence.</li>
  <li><strong>Label distribution:</strong> Financial PhraseBank skews neutral-heavy. Rare strongly negative or positive samples may receive lower confidence.</li>
  <li><strong>English only:</strong> Both training datasets are English. No multilingual support.</li>
</ul>
 
---
 
## Training Framework
 
<ul>
  <li><strong>PyTorch</strong> with a custom training loop for KD and PKD loss computation</li>
  <li><strong>Hugging Face Transformers</strong> for model loading, tokenization, and checkpointing</li>
  <li>Teacher model frozen throughout distillation; only student weights updated</li>
</ul>
 
---

## Citation
 
If you use this model or the distillation pipeline in your work, please cite:
 
```bibtex
@misc{pkd-albert-finbert,
  author    = {Ha Dang Vu},
  title     = {PKD-ALBERT: Lightweight Financial Sentiment via Patient Knowledge Distillation},
  year      = {2025},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/hadangvu/pkd-albert-student}
}
```

## Related Work
 
<ul>
  <li><a href="https://huggingface.co/ProsusAI/finbert">ProsusAI/finbert</a> — Teacher model</li>
  <li><a href="https://arxiv.org/abs/1908.09355">Patient Knowledge Distillation (Sun et al., 2019)</a> — PKD method</li>
  <li><a href="https://www.researchgate.net/publication/251231364">Financial PhraseBank (Malo et al., 2014)</a> — Evaluation dataset</li>
</ul>
 
---
 
<em>Built as part of a financial NLP research project exploring efficient model compression for domain-specific sentiment analysis.</em>