# Changelog — Spam XAI Project

> ENGT 375: Applied Machine Learning for Engineering Technology — Spring 2026, ODU
>
> Merged from `CHANGELOG.md` (Streamlit version) and `CHANGELOG_gradio.md` (Gradio version)
> on 2026-04-07. Pre-merge entries are preserved in their original sections below.
>
> Reconstructed retroactively from file timestamps and code comments. This project
> does not use traditional version control; dates reflect filesystem modification times.

---

## v1.5.3 — 2026-04-16 (HuggingFace sync)

### Summary
Synced latest README, MODEL_README, and project files to HuggingFace via
`update-huggingface.command`. No code changes.

---

## v1.5.2 — 2026-04-14 (Documentation sync with deployed artifacts)

### Summary
Audit pass against the actual code base and saved model artifacts. Fixed stale
threshold references in `README.md` and `MODEL_README.md` that still quoted the
original v0.1.0 value of 0.3714. The deployed `optimal_threshold.joblib` file
now holds 0.5176 (written by the v1.5 retrain on 2026-04-07), and the docs
have been updated to match.

### Changes
- `MODEL_README.md` — Training Details table: "Optimal threshold" 0.3714 → 0.5176
- `MODEL_README.md` — Files table: `optimal_threshold.joblib` description
  updated to reference 0.5176
- `README.md` — Performance section: threshold statement updated to 0.5176 with
  a note that the value is read directly from `optimal_threshold.joblib`

### Rationale
The code base is the source of truth. `joblib.load("optimal_threshold.joblib")`
returned `0.5175934056031538` at audit time, confirming the v1.5 retrain's
threshold-optimization output (CHANGELOG line 42). The MODEL_README was never
updated after that retrain. No code changes, no retraining required.

---

## v1.5.1 — 2026-04-14 (HF Spaces context)

### Summary
No code changes to v1 in this session — v1's Space (`VoltageVagabond/spam-xai-classifier`)
remained healthy throughout. The session used v1 as the reference baseline while
debugging deployment issues for the v2 Space and the liquid Space.

### Q&A from this session

**Q: Why does v1's Space work without any of the issues v2 had?**
A: v1 pins `scikit-learn==1.6.1` exactly, and its model artifacts were trained against
that same version. v2's pin (`>=1.3.0` originally) plus a model serialized by sklearn 1.8.0
caused a `multi_class` AttributeError on load. The lesson: pin sklearn exactly, and make
sure the Space's Python version matches the version that produced the model.

**Q: Does my model repo `spam-xai-model` need to be public for v1's Space to work?**
A: No. v1 bundles all its model files (`voting_model.joblib`, etc.) directly in the
Space repo, so it never has to fetch from another repo at runtime. Only the liquid
Space needs an external model repo (because LoRA adapters are too large to bundle).

### Changes
- None (v1 untouched). This entry exists for cross-reference with the v2 changelog.

---

## v1.5 — 2026-04-07

### Full retrain + project cleanup

- **Full ensemble retrain** — ran `retrain-full.command` with no feedback corrections; trained VotingClassifier (RF + LR + SVM) on 190,543 emails; achieved 97% accuracy, 0.97 macro F1, optimal threshold 0.5176
- **Deleted project cruft** — removed macOS resource forks (`._*`), `__pycache__/`, `.DS_Store`, stale `.server.log`
- **Deleted legacy app files** — removed `app_legacy.py` and `app_student_legacy.py` (old Streamlit-era versions, superseded by `app.py`)
- **Deleted legacy notebooks** — removed `notebooks/_legacy/` and `notebooks/.ipynb_checkpoints/`
- **Deleted legacy docs** — removed `docs/_legacy_plans/` (pre-merge design notes from March)
- **Deleted old generic model backup** — removed `models_backup/`; kept `models_backup_pre_fast/` and `models_backup_pre_full/` as rollback snapshots for the two most recent retrains

---

## v1.4 — 2026-04-07

### Bug fixes and UI improvements

- **Fixed PyArrow indexing crash in `retrain.py`** — HuggingFace datasets return PyArrow-backed columns; `.values` returned a PyArrow array that `train_test_split` couldn't index. Changed to `.to_numpy()` at all three call sites (lines 159, 160, 183, 184)
- **Fixed fast-mode model type mismatch in `app.py`** — fast retrain saves a bare `RandomForestClassifier` but `app.py` was unconditionally calling `.named_estimators_['rf']` (a `VotingClassifier` attribute). Added `hasattr` check so both fast (RF) and full (VotingClassifier) models load correctly
- **Fixed ELI5 rendering in Gradio** — `eli5.format_as_html()` wraps methodology boilerplate in a `<pre>` block that overflows the container. Now strips `<pre>` blocks and wraps output in a `overflow-wrap: break-word` div so only the contribution table shows
- **Widened app layout** — increased `.gradio-container` `max-width` from 1180px to 1600px to reduce cramped appearance on wide monitors

---

## v1.3 — 2026-04-07

### Parallelized preprocessing; updated time estimates

- **Parallelized `preprocess_text` in `retrain.py`** — replaced single-threaded `.apply()` with `joblib.Parallel(n_jobs=-1, prefer='threads')` to use all CPU cores during text cleaning/stemming
- **Updated time estimates** to `~5-10 minutes` (fast mode) in `retrain.py` header, runtime print statement, and `retrain-fast.command` banner; previous estimate of `~2-5 minutes` was inaccurate for the 190K-email corpus

---

## v1.2 — 2026-04-07

### Training data overhaul — replaced Kaggle with research-grade corpora

- **Dropped Kaggle `spam_Emails_data.csv`** — dataset was noisy/synthetic-heavy and capped at 100K rows; replaced with three research-grade sources
- **Added Enron corpus** (~33K real corporate emails from `data/raw/enron/enron_spam_data.csv`) — file already existed but was not being loaded by the retrain scripts
- **Added puyang2025/seven-phishing-email-datasets** (HuggingFace) — 138K emails from 7 research corpora (TREC-05, TREC-06, TREC-07, CEAS-08, SpamAssassin, Ling-Spam; Enron sub-corpus excluded to avoid duplication). Saved to `data/raw/puyang2025/seven_phishing_emails.parquet`
- **Added zefang-liu/phishing-email-dataset** (HuggingFace) — 18,634 phishing-labeled emails. Saved to `data/raw/zefang/phishing_emails.parquet`
- **New total training data: ~190,543 emails** (was ~100K Kaggle sample)
- **Updated `retrain.py` and `retrain_student.py`** to load all three new sources; removed dead `email-dataset-main` directory-traversal code
- **Updated `build_datasets.py` and `build_liquid_datasets.py`** to include puyang2025 and zefang as additional sources for MLX/Liquid LLM fine-tuning data
- **Updated `HOW_TO_RUN.html`** to reflect new data sources and file paths

---

## v1.1 — 2026-04-07

### Merged spam-classifier-gradio into spam-xai-project

- **File merge:** Moved all of `spam-classifier-gradio/` into `spam-xai-project/`. The two projects shared the same data, preprocessing, and feature engineering — keeping them separate was creating drift and duplicate code.
- The Gradio app (`app.py`), VotingClassifier ensemble training (`train_ensemble.py`), retrain script with feedback support (`retrain.py`), and the Gradio notebook (`notebooks/spam_classifier_gradio.ipynb`) all live in the merged project now.
- **Preserved as legacy** (renamed but kept):
  - `app_legacy.py`, `app_student_legacy.py` — older Streamlit/Gradio app versions
  - `MODEL_README_legacy.md`, `README_legacy.md` — older HF cards
  - `retrain_legacy.py` — older RF-only retrain
  - `notebooks/_legacy/spam_classifier_xai.ipynb`, `_legacy/spam_classifier_xai_executed.ipynb`
  - `models/_baseline_rf/` — older RF-only model artifacts
- **Reference docs added:** Built out `docs/references/` with 20 PDFs (LIME, SHAP, TreeSHAP, Anchors, Pedregosa sklearn, Kuzlu et al. solar XAI, 5 spam-detection survey papers) and 11 HTML guides (LIME/SHAP/ELI5 docs, sklearn user guide pages, Gradio quickstart, HF Spaces docs, Molnar Interpretable ML book). All linked from `docs/references/how-to.html`. Shared papers (Attention, LoRA, QLoRA, PEFT survey) live in the top-level `references/` folder used by all three sibling projects.
- **Notebook style cleanup:** The student notebook (`spam_classifier_xai_student.ipynb`) and the Gradio notebook (`spam_classifier_gradio.ipynb`) were rewritten in places to remove "senior engineer" code patterns: replaced lambdas with named helpers, broke chained list-comp+sort+slice into explicit loops, and softened comments to read more like an undergraduate beginner wrote them. Functionality unchanged. Backups saved as `*.ipynb.bak`.
- **HF deployment:** Updated `update-huggingface.command` and `upload_to_hf.py` to push the merged project to TWO new HF repos:
  - `VoltageVagabond/spam-xai-classifier` (Space, Gradio SDK)
  - `VoltageVagabond/spam-xai-model` (Model)
  - The old `spam-classifier-gradio` Space and `spam-classifier-gradio-model` Model are archived (not deleted).

### Cleanup pass — same day

- Merged two `CHANGELOG.md` files into this one
- Merged `MODEL_README.md` + `MODEL_README_legacy.md` into one model card
- Merged `README.md` body content (kept HF Space YAML frontmatter intact)
- Consolidated retrain scripts: one canonical `retrain.py` with `--mode {fast,full}`, plus `retrain-fast.command` and `retrain-full.command` matching the Liquid project's pattern
- Wired all three launcher commands to existing target files (`launch-app.command`, `launch-gradio.command`, `launch-notebook.command`)
- Deleted three Windows `.bat` files (`retrain.bat`, `run_app.bat`, `run_app_student.bat`) — macOS-only project now

---

## Pre-merge — Gradio version (originally `CHANGELOG_gradio.md`)

### v0.2.1 — 2026-03-28
**Retrain with Auto-Backup**
- `retrain.command` now automatically backs up `models/` to `models_backup/` before retraining

### v0.2.0 — 2026-03-28
**HuggingFace Upload + Retrain Command**
- Uploaded project to HuggingFace Space: `VoltageVagabond/spam-classifier-gradio` (Gradio SDK)
- Created `README.md` with HF Space YAML frontmatter
- Uploaded model `.joblib` files to HF Space for live demo
- Added `retrain.command` for the sklearn ensemble
- Uploaded all training data to HF dataset: `VoltageVagabond/spam-email-dataset`

### v0.1.3 — 2026-03-23
**UI Fix: Scroll-to-Top Bug**
- Fixed page jumping to top when pasting email or clicking Classify
- Added `autoscroll=False`, `cache_examples=False`, `scroll_to_output=False`

### v0.1.2 — 2026-03-23
**Bug Fixes + Real-World Testing**
- Fixed SHAP crash: `shap_values` returned multi-dimensional arrays from KernelExplainer; now `.flatten()` to 1D before sorting
- Added try/except around SHAP so Result and LIME tabs still show if SHAP fails
- Pinned `gradio==4.19.2` (Gradio 4.44.1 has a Python 3.9 bug)
- Added `launch-notebook.command` for Jupyter notebook

**Known Limitation: Legitimate Marketing Emails Misclassified**
- Tested with a real Lenovo Rewards referral email — classified as **SPAM at 78% confidence** (false positive)
- Root cause: dollar signs, "earn", "rewards", "purchase" vocabulary overlaps heavily with spam training patterns
- LIME and SHAP correctly showed which features drove the misclassification — demonstrating the value of XAI for understanding model failures
- This is a known weakness of bag-of-words/TF-IDF classifiers

### v0.1.1 — 2026-03-23
**Bug Fix: SHAP numpy indexing**
- Fixed `TypeError: only integer scalar arrays can be converted to a scalar index` in SHAP plot generation
- `np.argsort` returns numpy int64 indices; Python list indexing needs plain `int`

### v0.1.0 — 2026-03-23
**Initial Build**
- Created fresh Gradio project (replacing old Streamlit version)
- Ported preprocessing and 24 metadata features from old `utils_student.py`
- Loaded Kaggle spam dataset (~190K emails, capped at 100K stratified sample)
- Trained and compared 3 models:
  - Random Forest: 97.75% accuracy, F1=0.976
  - Logistic Regression: 96.57% accuracy, F1=0.964
  - SVM (LinearSVC + calibration): 96.89% accuracy, F1=0.967
- Combined into VotingClassifier (soft voting): 97.40% accuracy, F1=0.973
- Optimal threshold: 0.3714 (targeting 99% ham precision)
- Built Gradio interface with text + .txt file upload, Result tab, LIME tab, SHAP tab
- 4 built-in example emails

---

## Pre-merge — Streamlit version (originally `CHANGELOG.md`)

### v0.3.1 — 2026-03-28
**Retrain with Auto-Backup** (same fix as gradio v0.2.1)

### v0.3 — 2026-03-28
**HuggingFace Upload + Command Launchers**
- Uploaded to HuggingFace Space: `VoltageVagabond/spam-xai-project` (Docker + Streamlit)
- Created Dockerfile for HF Space deployment
- Added `retrain.command` and `launch-app.command`

### v1.0 — 2026-03-23
**Documentation**
- Added `docs/` directory with project documentation
- Created the original `CHANGELOG.md`

### v0.9 — 2026-03-09
**Interactive feature explorer**
- Generated `spam-feature-explorer.html` — standalone HTML/JS page for interactively exploring the 24 metadata features

### v0.8 — 2026-03-08
**Student version and LLM feature extraction**
- Created `app_student.py` — full rewrite of `app.py` with extensive inline comments
- Imports all shared logic from `utils_student.py`
- Added `streamlit_js_eval` for persisting classification results in browser localStorage
- Added user feedback system: `save_feedback()` writes corrections to `data/feedback/feedback_log.csv`
- Created `retrain_student.py` — student version of training script with explanatory comments, optional LLM feature extraction via Ollama, SHA-256 hash-based caching of LLM features
- Defined 6 LLM intent/tone features: `intent_promotional`, `intent_transactional`, `intent_personal`, `intent_phishing`, `tone_urgency`, `tone_formality`
- `SKIP_LLM_TRAINING` flag defaults to `True` for fast retraining
- Created `test_accuracy.py` for quick model smoke-testing
- Created student notebook (`spam_classifier_xai_student.ipynb`)

### v0.7 — 2026-03-06
**New training datasets and MinMaxScaler**
- Switched primary training data from SpamAssassin + Enron to Kaggle 190K + GitHub email-dataset
- Added `MinMaxScaler` for metadata features
- Added user feedback integration: corrections weighted 5x and injected into training
- Reduced GridSearchCV grid from 36 to 4 combinations and CV from 5 to 3 folds

### v0.6 — 2026-03-06
**Expanded feature engineering (11 to 24 metadata features)**
- Created `utils_student.py` — shared utilities module
- Added 13 new metadata features (features 12-24): `has_specific_date`, `has_specific_time`, `date_reference_count`, `has_unsubscribe`, `has_physical_address`, `has_proper_greeting`, `has_contact_info`, `registration_language_score`, `cta_to_info_ratio`, `shortener_url_ratio`, `legitimate_platform_count`, `gov_edu_url_count`, `question_mark_count`
- Expanded `generate_newsletters_student.py` from 50 to 600 target newsletters

### v0.5 — 2026-03-05
**OCR support, dark mode, and header feature extraction**
- Optional `pytesseract` + Pillow for classifying email screenshots
- `extract_header_features()` parses From/To/Subject headers to detect government domains, mailing-list patterns, etc.
- Full dark mode with theme-aware matplotlib + ELI5 HTML overlay
- Created `run_app_student.bat`, `_restyle_notebook.py`

### v0.4 — 2026-03-04
**Newsletter augmentation and generation**
- Created `generate_newsletters.py` producing 50 synthetic government/institutional newsletters via Ollama/Qwen
- **Change 9** in `retrain.py`: sentence-shuffled newsletter variants (5 per ham email) to improve classification of legitimate promotional-style emails
- Created initial Jupyter notebook (`spam_classifier_xai.ipynb`)

### v0.3 — 2026-03-04
**Ollama/Qwen LLM integration**
- **Change 3:** `get_llm_second_opinion()` calling Ollama with Qwen 3.5 to independently classify emails
- RF and LLM confidence scores blended with context-aware weighting
- Added AI Explanation tab using Qwen 3.5 to produce plain-English explanations from top-5 LIME/SHAP/ELI5 features
- **Change 1:** Default threshold raised to 0.60

### v0.2 — 2026-03-04
**Context-aware phrase lists and domain whitelist**
- Added `SPAM_CONTEXT_PHRASES` (14 phrases: "act now", "limited time", etc.) and `HAM_CONTEXT_PHRASES` (12 phrases: "click to unsubscribe", "official notice", etc.)
- Added `check_domain_trust()` extracting sender domain from `From:` / `Return-Path:` headers

### v0.1 — 2026-03-04
**Initial Streamlit app and training pipeline**
- Created `requirements.txt` with scikit-learn, LIME, SHAP, ELI5, Streamlit, NLTK, matplotlib, scipy
- Built `retrain.py` loading SpamAssassin + Enron-Spam, 3000 TF-IDF features + 11 metadata features
- GridSearchCV over a Random Forest, `class_weight='balanced'`, isotonic probability calibration, optimal threshold targeting 99% ham precision
- Created `app.py` — Streamlit UI with text-area input, example emails, confidence gauge, four explanation tabs (LIME, SHAP, ELI5, Comparison)
- Added `run_app.bat` and `HOW_TO_RUN.html`

---

## Training Methodology Reference (for paper)

### Complete Training Pipeline (current — post-merge)

1. **Data sources:**
   - Kaggle spam dataset (`data/spam_Emails_data.csv`, ~193K emails, stratified-sampled to 100K)
   - GitHub email-dataset (folder 1 = ham, folder 2 = spam)
   - Optional: SpamAssassin + Enron + synthetic newsletters (legacy data sources)

2. **Preprocessing:** HTML removal, URL removal, email address removal, non-alphabetic removal, lowercasing, NLTK stopword removal, Porter stemming

3. **Feature engineering:**
   - **TF-IDF:** 3,000 features, ngram_range=(1,3), min_df=2, max_df=0.90, sublinear_tf=True
   - **24 hand-crafted metadata features:** exclamation_density, dollar_sign_count, caps_word_ratio, spam_phrase_count, ham_phrase_count, net_spam_context, url_count, html_tag_count, email_length, avg_sentence_length, capitalization_ratio, has_specific_date, has_specific_time, date_reference_count, has_unsubscribe, has_physical_address, has_proper_greeting, has_contact_info, registration_language_score, cta_to_info_ratio, shortener_url_ratio, legitimate_platform_count, gov_edu_url_count, question_mark_count
   - **MinMaxScaler** on metadata features (so they match the 0-1 TF-IDF scale)
   - Combined via `scipy.sparse.hstack` → 3,024 total features

4. **Train/test split:** 70/30, stratified, random_state=42

5. **Models:**
   - Random Forest: n_estimators=200, class_weight='balanced' → 97.75% accuracy, F1=0.976
   - Logistic Regression: max_iter=1000, class_weight='balanced' → 96.57% accuracy, F1=0.964
   - SVM (LinearSVC + CalibratedClassifierCV): class_weight='balanced' → 96.89% accuracy, F1=0.967
   - **VotingClassifier (soft voting, all 3): 97.40% accuracy, F1=0.973** ← deployed model

6. **Threshold optimization:** Precision-recall curve targeting 99% ham precision → optimal threshold 0.3714

7. **Explainability:**
   - LIME: `LimeTabularExplainer`, 200-row training sample, 10 features per explanation
   - SHAP: `KernelExplainer` on 24 metadata features only (TreeExplainer too slow on 3,000+ features), 50-row background
   - ELI5: `PermutationImportance` on the full 3,024-feature space

8. **Framework:** scikit-learn 1.6.1
9. **Hardware:** MacBook Pro M4 Pro, 24 GB unified RAM

### Key Hyperparameters

| Parameter | Value | Rationale |
|-----------|-------|-----------|
| TF-IDF max_features | 3,000 | Balance between vocabulary coverage and dimensionality |
| TF-IDF ngram_range | (1, 3) | Capture phrases like "act now", "limited time offer" |
| TF-IDF min_df | 2 | Remove extremely rare words |
| TF-IDF max_df | 0.90 | Remove words appearing in 90%+ of emails |
| TF-IDF sublinear_tf | True | Logarithmic term frequency scaling |
| RF n_estimators | 200 | Sufficient trees for stable predictions |
| class_weight | 'balanced' | Handle spam/ham class imbalance |
| test_size | 0.3 | 70/30 split |
| random_state | 42 | Reproducibility |

### Evolution of the Project

| Version | Features | Model | Data | UI |
|---------|----------|-------|------|-----|
| v0.1 (Streamlit) | 11 metadata + 3000 TF-IDF | RF + GridSearchCV + Calibration | SpamAssassin + Enron | Streamlit |
| v0.6 | 24 metadata + 3000 TF-IDF | Same | + Kaggle + GitHub | Streamlit |
| v0.8 | 24 metadata + 3000 TF-IDF + optional LLM | Same | + user feedback (5x weighted) | Streamlit + student version |
| v0.1.0 (Gradio) | 24 metadata + 3000 TF-IDF | RF + LR + SVM VotingClassifier | Kaggle 100K | Gradio (rewrite from Streamlit) |
| v1.1 (merged) | 24 metadata + 3000 TF-IDF | VotingClassifier (canonical) | Kaggle + GitHub + optional Spam Assassin | Gradio (canonical) |

### Known Limitations

- Legitimate marketing emails (e.g., Lenovo Rewards) misclassified as spam at high confidence due to vocabulary overlap with spam patterns (dollar signs, "earn", "rewards")
- TF-IDF is bag-of-words — cannot understand email intent or context
- SHAP limited to metadata features only (KernelExplainer too slow on 3,000+ features)
- The voting model is large (152 MB) — pushing to a free HF Space requires explicit inclusion in the upload script

### Citations

- Pedregosa, F., et al. (2011). "Scikit-learn: Machine Learning in Python." JMLR 12, 2825-2830
- Ribeiro, M.T., et al. (2016). "'Why Should I Trust You?': Explaining the Predictions of Any Classifier." KDD 2016 (LIME)
- Lundberg, S.M. & Lee, S.I. (2017). "A Unified Approach to Interpreting Model Predictions." NeurIPS 2017 (SHAP)
- Lundberg, S.M., et al. (2020). "From local explanations to global understanding with explainable AI for trees." Nature MI 2(1) (TreeSHAP)
- Kuzlu, M., et al. (2020). "Gaining Insight Into Solar PV Power Generation Forecasting Utilizing Explainable AI Tools." IEEE Access 8 (XAI feature reduction methodology)
- Breiman, L. (2001). "Random Forests." Machine Learning 45(1)