CHANGELOG.md

Changelog — Spam XAI Project

ENGT 375: Applied Machine Learning for Engineering Technology — Spring 2026, ODU

Merged from CHANGELOG.md (Streamlit version) and CHANGELOG_gradio.md (Gradio version) on 2026-04-07. Pre-merge entries are preserved in their original sections below.

Reconstructed retroactively from file timestamps and code comments. This project does not use traditional version control; dates reflect filesystem modification times.

---

v1.5.3 — 2026-04-16 (HuggingFace sync)

Summary

Synced latest README, MODEL_README, and project files to HuggingFace via update-huggingface.command. No code changes.

---

v1.5.2 — 2026-04-14 (Documentation sync with deployed artifacts)

Summary

Audit pass against the actual code base and saved model artifacts. Fixed stale threshold references in README.md and MODEL_README.md that still quoted the original v0.1.0 value of 0.3714. The deployed optimal_threshold.joblib file now holds 0.5176 (written by the v1.5 retrain on 2026-04-07), and the docs have been updated to match.

Changes

• MODEL_README.md — Training Details table: "Optimal threshold" 0.3714 → 0.5176
• MODEL_README.md — Files table: optimal_threshold.joblib description updated to reference 0.5176
• README.md — Performance section: threshold statement updated to 0.5176 with a note that the value is read directly from optimal_threshold.joblib

Rationale

The code base is the source of truth. joblib.load("optimal_threshold.joblib") returned 0.5175934056031538 at audit time, confirming the v1.5 retrain's threshold-optimization output (CHANGELOG line 42). The MODEL_README was never updated after that retrain. No code changes, no retraining required.

---

v1.5.1 — 2026-04-14 (HF Spaces context)

Summary

No code changes to v1 in this session — v1's Space (VoltageVagabond/spam-xai-classifier) remained healthy throughout. The session used v1 as the reference baseline while debugging deployment issues for the v2 Space and the liquid Space.

Q&A from this session

Q: Why does v1's Space work without any of the issues v2 had? A: v1 pins scikit-learn==1.6.1 exactly, and its model artifacts were trained against that same version. v2's pin (>=1.3.0 originally) plus a model serialized by sklearn 1.8.0 caused a multi_class AttributeError on load. The lesson: pin sklearn exactly, and make sure the Space's Python version matches the version that produced the model.

Q: Does my model repo spam-xai-model need to be public for v1's Space to work? A: No. v1 bundles all its model files (voting_model.joblib, etc.) directly in the Space repo, so it never has to fetch from another repo at runtime. Only the liquid Space needs an external model repo (because LoRA adapters are too large to bundle).

Changes

• None (v1 untouched). This entry exists for cross-reference with the v2 changelog.

---

v1.5 — 2026-04-07

Full retrain + project cleanup

• Full ensemble retrain — ran retrain-full.command with no feedback corrections; trained VotingClassifier (RF + LR + SVM) on 190,543 emails; achieved 97% accuracy, 0.97 macro F1, optimal threshold 0.5176
• Deleted project cruft — removed macOS resource forks (._*), __pycache__/, .DS_Store, stale .server.log
• Deleted legacy app files — removed app_legacy.py and app_student_legacy.py (old Streamlit-era versions, superseded by app.py)
• Deleted legacy notebooks — removed notebooks/_legacy/ and notebooks/.ipynb_checkpoints/
• Deleted legacy docs — removed docs/_legacy_plans/ (pre-merge design notes from March)
• Deleted old generic model backup — removed models_backup/; kept models_backup_pre_fast/ and models_backup_pre_full/ as rollback snapshots for the two most recent retrains

---

v1.4 — 2026-04-07

Bug fixes and UI improvements

• Fixed PyArrow indexing crash in retrain.py — HuggingFace datasets return PyArrow-backed columns; .values returned a PyArrow array that train_test_split couldn't index. Changed to .to_numpy() at all three call sites (lines 159, 160, 183, 184)
• Fixed fast-mode model type mismatch in app.py — fast retrain saves a bare RandomForestClassifier but app.py was unconditionally calling .named_estimators_['rf'] (a VotingClassifier attribute). Added hasattr check so both fast (RF) and full (VotingClassifier) models load correctly
• Fixed ELI5 rendering in Gradio — eli5.format_as_html() wraps methodology boilerplate in a <pre> block that overflows the container. Now strips <pre> blocks and wraps output in a overflow-wrap: break-word div so only the contribution table shows
• Widened app layout — increased .gradio-container max-width from 1180px to 1600px to reduce cramped appearance on wide monitors

---

v1.3 — 2026-04-07

Parallelized preprocessing; updated time estimates

• Parallelized preprocess_text in retrain.py — replaced single-threaded .apply() with joblib.Parallel(n_jobs=-1, prefer='threads') to use all CPU cores during text cleaning/stemming
• Updated time estimates to ~5-10 minutes (fast mode) in retrain.py header, runtime print statement, and retrain-fast.command banner; previous estimate of ~2-5 minutes was inaccurate for the 190K-email corpus

---

v1.2 — 2026-04-07

Training data overhaul — replaced Kaggle with research-grade corpora

• Dropped Kaggle spam_Emails_data.csv — dataset was noisy/synthetic-heavy and capped at 100K rows; replaced with three research-grade sources
• Added Enron corpus (~33K real corporate emails from data/raw/enron/enron_spam_data.csv) — file already existed but was not being loaded by the retrain scripts
• Added puyang2025/seven-phishing-email-datasets (HuggingFace) — 138K emails from 7 research corpora (TREC-05, TREC-06, TREC-07, CEAS-08, SpamAssassin, Ling-Spam; Enron sub-corpus excluded to avoid duplication). Saved to data/raw/puyang2025/seven_phishing_emails.parquet
• Added zefang-liu/phishing-email-dataset (HuggingFace) — 18,634 phishing-labeled emails. Saved to data/raw/zefang/phishing_emails.parquet
• New total training data: ~190,543 emails (was ~100K Kaggle sample)
• Updated retrain.py and retrain_student.py to load all three new sources; removed dead email-dataset-main directory-traversal code
• Updated build_datasets.py and build_liquid_datasets.py to include puyang2025 and zefang as additional sources for MLX/Liquid LLM fine-tuning data
• Updated HOW_TO_RUN.html to reflect new data sources and file paths

---

v1.1 — 2026-04-07

Merged spam-classifier-gradio into spam-xai-project

• File merge: Moved all of spam-classifier-gradio/ into spam-xai-project/. The two projects shared the same data, preprocessing, and feature engineering — keeping them separate was creating drift and duplicate code.
• The Gradio app (app.py), VotingClassifier ensemble training (train_ensemble.py), retrain script with feedback support (retrain.py), and the Gradio notebook (notebooks/spam_classifier_gradio.ipynb) all live in the merged project now.
• Preserved as legacy (renamed but kept):
  • app_legacy.py, app_student_legacy.py — older Streamlit/Gradio app versions
  • MODEL_README_legacy.md, README_legacy.md — older HF cards
  • retrain_legacy.py — older RF-only retrain
  • notebooks/_legacy/spam_classifier_xai.ipynb, _legacy/spam_classifier_xai_executed.ipynb
  • models/_baseline_rf/ — older RF-only model artifacts
• Reference docs added: Built out docs/references/ with 20 PDFs (LIME, SHAP, TreeSHAP, Anchors, Pedregosa sklearn, Kuzlu et al. solar XAI, 5 spam-detection survey papers) and 11 HTML guides (LIME/SHAP/ELI5 docs, sklearn user guide pages, Gradio quickstart, HF Spaces docs, Molnar Interpretable ML book). All linked from docs/references/how-to.html. Shared papers (Attention, LoRA, QLoRA, PEFT survey) live in the top-level references/ folder used by all three sibling projects.
• Notebook style cleanup: The student notebook (spam_classifier_xai_student.ipynb) and the Gradio notebook (spam_classifier_gradio.ipynb) were rewritten in places to remove "senior engineer" code patterns: replaced lambdas with named helpers, broke chained list-comp+sort+slice into explicit loops, and softened comments to read more like an undergraduate beginner wrote them. Functionality unchanged. Backups saved as *.ipynb.bak.
• HF deployment: Updated update-huggingface.command and upload_to_hf.py to push the merged project to TWO new HF repos:
  • VoltageVagabond/spam-xai-classifier (Space, Gradio SDK)
  • VoltageVagabond/spam-xai-model (Model)
  • The old spam-classifier-gradio Space and spam-classifier-gradio-model Model are archived (not deleted).

Cleanup pass — same day

• Merged two CHANGELOG.md files into this one
• Merged MODEL_README.md + MODEL_README_legacy.md into one model card
• Merged README.md body content (kept HF Space YAML frontmatter intact)
• Consolidated retrain scripts: one canonical retrain.py with --mode {fast,full}, plus retrain-fast.command and retrain-full.command matching the Liquid project's pattern
• Wired all three launcher commands to existing target files (launch-app.command, launch-gradio.command, launch-notebook.command)
• Deleted three Windows .bat files (retrain.bat, run_app.bat, run_app_student.bat) — macOS-only project now

---

Pre-merge — Gradio version (originally CHANGELOG_gradio.md)

v0.2.1 — 2026-03-28

Retrain with Auto-Backup

• retrain.command now automatically backs up models/ to models_backup/ before retraining

v0.2.0 — 2026-03-28

HuggingFace Upload + Retrain Command

• Uploaded project to HuggingFace Space: VoltageVagabond/spam-classifier-gradio (Gradio SDK)
• Created README.md with HF Space YAML frontmatter
• Uploaded model .joblib files to HF Space for live demo
• Added retrain.command for the sklearn ensemble
• Uploaded all training data to HF dataset: VoltageVagabond/spam-email-dataset

v0.1.3 — 2026-03-23

UI Fix: Scroll-to-Top Bug

• Fixed page jumping to top when pasting email or clicking Classify
• Added autoscroll=False, cache_examples=False, scroll_to_output=False

v0.1.2 — 2026-03-23

Bug Fixes + Real-World Testing

• Fixed SHAP crash: shap_values returned multi-dimensional arrays from KernelExplainer; now .flatten() to 1D before sorting
• Added try/except around SHAP so Result and LIME tabs still show if SHAP fails
• Pinned gradio==4.19.2 (Gradio 4.44.1 has a Python 3.9 bug)
• Added launch-notebook.command for Jupyter notebook

Known Limitation: Legitimate Marketing Emails Misclassified

• Tested with a real Lenovo Rewards referral email — classified as SPAM at 78% confidence (false positive)
• Root cause: dollar signs, "earn", "rewards", "purchase" vocabulary overlaps heavily with spam training patterns
• LIME and SHAP correctly showed which features drove the misclassification — demonstrating the value of XAI for understanding model failures
• This is a known weakness of bag-of-words/TF-IDF classifiers

v0.1.1 — 2026-03-23

Bug Fix: SHAP numpy indexing

• Fixed TypeError: only integer scalar arrays can be converted to a scalar index in SHAP plot generation
• np.argsort returns numpy int64 indices; Python list indexing needs plain int

v0.1.0 — 2026-03-23

Initial Build

• Created fresh Gradio project (replacing old Streamlit version)
• Ported preprocessing and 24 metadata features from old utils_student.py
• Loaded Kaggle spam dataset (~190K emails, capped at 100K stratified sample)
• Trained and compared 3 models:
  • Random Forest: 97.75% accuracy, F1=0.976
  • Logistic Regression: 96.57% accuracy, F1=0.964
  • SVM (LinearSVC + calibration): 96.89% accuracy, F1=0.967
• Combined into VotingClassifier (soft voting): 97.40% accuracy, F1=0.973
• Optimal threshold: 0.3714 (targeting 99% ham precision)
• Built Gradio interface with text + .txt file upload, Result tab, LIME tab, SHAP tab
• 4 built-in example emails

---

Pre-merge — Streamlit version (originally CHANGELOG.md)

v0.3.1 — 2026-03-28

Retrain with Auto-Backup (same fix as gradio v0.2.1)

v0.3 — 2026-03-28

HuggingFace Upload + Command Launchers

• Uploaded to HuggingFace Space: VoltageVagabond/spam-xai-project (Docker + Streamlit)
• Created Dockerfile for HF Space deployment
• Added retrain.command and launch-app.command

v1.0 — 2026-03-23

Documentation

• Added docs/ directory with project documentation
• Created the original CHANGELOG.md

v0.9 — 2026-03-09

Interactive feature explorer

• Generated spam-feature-explorer.html — standalone HTML/JS page for interactively exploring the 24 metadata features

v0.8 — 2026-03-08

Student version and LLM feature extraction

• Created app_student.py — full rewrite of app.py with extensive inline comments
• Imports all shared logic from utils_student.py
• Added streamlit_js_eval for persisting classification results in browser localStorage
• Added user feedback system: save_feedback() writes corrections to data/feedback/feedback_log.csv
• Created retrain_student.py — student version of training script with explanatory comments, optional LLM feature extraction via Ollama, SHA-256 hash-based caching of LLM features
• Defined 6 LLM intent/tone features: intent_promotional, intent_transactional, intent_personal, intent_phishing, tone_urgency, tone_formality
• SKIP_LLM_TRAINING flag defaults to True for fast retraining
• Created test_accuracy.py for quick model smoke-testing
• Created student notebook (spam_classifier_xai_student.ipynb)

v0.7 — 2026-03-06

New training datasets and MinMaxScaler

• Switched primary training data from SpamAssassin + Enron to Kaggle 190K + GitHub email-dataset
• Added MinMaxScaler for metadata features
• Added user feedback integration: corrections weighted 5x and injected into training
• Reduced GridSearchCV grid from 36 to 4 combinations and CV from 5 to 3 folds

v0.6 — 2026-03-06

Expanded feature engineering (11 to 24 metadata features)

• Created utils_student.py — shared utilities module
• Added 13 new metadata features (features 12-24): has_specific_date, has_specific_time, date_reference_count, has_unsubscribe, has_physical_address, has_proper_greeting, has_contact_info, registration_language_score, cta_to_info_ratio, shortener_url_ratio, legitimate_platform_count, gov_edu_url_count, question_mark_count
• Expanded generate_newsletters_student.py from 50 to 600 target newsletters

v0.5 — 2026-03-05

OCR support, dark mode, and header feature extraction

• Optional pytesseract + Pillow for classifying email screenshots
• extract_header_features() parses From/To/Subject headers to detect government domains, mailing-list patterns, etc.
• Full dark mode with theme-aware matplotlib + ELI5 HTML overlay
• Created run_app_student.bat, _restyle_notebook.py

v0.4 — 2026-03-04

Newsletter augmentation and generation

• Created generate_newsletters.py producing 50 synthetic government/institutional newsletters via Ollama/Qwen
• Change 9 in retrain.py: sentence-shuffled newsletter variants (5 per ham email) to improve classification of legitimate promotional-style emails
• Created initial Jupyter notebook (spam_classifier_xai.ipynb)

v0.3 — 2026-03-04

Ollama/Qwen LLM integration

• Change 3: get_llm_second_opinion() calling Ollama with Qwen 3.5 to independently classify emails
• RF and LLM confidence scores blended with context-aware weighting
• Added AI Explanation tab using Qwen 3.5 to produce plain-English explanations from top-5 LIME/SHAP/ELI5 features
• Change 1: Default threshold raised to 0.60

v0.2 — 2026-03-04

Context-aware phrase lists and domain whitelist

• Added SPAM_CONTEXT_PHRASES (14 phrases: "act now", "limited time", etc.) and HAM_CONTEXT_PHRASES (12 phrases: "click to unsubscribe", "official notice", etc.)
• Added check_domain_trust() extracting sender domain from From: / Return-Path: headers

v0.1 — 2026-03-04

Initial Streamlit app and training pipeline

• Created requirements.txt with scikit-learn, LIME, SHAP, ELI5, Streamlit, NLTK, matplotlib, scipy
• Built retrain.py loading SpamAssassin + Enron-Spam, 3000 TF-IDF features + 11 metadata features
• GridSearchCV over a Random Forest, class_weight='balanced', isotonic probability calibration, optimal threshold targeting 99% ham precision
• Created app.py — Streamlit UI with text-area input, example emails, confidence gauge, four explanation tabs (LIME, SHAP, ELI5, Comparison)
• Added run_app.bat and HOW_TO_RUN.html

---

Training Methodology Reference (for paper)

Complete Training Pipeline (current — post-merge)

1. Data sources:

   • Kaggle spam dataset (data/spam_Emails_data.csv, ~193K emails, stratified-sampled to 100K)
   • GitHub email-dataset (folder 1 = ham, folder 2 = spam)
   • Optional: SpamAssassin + Enron + synthetic newsletters (legacy data sources)

2. Preprocessing: HTML removal, URL removal, email address removal, non-alphabetic removal, lowercasing, NLTK stopword removal, Porter stemming

3. Feature engineering:

   • TF-IDF: 3,000 features, ngram_range=(1,3), min_df=2, max_df=0.90, sublinear_tf=True
   • 24 hand-crafted metadata features: exclamation_density, dollar_sign_count, caps_word_ratio, spam_phrase_count, ham_phrase_count, net_spam_context, url_count, html_tag_count, email_length, avg_sentence_length, capitalization_ratio, has_specific_date, has_specific_time, date_reference_count, has_unsubscribe, has_physical_address, has_proper_greeting, has_contact_info, registration_language_score, cta_to_info_ratio, shortener_url_ratio, legitimate_platform_count, gov_edu_url_count, question_mark_count
   • MinMaxScaler on metadata features (so they match the 0-1 TF-IDF scale)
   • Combined via scipy.sparse.hstack → 3,024 total features

4. Train/test split: 70/30, stratified, random_state=42

5. Models:

   • Random Forest: n_estimators=200, class_weight='balanced' → 97.75% accuracy, F1=0.976
   • Logistic Regression: max_iter=1000, class_weight='balanced' → 96.57% accuracy, F1=0.964
   • SVM (LinearSVC + CalibratedClassifierCV): class_weight='balanced' → 96.89% accuracy, F1=0.967
   • VotingClassifier (soft voting, all 3): 97.40% accuracy, F1=0.973 ← deployed model

6. Threshold optimization: Precision-recall curve targeting 99% ham precision → optimal threshold 0.3714

7. Explainability:

   • LIME: LimeTabularExplainer, 200-row training sample, 10 features per explanation
   • SHAP: KernelExplainer on 24 metadata features only (TreeExplainer too slow on 3,000+ features), 50-row background
   • ELI5: PermutationImportance on the full 3,024-feature space

8. Framework: scikit-learn 1.6.1

9. Hardware: MacBook Pro M4 Pro, 24 GB unified RAM

Key Hyperparameters

Parameter	Value	Rationale
TF-IDF max_features	3,000	Balance between vocabulary coverage and dimensionality
TF-IDF ngram_range	(1, 3)	Capture phrases like "act now", "limited time offer"
TF-IDF min_df	2	Remove extremely rare words
TF-IDF max_df	0.90	Remove words appearing in 90%+ of emails
TF-IDF sublinear_tf	True	Logarithmic term frequency scaling
RF n_estimators	200	Sufficient trees for stable predictions
class_weight	'balanced'	Handle spam/ham class imbalance
test_size	0.3	70/30 split
random_state	42	Reproducibility

Evolution of the Project

Version	Features	Model	Data	UI
v0.1 (Streamlit)	11 metadata + 3000 TF-IDF	RF + GridSearchCV + Calibration	SpamAssassin + Enron	Streamlit
v0.6	24 metadata + 3000 TF-IDF	Same	+ Kaggle + GitHub	Streamlit
v0.8	24 metadata + 3000 TF-IDF + optional LLM	Same	+ user feedback (5x weighted)	Streamlit + student version
v0.1.0 (Gradio)	24 metadata + 3000 TF-IDF	RF + LR + SVM VotingClassifier	Kaggle 100K	Gradio (rewrite from Streamlit)
v1.1 (merged)	24 metadata + 3000 TF-IDF	VotingClassifier (canonical)	Kaggle + GitHub + optional Spam Assassin	Gradio (canonical)

Known Limitations

• Legitimate marketing emails (e.g., Lenovo Rewards) misclassified as spam at high confidence due to vocabulary overlap with spam patterns (dollar signs, "earn", "rewards")
• TF-IDF is bag-of-words — cannot understand email intent or context
• SHAP limited to metadata features only (KernelExplainer too slow on 3,000+ features)
• The voting model is large (152 MB) — pushing to a free HF Space requires explicit inclusion in the upload script

Citations

• Pedregosa, F., et al. (2011). "Scikit-learn: Machine Learning in Python." JMLR 12, 2825-2830
• Ribeiro, M.T., et al. (2016). "'Why Should I Trust You?': Explaining the Predictions of Any Classifier." KDD 2016 (LIME)
• Lundberg, S.M. & Lee, S.I. (2017). "A Unified Approach to Interpreting Model Predictions." NeurIPS 2017 (SHAP)
• Lundberg, S.M., et al. (2020). "From local explanations to global understanding with explainable AI for trees." Nature MI 2(1) (TreeSHAP)
• Kuzlu, M., et al. (2020). "Gaining Insight Into Solar PV Power Generation Forecasting Utilizing Explainable AI Tools." IEEE Access 8 (XAI feature reduction methodology)
• Breiman, L. (2001). "Random Forests." Machine Learning 45(1)