README.md

---
license: gemma
base_model: google/gemma-4-E2B-it
library_name: peft
pipeline_tag: text-generation
language:
- en
tags:
- phishing
- email-classification
- security
- lora
- gguf
- gemma4
datasets:
- zefang-liu/phishing-email-dataset
---

# Gemma-4-E2B-it — Phishing Email Classifier (LoRA)

LoRA fine-tune of [google/gemma-4-E2B-it](https://huggingface.co/google/gemma-4-E2B-it) that classifies emails as **`phishing`** or **`legit`** with a single-word answer.

This repo contains:
- the **LoRA adapter** (PEFT, ~50 MB) — reproducible, composable
- ready-to-run **GGUF** quants (merged model) for llama.cpp / LM Studio: `Q4_K_M` (3.4 GB) and `Q8_0` (5 GB)

## Results

Evaluated on 1,000 held-out emails (balanced, English):

| Model | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|
| gemma-4-E2B-it zero-shot | 85.7% | 90.3% | 78.9% | 0.842 |
| **+ this LoRA** | **95.6%** | **98.4%** | **98.0%** | **0.982** |

Notes: 26/1000 outputs were off-format and counted as errors (accuracy on parsable outputs: 98.2%). Zero-shot baseline measured on a 300-email subset.

## Usage

The model expects this exact system prompt (it was trained with it):

```
You are an email security classifier. Classify the email as 'phishing' or 'legit'. Respond with one word only.
```

### LM Studio / llama.cpp (GGUF)

Download `phishing-gemma4-e2b-Q4_K_M.gguf`, set the system prompt above, temperature 0, paste an email as the user message. Requires a llama.cpp build recent enough for the `gemma4` architecture.

### Transformers + PEFT (adapter)

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

BASE = "google/gemma-4-E2B-it"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "vete-speis/gemma-4-E2B-it-phishing")

msgs = [
    {"role": "system", "content": "You are an email security classifier. Classify the email as 'phishing' or 'legit'. Respond with one word only."},
    {"role": "user", "content": "Your account has been suspended. Verify now: http://secure-login.example.xyz"},
]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=5, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))  # -> phishing
```

## Training

- **Base:** google/gemma-4-E2B-it (2.3B effective params)
- **Data:** [zefang-liu/phishing-email-dataset](https://huggingface.co/datasets/zefang-liu/phishing-email-dataset), cleaned, class-balanced 50/50, 10.5k train / 500 val / 1,000 test, emails truncated to 1,500 chars
- **Method:** SFT with TRL `SFTTrainer` + PEFT LoRA — r=16, alpha=32, dropout 0.05, targets: the inner `nn.Linear` of q/k/v/o projections (Gemma 4 wraps them in `Gemma4ClippableLinear`, which PEFT cannot wrap directly)
- **Run:** 1 epoch, effective batch 16 (4×4), seq len 512, lr 2e-4 cosine, bf16, gradient checkpointing — ~15 min on a single H100 (Modal)

## Limitations

- Trained on **English** emails. It generalizes surprisingly well to other languages (the base is multilingual), but no formal eval outside English.
- The dataset's "phishing" class includes generic spam, so the model leans toward flagging unsolicited marketing/cold-outreach emails as `phishing`. If you need a strict spam ≠ phishing distinction, you need 3-class data.
- ~2.6% of outputs may be off-format; parse with a contains-check on `phishing`/`legit` and treat anything else as abstention.
- Not a substitute for a mail security gateway. Use as a triage aid.

## License

Gemma derivatives are governed by the [Gemma Terms of Use](https://ai.google.dev/gemma/terms). The adapter and GGUF files here are derivatives of google/gemma-4-E2B-it.