---
language:
- en
license: apache-2.0
tags:
- glove
- lora
- distillation
- bpe
- cl100k_base
- ffn
base_model: jsanzolac/bpe_glove_512
datasets:
- jsanzolac/qwen3_emb_512
- jsanzolac/qwen3_emb_512_packed
---

# bpe_glove_512_lora_v1_ffn

Warm-start from `jsanzolac/bpe_glove_512_lora_v1/rank_512` plus a per-token FFN inserted
between the GloVe-attention output and the alpha-pool collapse.

**Trainable:** `A`, `B`, FFN.  **Frozen:** `E`, teacher.  
**Loss:** `λ_c·InfoNCE + λ_D·‖ρ_T − ρ_S‖²_F` with `λ_c=1.0`, `λ_D=0.1`.  
Density is computed on the **post-FFN** per-token states; InfoNCE is on the alpha-pooled sentence vector.

Files:
- `rank_512/checkpoint_final.pt` — A + B + FFN state dict (E is non-persistent; re-inject from `jsanzolac/bpe_glove_512/vectors.txt`).
- `rank_512/config.json` — full hyperparameters.
- `rank_512/vectors_drifted.txt` — `E + B(A(·))` per vocab row, GloVe text format. Note: this captures only the static drifted embedding lookup, **not** the FFN's effect (which is contextual). To use the model end-to-end, instantiate `DriftingGloVeStudentFFN` and run forward.
- `rank_512/train_log.jsonl` — per-step metrics.