README.md

---
language:
- en
license: apache-2.0
tags:
- glove
- lora
- distillation
- hard-negatives
- qwen3-embedding
base_model: jsanzolac/bpe_glove_300_lora_r300_qwen3
datasets:
- jsanzolac/qwen3_emb_300_packed_cl100k
- jsanzolac/qwen3_emb_512_hard_negatives
---

# bpe_glove_300_lora_r300_qwen3_hardnegs

Continuation of `jsanzolac/bpe_glove_300_lora_r300_qwen3` — same `DriftingGloVeStudent` rank=300 over a frozen
300-d cl100k BPE-GloVe — trained for an additional 150,000 steps with **mined
hard negatives** from `jsanzolac/qwen3_emb_512_hard_negatives`.

**Loss:** `cross_entropy(v @ [v_T ‖ v_hards]^T / τ) + λ_MSE · MSE(v, v_T)`  with
`τ = 0.05`, `λ_MSE = 1.0`, `H = 64` mined hard negatives per anchor.

**Warm-start:** `A.weight` + `B.weight` from `jsanzolac/bpe_glove_300_lora_r300_qwen3/rank_300/checkpoint_final.pt`. Optimizer
state was not in the source checkpoint, so this run uses a fresh LR schedule
(5e-4 → 1e-5 cosine over 150,000 steps).

**Frozen:** `E` (300-d GloVe from `jsanzolac/drifting-glove-distilled-r300`), teacher (only used to produce the
cached `v_T` targets in `jsanzolac/qwen3_emb_300_packed_cl100k` — not loaded here).

Files under `rank_300/`:
- `checkpoint_final.pt` — `A.weight` + `B.weight` (E excluded; reinject from `jsanzolac/drifting-glove-distilled-r300`).
- `config.json`
- `vectors_drifted.txt` / `.parquet` — `E + B(A(·))` per vocab row.
- `train_log.jsonl`