| --- |
| language: |
| - en |
| license: apache-2.0 |
| tags: |
| - glove |
| - lora |
| - distillation |
| - hard-negatives |
| - qwen3-embedding |
| base_model: jsanzolac/bpe_glove_300_lora_r300_qwen3 |
| datasets: |
| - jsanzolac/qwen3_emb_300_packed_cl100k |
| - jsanzolac/qwen3_emb_512_hard_negatives |
| --- |
| |
| # bpe_glove_300_lora_r300_qwen3_hardnegs |
|
|
| Continuation of `jsanzolac/bpe_glove_300_lora_r300_qwen3` — same `DriftingGloVeStudent` rank=300 over a frozen |
| 300-d cl100k BPE-GloVe — trained for an additional 150,000 steps with **mined |
| hard negatives** from `jsanzolac/qwen3_emb_512_hard_negatives`. |
|
|
| **Loss:** `cross_entropy(v @ [v_T ‖ v_hards]^T / τ) + λ_MSE · MSE(v, v_T)` with |
| `τ = 0.05`, `λ_MSE = 1.0`, `H = 64` mined hard negatives per anchor. |
|
|
| **Warm-start:** `A.weight` + `B.weight` from `jsanzolac/bpe_glove_300_lora_r300_qwen3/rank_300/checkpoint_final.pt`. Optimizer |
| state was not in the source checkpoint, so this run uses a fresh LR schedule |
| (5e-4 → 1e-5 cosine over 150,000 steps). |
|
|
| **Frozen:** `E` (300-d GloVe from `jsanzolac/drifting-glove-distilled-r300`), teacher (only used to produce the |
| cached `v_T` targets in `jsanzolac/qwen3_emb_300_packed_cl100k` — not loaded here). |
|
|
| Files under `rank_300/`: |
| - `checkpoint_final.pt` — `A.weight` + `B.weight` (E excluded; reinject from `jsanzolac/drifting-glove-distilled-r300`). |
| - `config.json` |
| - `vectors_drifted.txt` / `.parquet` — `E + B(A(·))` per vocab row. |
| - `train_log.jsonl` |
|
|
