README.md

---
language:
- en
license: apache-2.0
tags:
- glove
- lora
- distillation
- hard-negatives
- qkv-split
base_model: jsanzolac/bpe_glove_300_lora_r300_qwen3
datasets:
- jsanzolac/qwen3_emb_300_packed_cl100k
- jsanzolac/qwen3_emb_512_hard_negatives
---

# bpe_glove_300_qkv_v_only_hardnegs

QKV-split LoRA student on top of the 300-d cl100k BPE-GloVe (`jsanzolac/drifting-glove-distilled-r300`).

- **Q** = frozen E.
- **K** = E + frozen A_K·B_K, loaded from `jsanzolac/bpe_glove_300_lora_r300_qwen3/rank_300/checkpoint_final.pt`.
- **V** = E + trainable A_V·B_V (rank=300, full).

**Loss:** `InfoNCE(v_S vs [v_T ‖ v_hards], τ=0.05) + 0.1·MSE(v_S, v_T)`
with `H = 64` mined hard negatives per anchor at batch=256.

**Schedule:** lr 0.0005 → 1e-05 cosine over 150,000 steps, warmup 1000.
Optimizer: AdamW, weight decay 0.01.

Files under `rank_300/`:
- `checkpoint_final.pt` — A_V.weight + B_V.weight (frozen E, A_K, B_K NOT included).
- `config.json`
- `vectors_drifted_V.txt` — `E + B_V(A_V(·))` per vocab row (V-side static drift only).
- `train_log.jsonl`

**To reconstruct the full model at inference:** load E + (A_K, B_K) from `jsanzolac/bpe_glove_300_lora_r300_qwen3`, load (A_V, B_V) from this repo, then run the QKV forward pass.