README.md

metadata

language:
  - en
license: apache-2.0
tags:
  - glove
  - lora
  - distillation
  - hard-negatives
  - projection-head
base_model: jsanzolac/bpe_glove_300_lora_r300_qwen3
datasets:
  - jsanzolac/qwen3_emb_300_packed_cl100k
  - jsanzolac/qwen3_emb_512_hard_negatives

bpe_glove_300_lora_r300_qwen3_proj_hardnegs

r300 backbone (A, B warm-started from jsanzolac/bpe_glove_300_lora_r300_qwen3) + a sentence-level projection head (Linear(300, 1200) → GELU → Dropout(0.1) → Linear(1200, 300) → L2-norm). Both backbone (A, B) and the projection head are trainable.

Loss: InfoNCE(v_proj vs [v_T ‖ v_hards], τ=0.01) with H = 64 mined hard negatives per anchor at batch=256. No MSE term.

Schedule: lr 0.0005 → 1e-05 cosine over 150,000 steps, warmup 1000. Optimizer: AdamW, weight decay 0.01.

Files under rank_300/:

• checkpoint_final.pt — A.weight, B.weight, plus the projection head's proj.l1.{weight,bias} and proj.l2.{weight,bias}. E is excluded (non-persistent buffer); re-inject from jsanzolac/drifting-glove-distilled-r300 at load time.
• config.json
• vectors_drifted_pre_proj.txt — E + B(A(·)) per vocab row (PRE-projection; the head is contextual).
• train_log.jsonl