README.md

metadata

language:
  - en
license: apache-2.0
tags:
  - glove
  - lora
  - distillation
  - hard-negatives
  - qkv-split
base_model: jsanzolac/bpe_glove_300_lora_r300_qwen3
datasets:
  - jsanzolac/qwen3_emb_300_packed_cl100k
  - jsanzolac/qwen3_emb_512_hard_negatives

bpe_glove_300_qkv_v_only_hardnegs

QKV-split LoRA student on top of the 300-d cl100k BPE-GloVe (jsanzolac/drifting-glove-distilled-r300).

• Q = frozen E.
• K = E + frozen A_K·B_K, loaded from jsanzolac/bpe_glove_300_lora_r300_qwen3/rank_300/checkpoint_final.pt.
• V = E + trainable A_V·B_V (rank=300, full).

Loss: InfoNCE(v_S vs [v_T ‖ v_hards], τ=0.05) + 0.1·MSE(v_S, v_T) with H = 64 mined hard negatives per anchor at batch=256.

Schedule: lr 0.0005 → 1e-05 cosine over 150,000 steps, warmup 1000. Optimizer: AdamW, weight decay 0.01.

Files under rank_300/:

• checkpoint_final.pt — A_V.weight + B_V.weight (frozen E, A_K, B_K NOT included).
• config.json
• vectors_drifted_V.txt — E + B_V(A_V(·)) per vocab row (V-side static drift only).
• train_log.jsonl

To reconstruct the full model at inference: load E + (A_K, B_K) from jsanzolac/bpe_glove_300_lora_r300_qwen3, load (A_V, B_V) from this repo, then run the QKV forward pass.