leodriesch/novice-qwen3.5-4b-rf0.1

A Novice checkpoint produced with the Master/Novice framework: the base model's MLP weights are disentangled into truncatable factors (Q/U/s/Vh) so a weaker model can be reconstructed at any chosen rank.

How it was made

Base model	Qwen/Qwen3.5-4B
Rank fraction	0.1
Disentangle steps	40
Learning rate	0.001
Factored MLP layers	96

Files

• *.factors.pt + manifest.pt — the disentangled factors, stored one shard per MLP linear plus a manifest. This sharded layout lets the loader materialize one layer at a time, so memory stays near the base model + a single layer (the full factor set is several × the model and would otherwise OOM).
• novice-qwen3.5-4b-rf0.1_tuning.csv — per-step tuning trace (one row per step per matrix: cost, entropy, effective rank) plus the run's hyperparameters, for plotting/analysis.

Usage

from huggingface_hub import snapshot_download
from novice_model import NoviceModel

# download just the factor shards + manifest
ckpt_dir = snapshot_download("leodriesch/novice-qwen3.5-4b-rf0.1", allow_patterns=["*.pt"])

# loads layer-by-layer, truncating to the chosen rank — memory-flat
fast = NoviceModel.from_sharded_checkpoint(ckpt_dir, rank_fraction=0.1, device="cuda")
out = fast.generate(**fast.tokenizer("Hello", return_tensors="pt").to(fast.device))
print(fast.tokenizer.decode(out[0]))

Tuning curves

Per-step disentanglement cost for every MLP matrix (log scale), colored by linear type. gate_proj/up_proj converge strongly; down_proj (transposed shape) is far less compressible at this rank.