| --- |
| license: apache-2.0 |
| language: en |
| tags: |
| - draft-refine |
| - block-diffusion |
| - nce |
| - qwen3-4b |
| --- |
| |
| # Qwen3-4B Stage-3 NCE — temp07-perround variant |
|
|
| Stage-3 Noise-Contrastive-Estimation training resumed from the Stage-2 |
| end ckpt at step 7335. NCE phase trains the scorer head to rank K=4 |
| candidate completions per block. Differs from beam-perround in proposal |
| sampling: this variant uses `softmax_sampling` at proposal_temperature |
| =0.7 (vs argmax in beam-perround) — closer to training-time scorer |
| noise. |
| |
| ## Files |
| |
| | File | Size | |
| |---|---| |
| | `model.pt` | 8.5 GB | |
| | `optimizer.pt` | 0.45 GB | |
| | `scheduler.pt` | 1.7 KB | |
| | `eval_batches.pt` | 13 MB | |
| | `rng_rank{0..23}.pt` | 14.7 KB each | |
|
|
| Total: ~9.54 GB / 30 files. Full resume state for re-training. |
|
|
| ## Step + lineage |
|
|
| - Resume from: Stage-2 ckpt at step 7335 |
| - This ckpt: step 9124 (1789 NCE-phase steps) |
| - Proposal: `combine=softmax_sampling`, `proposal_temperature=0.7` |
| - Config: `configs/large_scale/qwen3_4b_stage3_nce_resume7335_temp07_perround_6n.yaml` |
|
|
| ## Note on inference reproducibility |
|
|
| When evaluating this ckpt, use: |
| ``` |
| combine=softmax_sampling proposal_temperature=0.7 |
| ``` |
| to MATCH the training distribution. Using `combine=beam_bayes argmax` |
| (as for the perround/lastround variants) on this ckpt produces an |
| out-of-distribution scorer signal. |
|
|
| ## Related archives |
|
|
| - `haotiansun014/qwen3-4b-stage3-nce-7335-perround-archive` |
| - `haotiansun014/qwen3-4b-stage3-nce-7335-lastround-archive` |
|
|
