File size: 4,870 Bytes
88959c5 b35b735 a338a8f 88959c5 a338a8f 7cfa7b9 a338a8f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 | ---
license: gemma
language:
- en
- zh
library_name: peft
tags:
- lora
- peft
- distillation
- svd
- cross-architecture
- adaptive-rank
- gemma
- llama
- nemotron
base_model: Changgil/google-gemma-3-27b-it-text
---
# Nemotron-70B → Gemma-3 27B (Text) SVD-LoRA Adapter (Adaptive Rank)
中文版本請見：**[README_ZH.md](README_ZH.md)**
This repository provides a **PEFT LoRA adapter** for `Changgil/google-gemma-3-27b-it-text`, distilled from `nvidia/Llama-3.1-Nemotron-70B-Instruct-HF` using **weight-delta SVD-LoRA distillation** (cross-architecture).
- **Base model (student / required):** `Changgil/google-gemma-3-27b-it-text`
- **Teacher model (reference):** `nvidia/Llama-3.1-Nemotron-70B-Instruct-HF`
- **Artifact:** LoRA adapter (PEFT) — *not* a full merged model
- **Scope:** Applies to attention + MLP modules (`self_attn|mlp`)
---
## What is this?
This adapter approximates the teacher→student **weight delta** (Δ) with low-rank factors, and stores them as LoRA matrices. It is designed for **cross-architecture** distillation where teacher/student differ in layer count and hidden size.
Key build characteristics (as used for this adapter):
- **SVD backend:** `aurora` (AURORA-SVD)
- **Adaptive rank:** enabled via energy threshold
- **Teacher mixing:** `lsq` (per-matrix least-squares mixing)
- **Calibration:** RMS-based calibration from Alpaca-format samples
---
## Quickstart (Transformers + PEFT)
> This is an adapter. You must load the base model first.
```python
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
base_id = "Changgil/google-gemma-3-27b-it-text"
adapter_id = "win10/Nemotron2Gemma-AURORA-LoRA-27B-IT-0p95"
tokenizer = AutoTokenizer.from_pretrained(base_id, use_fast=True)
base = AutoModelForCausalLM.from_pretrained(
base_id,
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(base, adapter_id)
model.eval()
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain knowledge distillation in 5 bullet points."},
]
inputs = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
tokenize=True,
return_tensors="pt",
)
with torch.no_grad():
out = model.generate(
inputs.to(model.device),
max_new_tokens=512,
do_sample=False,
)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```
---
## Optional: Merge the adapter into the base weights
If you need a single merged checkpoint for inference:
```python
from peft import PeftModel
merged = model.merge_and_unload()
merged.save_pretrained("./merged_model", safe_serialization=True)
tokenizer.save_pretrained("./merged_model")
```
---
## Reproducibility (build command)
The adapter was produced with a command equivalent to:
```bash
python universal_distill_v4_1_0_aurora_svd_innovations.py \
--teacher E:\text-generation-webui-1.14\user_data\models\Llama-3.1-Nemotron-70B-Instruct-HF \
--student E:\text-generation-webui-1.14\user_data\models\google-gemma-3-27b-it-text \
--output ./Llama-3.1-Nemotron-70B-Instruct-HF-gemma-3-27b-it-text-lora-adaptive \
--svd-mode aurora \
--energy-threshold 0.95 \
--min-rank 256 \
--max-rank 5376 \
--interp-mode lsq \
--svd-rand-iter 2 \
--svd-rand-oversamples 8 \
--svd-aurora-steps 100 \
--svd-aurora-order 2 \
--calib-format alpaca \
--calib-alpaca-template classic \
--calib-max-samples 128 \
--calib-max-length 65536 \
--calib-batch-size 2 \
--calib-save .\calib_stats_Yi-70B-200k_alpaca-taiwan-dataset.safetensors \
--calib-mode rms \
--include "self_attn|mlp"
```
Observed run summary (example log):
- Teacher tensors: 723
- Student tensors: 808
- Teacher: GQA + SwiGLU, 80 layers, hidden 8192
- Student: GQA + standard FFN, 62 layers, hidden 5376
- TIES: enabled (density=0.3)
- DARE: disabled
---
## Compatibility notes
- This adapter targets the exact module naming / shapes of `Changgil/google-gemma-3-27b-it-text`.
- If you use a different Gemma-3 27B variant, it must be shape-compatible (otherwise adapter load will fail).
---
## Limitations
- This is **weight-space distillation** (delta approximation). It can transfer behavior/style partially, but it is not guaranteed to fully match the teacher across all tasks.
- Output quality depends on base model prompting/chat template and decoding settings.
---
## Source models
- Base model: https://huggingface.co/Changgil/google-gemma-3-27b-it-text
- Teacher model: https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct-HF
---
## License
Please follow the license and usage terms of the **base model** and **teacher model** as listed on their Hugging Face pages. This repository only provides an adapter; downstream usage must remain compliant with upstream terms.
|