Instructions to use jkim96/phi-4-DASHQ-INT3-g64 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jkim96/phi-4-DASHQ-INT3-g64 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jkim96/phi-4-DASHQ-INT3-g64", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jkim96/phi-4-DASHQ-INT3-g64", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("jkim96/phi-4-DASHQ-INT3-g64", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jkim96/phi-4-DASHQ-INT3-g64 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jkim96/phi-4-DASHQ-INT3-g64"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkim96/phi-4-DASHQ-INT3-g64",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jkim96/phi-4-DASHQ-INT3-g64

SGLang

How to use jkim96/phi-4-DASHQ-INT3-g64 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jkim96/phi-4-DASHQ-INT3-g64" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkim96/phi-4-DASHQ-INT3-g64",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jkim96/phi-4-DASHQ-INT3-g64" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jkim96/phi-4-DASHQ-INT3-g64",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jkim96/phi-4-DASHQ-INT3-g64 with Docker Model Runner:
```
docker model run hf.co/jkim96/phi-4-DASHQ-INT3-g64
```

jkim96 commited on 3 days ago

Commit

7e602b4

verified ·

1 Parent(s): dab3fac

Upload DASH-Q quantized checkpoint (INTNone, gNone)

Browse files

Files changed (12) hide show

README.md +73 -0
chat_template.jinja +1 -0
config.json +56 -0
dashq_config.json +1960 -0
dashq_kernel.py +301 -0
generation_config.json +11 -0
model-00001-of-00002.safetensors +3 -0
model-00002-of-00002.safetensors +3 -0
model.safetensors.index.json +571 -0
modeling_dashq.py +227 -0
tokenizer.json +0 -0
tokenizer_config.json +12 -0

README.md ADDED Viewed

	@@ -0,0 +1,73 @@

+---
+license: apache-2.0
+base_model: microsoft/phi-4
+library_name: transformers
+tags:
+- dashq
+- quantized
+- post-training-quantization
+---
+# phi-4-DASHQ-INT3-g64
+This repository contains a DASH-Q packed quantized checkpoint for `microsoft/phi-4`.
+DASH-Q checkpoints require the lightweight DASH-Q runtime package for loading. They are not plain Transformers checkpoints because linear layers are stored as `PackedQuantizedLinear` modules.
+## Usage
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+model = AutoModelForCausalLM.from_pretrained(
+    "jkim96/phi-4-DASHQ-INT3-g64", trust_remote_code=True, device_map="cuda", dtype="auto"
+)
+tokenizer = AutoTokenizer.from_pretrained("jkim96/phi-4-DASHQ-INT3-g64")
+messages = [{"role": "user", "content": "Explain 2-bit quantization in one sentence."}]
+text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
+inputs = tokenizer(text, return_tensors="pt").to(model.device)
+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=256)[0]))
+```
+`trust_remote_code=True` is required: the checkpoint ships its quantized-layer
+implementation (`modeling_dashq.py`) and Triton kernels (`dashq_kernel.py`).
+Without Triton, or on CPU, it falls back to dequantize-and-matmul in PyTorch.
+### Requirements
+| Package | Minimum | Verified with |
+| --- | --- | --- |
+| `torch` | 2.4 | 2.12.1+cu130 |
+| `transformers` | 5.8 | 5.9.0 |
+| `triton` | 3.0 (Linux; bundled with CUDA builds of PyTorch) | 3.7.1 |
+| `huggingface_hub` | 1.5 (pulled in by transformers) | 1.15.0 |
+## Quantization
+| Field | Value |
+| --- | --- |
+| Base model | `microsoft/phi-4` |
+| Bits | `3` |
+| Group size | `64` |
+| Scale/zero dtype | `float16` |
+| Calibration dataset | `wikitext2` |
+| Calibration samples | `128` |
+| Sequence length | `2048` |
+| Original size | `29.3190 GB` |
+| Quantized size | `8.3607 GB` |
+## Evaluation
+| Metric | Value |
+| --- | ---: |
+| `wikitext2_ppl` | 6.7354 |
+| `zero-shot accuracy avg` | 68.5413 |
+| `arc_challenge` | 55.2048 |
+| `arc_easy` | 73.9057 |
+| `commonsense_qa` | 76.3309 |
+| `hellaswag` | 80.4621 |
+| `lambada_openai` | 73.5494 |
+| `openbookqa` | 44.8000 |
+| `piqa` | 79.8694 |
+| `truthfulqa_mc2` | 57.9272 |
+| `winogrande` | 74.8224 |

chat_template.jinja ADDED Viewed

	@@ -0,0 +1 @@

+ {% for message in messages %}{% if (message['role'] == 'system') %}{{'<|im_start|>system<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'user') %}{{'<|im_start|>user<|im_sep|>' + message['content'] + '<|im_end|>'}}{% elif (message['role'] == 'assistant') %}{{'<|im_start|>assistant<|im_sep|>' + message['content'] + '<|im_end|>'}}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant<|im_sep|>' }}{% endif %}

config.json ADDED Viewed

	@@ -0,0 +1,56 @@

+{
+  "architectures": [
+    "DashQPhi3ForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoModelForCausalLM": "modeling_dashq.DashQPhi3ForCausalLM"
+  },
+  "bos_token_id": 100257,
+  "dashq": {
+    "format": "dashq-packed-linear",
+    "format_version": 1,
+    "layer_metadata": "dashq_config.json",
+    "method": "dashq",
+    "n_quantized_modules": 160,
+    "params": {
+      "bits": 3,
+      "group_size": 64,
+      "low_memory_optimization": false,
+      "moe_hessian_scope": "shared",
+      "n_samples": 128,
+      "scale_zero_dtype": "float16",
+      "symmetric": false,
+      "use_error_compensation": true,
+      "use_optimal_shrinkage": true,
+      "use_weighted_quantization": true
+    }
+  },
+  "dtype": "bfloat16",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 100265,
+  "hidden_act": "silu",
+  "hidden_size": 5120,
+  "initializer_range": 0.02,
+  "intermediate_size": 17920,
+  "max_position_embeddings": 16384,
+  "model_type": "phi3",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 10,
+  "original_max_position_embeddings": 16384,
+  "pad_token_id": 100349,
+  "resid_pdrop": 0.0,
+  "rms_norm_eps": 1e-05,
+  "rope_parameters": {
+    "partial_rotary_factor": 1.0,
+    "rope_theta": 250000,
+    "rope_type": "default"
+  },
+  "sliding_window": null,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.15.0",
+  "use_cache": false,
+  "vocab_size": 100352
+}

dashq_config.json ADDED Viewed

	@@ -0,0 +1,1960 @@

+{
+  "base_model": "microsoft/phi-4",
+  "format": "dashq-packed-linear",
+  "format_version": 1,
+  "method": "dashq",
+  "model_class": "causal_lm",
+  "params": {
+    "bits": 3,
+    "group_size": 64,
+    "low_memory_optimization": false,
+    "moe_hessian_scope": "shared",
+    "n_samples": 128,
+    "scale_zero_dtype": "float16",
+    "symmetric": false,
+    "use_error_compensation": true,
+    "use_optimal_shrinkage": true,
+    "use_weighted_quantization": true
+  },
+  "quantized_modules": {
+    "model.layers.0.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.0.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.0.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.0.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.1.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.1.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.1.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.1.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.10.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.10.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.10.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.10.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.11.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.11.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.11.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.11.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.12.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.12.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.12.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.12.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.13.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.13.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.13.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.13.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.14.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.14.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.14.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.14.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.15.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.15.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.15.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.15.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.16.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.16.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.16.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.16.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.17.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.17.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.17.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.17.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.18.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.18.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.18.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.18.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.19.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.19.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.19.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.19.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.2.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.2.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.2.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.2.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.20.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.20.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.20.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.20.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.21.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.21.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.21.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.21.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.22.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.22.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.22.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.22.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.23.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.23.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.23.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.23.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.24.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.24.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.24.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.24.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.25.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.25.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.25.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.25.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.26.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.26.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.26.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.26.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.27.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.27.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.27.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.27.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.28.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.28.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.28.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.28.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.29.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.29.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.29.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.29.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.3.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.3.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.3.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.3.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.30.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.30.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.30.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.30.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.31.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.31.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.31.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.31.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.32.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.32.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.32.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.32.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.33.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.33.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.33.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.33.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.34.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.34.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.34.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.34.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.35.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.35.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.35.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.35.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.36.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.36.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.36.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.36.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.37.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.37.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.37.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.37.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.38.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.38.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.38.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.38.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.39.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.39.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.39.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.39.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.4.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.4.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.4.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.4.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.5.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.5.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.5.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.5.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.6.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.6.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.6.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.6.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.7.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.7.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.7.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.7.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.8.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.8.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.8.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.8.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.9.mlp.down_proj": {
+      "group_size": 64,
+      "in_features": 17920,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 1433600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 17920,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.9.mlp.gate_up_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 2867200,
+      "out_features": 35840,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.9.self_attn.o_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 409600,
+      "out_features": 5120,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    },
+    "model.layers.9.self_attn.qkv_proj": {
+      "group_size": 64,
+      "in_features": 5120,
+      "linear_dtype": "bfloat16",
+      "nbits": 3,
+      "num_groups": 614400,
+      "out_features": 7680,
+      "packing": "int3_packed_u32",
+      "quant_in_features": 5120,
+      "runtime_backend": "torch",
+      "scale_zero_dtype": "float16"
+    }
+  },
+  "results": {
+    "Method": "dashq",
+    "Model": "microsoft/phi-4",
+    "ModelSizeGB": 8.360666784,
+    "OriginalSizeGB": 29.319042992,
+    "PPL": 6.735445976257324,
+    "Params": "{'bits': 3, 'group_size': 64, 'scale_zero_dtype': 'float16', 'n_samples': 128, 'moe_hessian_scope': 'shared', 'use_error_compensation': True, 'use_optimal_shrinkage': True, 'use_weighted_quantization': True, 'symmetric': False, 'low_memory_optimization': False}",
+    "QuantTime": 1049.0759711265564,
+    "arc_challenge": 55.20477815699659,
+    "arc_easy": 73.9057239057239,
+    "commonsense_qa": 76.33087633087634,
+    "hellaswag": 80.46205935072695,
+    "lambada_openai": 73.5493887056084,
+    "openbookqa": 44.800000000000004,
+    "piqa": 79.86942328618063,
+    "truthfulqa_mc2": 57.92717446545296,
+    "winogrande": 74.82241515390686,
+    "zeroshot_avg": 68.5413154839414
+  }
+}

dashq_kernel.py ADDED Viewed

	@@ -0,0 +1,301 @@

+"""Triton kernels for group-wise asymmetric integer weights.
+Weights are stored K-major: W_q has shape (K // elements_per_word, N) with
+values packed along K, and scale/zero have shape (K // group_size, N).
+Three kernels are selected by the number of input rows M:
+    M == 1        GEMV
+    2 <= M <= 32  fused dequantize-GEMM with split-K
+    M > 32        fused dequantize-GEMM, accumulator kept in registers
+Supported bit widths are 1, 2, 3, 4 and 8. 3-bit is stored as a 2-bit plane
+plus a 1-bit plane, so it occupies exactly 3 bits per weight.
+"""
+from __future__ import annotations
+from typing import Optional
+import torch
+import torch.nn as nn
+try:
+    import triton
+    import triton.language as tl
+    TRITON_AVAILABLE = True
+except Exception:  # triton is optional
+    TRITON_AVAILABLE = False
+SUPPORTED_NBITS = (1, 2, 3, 4, 8)
+_GEMM_CONFIG_CACHE = {}
+if TRITON_AVAILABLE:
+    @triton.jit
+    def _dashq_gemv_kernel(
+        x_ptr, w_ptr, lo_ptr, s_ptr, z_ptr, y_ptr,
+        N, K,
+        NBITS: tl.constexpr, EPS: tl.constexpr, GS: tl.constexpr,
+        BLOCK_N: tl.constexpr, BLOCK_K: tl.constexpr,
+    ):
+        pid_n = tl.program_id(0)
+        pid_k = tl.program_id(1) * 2
+        offs_n = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)
+        offs_n = tl.max_contiguous(tl.multiple_of(offs_n, BLOCK_N), BLOCK_N)
+        # 2 * BLOCK_K == GS, so a program covers exactly one scale group.
+        k_m = (pid_k * BLOCK_K) // GS
+        scales = tl.load(s_ptr + k_m * N + offs_n).to(tl.float32)
+        zeros = tl.load(z_ptr + k_m * N + offs_n).to(tl.float32)
+        acc = tl.zeros((BLOCK_N,), dtype=tl.float32)
+        offs_k = pid_k * BLOCK_K + tl.arange(0, BLOCK_K)
+        for _ in tl.static_range(2):
+            a = tl.load(x_ptr + offs_k, eviction_policy="evict_last").to(tl.float32)
+            if NBITS == 3:
+                hw = tl.load(
+                    w_ptr + (offs_k // 16)[:, None] * N + offs_n[None, :],
+                    eviction_policy="evict_first",
+                )
+                lw = tl.load(
+                    lo_ptr + (offs_k // 32)[:, None] * N + offs_n[None, :],
+                    eviction_policy="evict_first",
+                )
+                q = (((hw >> (((offs_k % 16) * 2)[:, None])) & 3) << 1) | (
+                    (lw >> ((offs_k % 32)[:, None])) & 1
+                )
+            else:
+                wv = tl.load(
+                    w_ptr + (offs_k // EPS)[:, None] * N + offs_n[None, :],
+                    eviction_policy="evict_first",
+                )
+                q = (wv >> (((offs_k % EPS) * NBITS)[:, None])) & ((1 << NBITS) - 1)
+            b = (q.to(tl.float32) - zeros[None, :]) * scales[None, :]
+            acc += tl.sum(a[:, None] * b, axis=0)
+            offs_k += BLOCK_K
+        tl.atomic_add(y_ptr + offs_n, acc, sem="relaxed")
+    @triton.jit
+    def _dashq_gemm_kernel(
+        x_ptr, w_ptr, lo_ptr, s_ptr, z_ptr, y_ptr,
+        M, N, K,
+        NBITS: tl.constexpr, EPS: tl.constexpr, GS: tl.constexpr,
+        BLOCK_M: tl.constexpr, BLOCK_N: tl.constexpr, BLOCK_K: tl.constexpr,
+        SPLIT_K: tl.constexpr, OUT_DTYPE: tl.constexpr,
+    ):
+        """y[M, N] = x[M, K] @ dequantize(w)[K, N]
+        BLOCK_K divides the group size, so a K-tile lies inside one group and the
+        scale/zero load is a single (1, BLOCK_N) vector.
+        """
+        pid_m = tl.program_id(0)
+        pid_n = tl.program_id(1)
+        pid_k = tl.program_id(2)
+        offs_m = pid_m * BLOCK_M + tl.arange(0, BLOCK_M)
+        offs_n = pid_n * BLOCK_N + tl.arange(0, BLOCK_N)
+        mask_m = offs_m < M
+        mask_n = offs_n < N
+        acc = tl.zeros((BLOCK_M, BLOCK_N), dtype=tl.float32)
+        for t in range(pid_k, tl.cdiv(K, BLOCK_K), SPLIT_K):
+            k0 = t * BLOCK_K
+            offs_k = k0 + tl.arange(0, BLOCK_K)
+            mask_k = offs_k < K
+            x = tl.load(x_ptr + offs_m[:, None] * K + offs_k[None, :],
+                        mask=mask_m[:, None] & mask_k[None, :], other=0.0)
+            if NBITS == 3:
+                hw = tl.load(w_ptr + (offs_k // 16)[:, None] * N + offs_n[None, :],
+                             mask=mask_k[:, None] & mask_n[None, :], other=0)
+                lw = tl.load(lo_ptr + (offs_k // 32)[:, None] * N + offs_n[None, :],
+                             mask=mask_k[:, None] & mask_n[None, :], other=0)
+                q = (((hw >> (((offs_k % 16) * 2)[:, None])) & 3) << 1) | (
+                    (lw >> ((offs_k % 32)[:, None])) & 1)
+            else:
+                wv = tl.load(w_ptr + (offs_k // EPS)[:, None] * N + offs_n[None, :],
+                             mask=mask_k[:, None] & mask_n[None, :], other=0)
+                q = (wv >> (((offs_k % EPS) * NBITS)[:, None])) & ((1 << NBITS) - 1)
+            g = k0 // GS
+            s = tl.load(s_ptr + g * N + offs_n, mask=mask_n, other=0.0).to(tl.float32)
+            z = tl.load(z_ptr + g * N + offs_n, mask=mask_n, other=0.0).to(tl.float32)
+            w = (q.to(tl.float32) - z[None, :]) * s[None, :]
+            acc += tl.dot(x, w.to(x.dtype), out_dtype=tl.float32)
+        out = acc.to(OUT_DTYPE)
+        y_ptrs = y_ptr + offs_m[:, None] * N + offs_n[None, :]
+        if SPLIT_K == 1:
+            tl.store(y_ptrs, out, mask=mask_m[:, None] & mask_n[None, :])
+        else:
+            tl.atomic_add(y_ptrs, out, mask=mask_m[:, None] & mask_n[None, :], sem="relaxed")
+def _pack_kmajor(q_kn: torch.Tensor, bits: int) -> torch.Tensor:
+    """(K, N) codes -> (K // eps, N) int32, value k in word k // eps."""
+    K, N = q_kn.shape
+    eps = 32 // bits
+    v = q_kn.to(torch.int32).reshape(K // eps, eps, N)
+    words = torch.zeros(K // eps, N, dtype=torch.int32, device=q_kn.device)
+    for j in range(eps):
+        words |= v[:, j, :] << (bits * j)
+    return words
+def _unpack_kmajor(words: torch.Tensor, bits: int, K: int) -> torch.Tensor:
+    eps = 32 // bits
+    WK, N = words.shape
+    shifts = (torch.arange(eps, device=words.device, dtype=torch.int32) * bits).view(1, eps, 1)
+    q = (words.view(WK, 1, N) >> shifts) & ((1 << bits) - 1)
+    return q.reshape(WK * eps, N)[:K]
+class TritonQuantLinear(nn.Module):
+    """Linear layer over group-wise asymmetric integer weights."""
+    def __init__(
+        self,
+        W_int: torch.Tensor,          # (out_features, in_features) integer codes
+        scale: torch.Tensor,          # (out_features, num_groups)
+        zero: torch.Tensor,           # (out_features, num_groups)
+        nbits: int,
+        group_size: int,
+        bias: Optional[torch.Tensor] = None,
+        out_dtype: torch.dtype = torch.float16,
+        block_n: int = 128,
+        num_warps: int = 1,
+    ) -> None:
+        super().__init__()
+        if not TRITON_AVAILABLE:
+            raise RuntimeError("Triton is not available.")
+        if nbits not in SUPPORTED_NBITS:
+            raise ValueError(f"Unsupported nbits: {nbits}")
+        out_features, in_features = W_int.shape
+        if in_features % group_size != 0:
+            raise ValueError("in_features must be divisible by group_size.")
+        if group_size % 2 != 0:
+            raise ValueError("group_size must be even.")
+        self.out_features = out_features
+        self.in_features = in_features
+        self.nbits = int(nbits)
+        self.group_size = int(group_size)
+        self.out_dtype = out_dtype
+        self.block_n = int(block_n)
+        self.num_warps = int(num_warps)
+        self.block_k = self.group_size // 2
+        q_kn = W_int.t().contiguous().to(torch.uint8)
+        if nbits == 3:
+            self.register_buffer("W_q", _pack_kmajor(q_kn >> 1, 2))
+            self.register_buffer("W_lo", _pack_kmajor(q_kn & 1, 1))
+            self.eps = 16
+        else:
+            self.register_buffer("W_q", _pack_kmajor(q_kn, nbits))
+            self.register_buffer("W_lo", torch.zeros(1, dtype=torch.int32, device=q_kn.device))
+            self.eps = 32 // nbits
+        del q_kn
+        self.register_buffer("scale", scale.t().contiguous().to(out_dtype))
+        self.register_buffer("zero", zero.t().contiguous().to(out_dtype))
+        if bias is not None:
+            self.register_buffer("bias", bias.detach().clone().to(out_dtype))
+        else:
+            self.bias = None
+        # The GEMV accumulates with atomics, so it starts from the bias.
+        acc_init = torch.zeros(out_features, dtype=torch.float32, device=self.W_q.device)
+        if bias is not None:
+            acc_init.copy_(self.bias.float())
+        self.register_buffer("_acc_init", acc_init)
+        self.register_buffer("_acc", acc_init.clone())
+        self._grid = (
+            (out_features + self.block_n - 1) // self.block_n,
+            in_features // self.group_size,
+        )
+    def dequantize_weight(self, dtype: torch.dtype) -> torch.Tensor:
+        """Returns W^T with shape (in_features, out_features)."""
+        if self.nbits == 3:
+            q = (_unpack_kmajor(self.W_q, 2, self.in_features).to(torch.int32) << 1) | (
+                _unpack_kmajor(self.W_lo, 1, self.in_features).to(torch.int32)
+            )
+        else:
+            q = _unpack_kmajor(self.W_q, self.nbits, self.in_features)
+        s = self.scale.repeat_interleave(self.group_size, dim=0).to(dtype)
+        z = self.zero.repeat_interleave(self.group_size, dim=0).to(dtype)
+        return (q.to(dtype) - z) * s
+    # (BLOCK_M, BLOCK_N, SPLIT_K, num_warps, num_stages), largest tile first;
+    # the first entry that fits in shared memory is cached per shape.
+    _SMALL_M_CONFIGS = ((16, 64, 8, 4, 2), (16, 64, 4, 4, 1))
+    _LARGE_M_CONFIGS = ((128, 128, 1, 8, 4), (128, 128, 1, 8, 3),
+                        (128, 64, 1, 4, 3), (64, 64, 1, 4, 2))
+    def _gemm(self, x2d: torch.Tensor) -> torch.Tensor:
+        M = x2d.shape[0]
+        N, K, gs = self.out_features, self.in_features, self.group_size
+        block_k = min(gs, 32)
+        configs = self._SMALL_M_CONFIGS if M <= 32 else self._LARGE_M_CONFIGS
+        cache_key = (M <= 32, N, K, gs, self.nbits)
+        if cache_key in _GEMM_CONFIG_CACHE:
+            configs = (_GEMM_CONFIG_CACHE[cache_key],)
+        tl_dtype = tl.float16 if self.out_dtype == torch.float16 else tl.bfloat16
+        last_err = None
+        for cfg in configs:
+            block_m, block_n, split_k, warps, stages = cfg
+            split_k = min(split_k, max(1, K // block_k))
+            alloc = torch.empty if split_k == 1 else torch.zeros
+            y = alloc(M, N, dtype=self.out_dtype, device=x2d.device)
+            grid = (triton.cdiv(M, block_m), triton.cdiv(N, block_n), split_k)
+            try:
+                _dashq_gemm_kernel[grid](
+                    x2d, self.W_q, self.W_lo, self.scale, self.zero, y,
+                    M, N, K,
+                    self.nbits, self.eps, gs,
+                    block_m, block_n, block_k, split_k, tl_dtype,
+                    num_warps=warps, num_stages=stages,
+                )
+            except triton.runtime.errors.OutOfResources as exc:
+                last_err = exc
+                continue
+            _GEMM_CONFIG_CACHE[cache_key] = cfg
+            return y
+        raise last_err
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        shape = x.shape
+        tokens = x.numel() // shape[-1]
+        if tokens == 1 and x.is_cuda:
+            self._acc.copy_(self._acc_init)
+            _dashq_gemv_kernel[self._grid](
+                x.reshape(-1), self.W_q, self.W_lo, self.scale, self.zero, self._acc,
+                self.out_features, self.in_features,
+                self.nbits, self.eps, self.group_size,
+                self.block_n, self.block_k,
+                num_warps=self.num_warps,
+            )
+            return self._acc.to(x.dtype).reshape(*shape[:-1], self.out_features)
+        x2d = x.reshape(tokens, -1)
+        if x.is_cuda and TRITON_AVAILABLE:
+            out = self._gemm(x2d)
+        else:
+            out = x2d @ self.dequantize_weight(x.dtype)
+        if self.bias is not None:
+            out = out + self.bias.to(out.dtype)
+        return out.to(x.dtype).reshape(*shape[:-1], self.out_features)
+    def extra_repr(self) -> str:
+        return (
+            f"in_features={self.in_features}, out_features={self.out_features}, "
+            f"nbits={self.nbits}, group_size={self.group_size}"
+        )

generation_config.json ADDED Viewed

	@@ -0,0 +1,11 @@

+{
+  "_from_model_config": true,
+  "bos_token_id": 100257,
+  "eos_token_id": [
+    100257,
+    100265
+  ],
+  "pad_token_id": 100349,
+  "transformers_version": "5.15.0",
+  "use_cache": false
+}

model-00001-of-00002.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b5a59a6a7cb6e9926d08acc87ad76d17a3bbce7c28330944ef0386c4ba7ca8b0
+size 4998820200

model-00002-of-00002.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:04b6470b75c9986bf8e38072c8193c71c22b0f809acc42b7ac5ee3d5e8b8e1d0
+size 3361846584

model.safetensors.index.json ADDED Viewed

	@@ -0,0 +1,571 @@

+{
+  "metadata": {
+    "total_parameters": 1028019200,
+    "total_size": 8360601600
+  },
+  "weight_map": {
+    "lm_head.weight": "model-00002-of-00002.safetensors",
+    "model.embed_tokens.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.0.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.0.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.0.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.1.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.1.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.1.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.1.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.10.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.10.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.10.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.10.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.11.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.11.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.11.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.11.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.12.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.12.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.12.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.12.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.13.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.13.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.13.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.13.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.14.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.14.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.14.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.14.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.15.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.15.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.15.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.15.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.16.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.16.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.16.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.16.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.17.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.17.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.17.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.17.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.18.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.18.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.18.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.18.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.19.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.19.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.19.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.19.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.2.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.2.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.2.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.2.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.20.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.20.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.20.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.20.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.21.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.21.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.21.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.21.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.22.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.22.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.22.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.22.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.23.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.23.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.23.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.23.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.24.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.24.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.24.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.24.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.25.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.25.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.25.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.25.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.26.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.26.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.26.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.26.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.27.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.27.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.27.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.27.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.28.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.28.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.28.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.28.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.29.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.29.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.29.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.29.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.3.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.3.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.3.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.3.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.30.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.30.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.30.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.30.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.31.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.31.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.31.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.31.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.32.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.32.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.32.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.32.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.32.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.32.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.32.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.33.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.33.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.33.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.33.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.34.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.34.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.34.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.34.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.35.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.35.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.35.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.35.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.36.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.36.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.36.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.36.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.36.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.36.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.36.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.36.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.37.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.37.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.37.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.37.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.37.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.37.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.37.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.37.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.38.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.38.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.38.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.38.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.38.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.38.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.38.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.38.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.39.input_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.mlp.down_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.39.mlp.down_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.39.mlp.down_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.39.mlp.gate_up_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.39.mlp.gate_up_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.39.mlp.gate_up_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.39.post_attention_layernorm.weight": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.o_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.o_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.o_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.qkv_proj.W_q_packed": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.qkv_proj.scale": "model-00002-of-00002.safetensors",
+    "model.layers.39.self_attn.qkv_proj.zero": "model-00002-of-00002.safetensors",
+    "model.layers.4.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.4.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.4.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.4.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.5.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.5.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.5.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.5.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.6.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.6.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.6.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.6.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.7.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.7.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.7.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.7.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.8.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.8.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.8.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.8.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.9.input_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.down_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_up_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_up_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.9.mlp.gate_up_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.9.post_attention_layernorm.weight": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.o_proj.zero": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.qkv_proj.W_q_packed": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.qkv_proj.scale": "model-00001-of-00002.safetensors",
+    "model.layers.9.self_attn.qkv_proj.zero": "model-00001-of-00002.safetensors",
+    "model.norm.weight": "model-00002-of-00002.safetensors"
+  }
+}

modeling_dashq.py ADDED Viewed

	@@ -0,0 +1,227 @@

+"""Inference code for this DASH-Q checkpoint.
+Generated by export_hf_repo.py -- do not edit by hand.
+Weights are group-wise asymmetric integers packed into int32 words; the layout of
+each quantized layer is described by `dashq_config.json`. At load time the layers
+are converted to the format used by the Triton kernels in `dashq_kernel.py`, with
+a PyTorch dequantize-and-matmul fallback when Triton is unavailable.
+"""
+from __future__ import annotations
+import json
+import os
+from typing import Any, Dict, Optional
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import AutoConfig, Phi3ForCausalLM
+try:
+    from .dashq_kernel import TRITON_AVAILABLE, SUPPORTED_NBITS, TritonQuantLinear
+except ImportError:  # loaded as a flat module by trust_remote_code
+    from dashq_kernel import TRITON_AVAILABLE, SUPPORTED_NBITS, TritonQuantLinear
+DASHQ_CONFIG_FILE = "dashq_config.json"
+def _unpack_int_values(packed: torch.Tensor, nbits: int, numel: int) -> torch.Tensor:
+    values_per_word = max(1, 32 // nbits)
+    mask = (1 << nbits) - 1
+    shifts = torch.arange(values_per_word, device=packed.device, dtype=torch.int32) * nbits
+    out = (packed.view(-1, 1) >> shifts.view(1, -1)) & mask
+    return out.reshape(-1)[:numel]
+class DashQPackedLinear(nn.Module):
+    """Checkpoint buffers for one quantized layer."""
+    def __init__(self, in_features: int, out_features: int, nbits: int, group_size: int,
+                 bias: bool, dtype: torch.dtype, quant_in_features: Optional[int] = None) -> None:
+        super().__init__()
+        self.in_features = int(in_features)
+        self.quant_in_features = int(quant_in_features or in_features)
+        self.out_features = int(out_features)
+        self.nbits = int(nbits)
+        self.group_size = int(group_size)
+        self.linear_dtype = dtype
+        self.numel = self.out_features * self.quant_in_features
+        self.num_groups = self.numel // self.group_size
+        values_per_word = max(1, 32 // self.nbits)
+        n_words = (self.numel + values_per_word - 1) // values_per_word
+        self.register_buffer("W_q_packed", torch.zeros(n_words, dtype=torch.int32))
+        self.register_buffer("scale", torch.zeros(self.num_groups, 1, dtype=torch.float16))
+        self.register_buffer("zero", torch.zeros(self.num_groups, 1, dtype=torch.float16))
+        if bias:
+            self.bias = nn.Parameter(torch.zeros(self.out_features, dtype=dtype), requires_grad=False)
+        else:
+            self.register_parameter("bias", None)
+        self.kernel: Optional[nn.Module] = None
+    def dequantize_weight(self, dtype: torch.dtype) -> torch.Tensor:
+        W_int = _unpack_int_values(self.W_q_packed, self.nbits, self.numel)
+        W_int = W_int.view(self.num_groups, self.group_size).to(dtype)
+        W = (W_int - self.zero.to(dtype)) * self.scale.to(dtype)
+        return W.view(self.out_features, self.quant_in_features)
+    @torch.no_grad()
+    def build_kernel(self) -> bool:
+        if self.kernel is not None:
+            return True
+        if not TRITON_AVAILABLE or self.nbits not in SUPPORTED_NBITS:
+            return False
+        if self.quant_in_features % self.group_size or self.group_size % 2:
+            return False
+        if self.W_q_packed is None or not self.W_q_packed.is_cuda:
+            return False
+        W_int = _unpack_int_values(self.W_q_packed, self.nbits, self.numel)
+        W_int = W_int.view(self.out_features, self.quant_in_features)
+        ng = self.quant_in_features // self.group_size
+        self.kernel = TritonQuantLinear(
+            W_int,
+            self.scale.view(self.out_features, ng),
+            self.zero.view(self.out_features, ng),
+            self.nbits,
+            self.group_size,
+            bias=self.bias.data if self.bias is not None else None,
+            out_dtype=self.linear_dtype,
+        )
+        del W_int
+        self._buffers["W_q_packed"] = None
+        self._buffers["scale"] = None
+        self._buffers["zero"] = None
+        return True
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        if self.kernel is not None:
+            return self.kernel(x)
+        weight = self.dequantize_weight(x.dtype)
+        bias = self.bias.to(x.dtype) if self.bias is not None else None
+        return F.linear(x, weight, bias)
+    def extra_repr(self) -> str:
+        return (f"in_features={self.in_features}, out_features={self.out_features}, "
+                f"nbits={self.nbits}, group_size={self.group_size}")
+def _set_module(root: nn.Module, name: str, new_module: nn.Module) -> None:
+    parts = name.split(".")
+    parent = root
+    for part in parts[:-1]:
+        parent = getattr(parent, part)
+    setattr(parent, parts[-1], new_module)
+def _get_module(root: nn.Module, name: str) -> Optional[nn.Module]:
+    obj = root
+    for part in name.split("."):
+        if not hasattr(obj, part):
+            return None
+        obj = getattr(obj, part)
+    return obj
+_DTYPES = {"float16": torch.float16, "bfloat16": torch.bfloat16, "float32": torch.float32}
+def _load_dashq_spec(model_id_or_path, **kwargs) -> Dict[str, Any]:
+    """Read dashq_config.json from a local dir or the Hub."""
+    path = None
+    if model_id_or_path is not None:
+        local = os.path.join(str(model_id_or_path), DASHQ_CONFIG_FILE)
+        if os.path.isfile(local):
+            path = local
+    if path is None and model_id_or_path is not None:
+        try:
+            from huggingface_hub import hf_hub_download
+            path = hf_hub_download(
+                repo_id=str(model_id_or_path),
+                filename=DASHQ_CONFIG_FILE,
+                revision=kwargs.get("revision"),
+                token=kwargs.get("token"),
+                cache_dir=kwargs.get("cache_dir"),
+            )
+        except Exception:
+            return {}
+    if path is None:
+        return {}
+    with open(path, encoding="utf-8") as f:
+        return json.load(f)
+def _swap_quantized_modules(model: nn.Module, modules: Dict[str, Any]) -> int:
+    count = 0
+    for name, meta in modules.items():
+        target = _get_module(model, name)
+        if target is None or isinstance(target, DashQPackedLinear):
+            continue
+        module = DashQPackedLinear(
+            in_features=meta["in_features"],
+            out_features=meta["out_features"],
+            nbits=meta["nbits"],
+            group_size=meta["group_size"],
+            bias=getattr(target, "bias", None) is not None,
+            dtype=_DTYPES.get(meta.get("linear_dtype", "float16"), torch.float16),
+            quant_in_features=meta.get("quant_in_features"),
+        )
+        _set_module(model, name, module)
+        count += 1
+    return count
+class DashQPhi3ForCausalLM(Phi3ForCausalLM):
+    """Phi3ForCausalLM whose linear layers hold DASH-Q packed quantized weights."""
+    def __init__(self, config):
+        super().__init__(config)
+        modules = (getattr(config, "dashq_modules", None) or {})
+        if modules:
+            _swap_quantized_modules(self, modules)
+    @classmethod
+    def from_pretrained(cls, pretrained_model_name_or_path=None, *args, **kwargs):
+        config = kwargs.pop("config", None)
+        if config is None and pretrained_model_name_or_path is not None:
+            config = AutoConfig.from_pretrained(
+                pretrained_model_name_or_path,
+                trust_remote_code=True,
+                revision=kwargs.get("revision"),
+                token=kwargs.get("token"),
+                cache_dir=kwargs.get("cache_dir"),
+            )
+        if config is not None and not getattr(config, "dashq_modules", None):
+            spec = _load_dashq_spec(pretrained_model_name_or_path, **kwargs)
+            config.dashq_modules = spec.get("quantized_modules", {})
+        model = super().from_pretrained(pretrained_model_name_or_path, *args, config=config, **kwargs)
+        model.build_dashq_kernels()
+        return model
+    def build_dashq_kernels(self, verbose: bool = True) -> "DashQPhi3ForCausalLM":
+        """Move the packed buffers to the Triton kernel layout (no-op off CUDA)."""
+        total = built = 0
+        for module in self.modules():
+            if isinstance(module, DashQPackedLinear):
+                total += 1
+                built += int(module.build_kernel())
+        if verbose and total:
+            if built:
+                print(f">> DASH-Q: {built}/{total} linear layers using the Triton decode kernel.")
+            else:
+                print(f">> DASH-Q: {total} quantized layers using the PyTorch fallback path.")
+        return self
+    def save_pretrained(self, *args, **kwargs):
+        released = any(
+            isinstance(m, DashQPackedLinear) and m.kernel is not None for m in self.modules()
+        )
+        if released:
+            raise RuntimeError(
+                "This model has been converted to the DASH-Q Triton kernel, so the packed "
+                "buffers are no longer materialized and saving would produce an incomplete "
+                "checkpoint. Reload with build_dashq_kernels() skipped if you need to re-save."
+            )
+        return super().save_pretrained(*args, **kwargs)

tokenizer.json ADDED Viewed

The diff for this file is too large to render. See raw diff

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,12 @@

+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 16384,
+  "pad_token": "<|dummy_85|>",
+  "tokenizer_class": "TokenizersBackend"
+}