Instructions to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bjk110/Qwen3.5-122B-A10B-abliterated-FP8")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForCausalLM

processor = AutoProcessor.from_pretrained("bjk110/Qwen3.5-122B-A10B-abliterated-FP8")
model = AutoModelForCausalLM.from_pretrained("bjk110/Qwen3.5-122B-A10B-abliterated-FP8", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bjk110/Qwen3.5-122B-A10B-abliterated-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bjk110/Qwen3.5-122B-A10B-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bjk110/Qwen3.5-122B-A10B-abliterated-FP8

SGLang

How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bjk110/Qwen3.5-122B-A10B-abliterated-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bjk110/Qwen3.5-122B-A10B-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bjk110/Qwen3.5-122B-A10B-abliterated-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bjk110/Qwen3.5-122B-A10B-abliterated-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use bjk110/Qwen3.5-122B-A10B-abliterated-FP8 with Docker Model Runner:
```
docker model run hf.co/bjk110/Qwen3.5-122B-A10B-abliterated-FP8
```

Ban commited on Mar 28

Commit

ee139d3

verified ·

1 Parent(s): 91967e4

Upload FP8 block-wise quantized model (128x128, weight_scale_inv)

Browse files

Files changed (49) hide show

.gitattributes +1 -0
README.md +125 -0
chat_template.jinja +154 -0
config.json +367 -0
generation_config.json +13 -0
model-00001-of-00039.safetensors +3 -0
model-00002-of-00039.safetensors +3 -0
model-00003-of-00039.safetensors +3 -0
model-00004-of-00039.safetensors +3 -0
model-00005-of-00039.safetensors +3 -0
model-00006-of-00039.safetensors +3 -0
model-00007-of-00039.safetensors +3 -0
model-00008-of-00039.safetensors +3 -0
model-00009-of-00039.safetensors +3 -0
model-00010-of-00039.safetensors +3 -0
model-00011-of-00039.safetensors +3 -0
model-00012-of-00039.safetensors +3 -0
model-00013-of-00039.safetensors +3 -0
model-00014-of-00039.safetensors +3 -0
model-00015-of-00039.safetensors +3 -0
model-00016-of-00039.safetensors +3 -0
model-00017-of-00039.safetensors +3 -0
model-00018-of-00039.safetensors +3 -0
model-00019-of-00039.safetensors +3 -0
model-00020-of-00039.safetensors +3 -0
model-00021-of-00039.safetensors +3 -0
model-00022-of-00039.safetensors +3 -0
model-00023-of-00039.safetensors +3 -0
model-00024-of-00039.safetensors +3 -0
model-00025-of-00039.safetensors +3 -0
model-00026-of-00039.safetensors +3 -0
model-00027-of-00039.safetensors +3 -0
model-00028-of-00039.safetensors +3 -0
model-00029-of-00039.safetensors +3 -0
model-00030-of-00039.safetensors +3 -0
model-00031-of-00039.safetensors +3 -0
model-00032-of-00039.safetensors +3 -0
model-00033-of-00039.safetensors +3 -0
model-00034-of-00039.safetensors +3 -0
model-00035-of-00039.safetensors +3 -0
model-00036-of-00039.safetensors +3 -0
model-00037-of-00039.safetensors +3 -0
model-00038-of-00039.safetensors +3 -0
model-00039-of-00039.safetensors +3 -0
model.safetensors.index.json +0 -0
preprocessor_config.json +21 -0
tokenizer.json +3 -0
tokenizer_config.json +31 -0
vllm_patches/patch_qwen35_moe_text.py +290 -0

.gitattributes CHANGED Viewed

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text

 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md ADDED Viewed

	@@ -0,0 +1,125 @@

+---
+base_model: wangzhang/Qwen3.5-122B-A10B-abliterated
+pipeline_tag: text-generation
+tags:
+- qwen3.5
+- moe
+- fp8
+- quantized
+- abliterated
+- uncensored
+- block-wise-fp8
+license: apache-2.0
+---
+# Qwen3.5-122B-A10B-abliterated-FP8
+[wangzhang/Qwen3.5-122B-A10B-abliterated](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterated)의
+**FP8 E4M3 block-wise 양자화** 버전입니다.
+[Qwen/Qwen3.5-122B-A10B-FP8](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8)과 동일한 저장 포맷을 사용합니다.
+## 모델 정보
+| 항목 | 값 |
+|------|-----|
+| 원본 모델 | [wangzhang/Qwen3.5-122B-A10B-abliterated](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterated) |
+| 양자화 | FP8 E4M3, block-wise 128×128, `weight_scale_inv` (bfloat16) |
+| 체크포인트 크기 | **115 GB** (원본 228 GB 대비 50% 감소) |
+| 활성 파라미터 | ~10B (MoE, 256 experts 중 10개 활성) |
+| 전체 파라미터 | 122B |
+| 최대 컨텍스트 | 32,768 tokens |
+| 검열 해제 | ✅ (abliterated, refusal rate 0.5%) |
+| Activation | Dynamic (체크포인트에 미포함) |
+| BF16 유지 대상 | lm_head, embed_tokens, norms, conv1d, router gate, in_proj_a/b |
+## 사용법
+### ⚠️ vLLM 패치 필요
+이 모델은 `qwen3_5_moe_text` (text-only MoE) 아키텍처를 사용합니다.
+vLLM v0.17~v0.18에서는 이 아키텍처가 정식 지원되지 않으므로,
+함께 제공되는 `vllm_patches/patch_qwen35_moe_text.py` 패치를 적용해야 합니다.
+### 서빙 (vLLM)
+```bash
+# 1. 패치 적용 (vLLM 프로세스 시작 전에 실행)
+python vllm_patches/patch_qwen35_moe_text.py
+# 2. 서빙
+vllm serve /path/to/Qwen3.5-122B-A10B-abliterated-FP8 \
+    --tensor-parallel-size 2 \
+    --trust-remote-code \
+    --enable-chunked-prefill \
+    --max-model-len 32768 \
+    --reasoning-parser qwen3
+```
+### Docker에서 패치 자동 적용
+entrypoint.sh 시작 부분에 추가:
+```bash
+if [ -f /patches/patch_qwen35_moe_text.py ]; then
+    python3 /patches/patch_qwen35_moe_text.py || true
+fi
+```
+docker-compose.yml에 볼륨 마운트:
+```yaml
+volumes:
+  - ./vllm_patches:/patches:ro
+```
+## 패치가 해결하는 문제
+| 문제 | 원인 | 해결 |
+|------|------|------|
+| `Qwen3_5MoeForCausalLM` 미인식 | vLLM registry 미등록 | TextOnlyShim 등록 |
+| Hybrid cache page-size 에러 | text-only CausalLM 경로 버그 | multimodal wrapper 경로 재사용 |
+| Vision encoder 초기화 실패 | wrapper가 vision 강제 초기화 | vision encoder 스킵 |
+| TP2 block_k 에러 | vision hidden_size=1152 참조 | dummy vision config 주입 |
+vLLM이 `qwen3_5_moe_text`를 정식 지원하면 패치가 불필요해집니다.
+## 벤치마크 (한국어 QA 12문항, DGX Spark TP=2)
+| 지표 | 공식 Qwen FP8 | **본 모델** | BF16+Runtime FP8 |
+|------|:---:|:---:|:---:|
+| 완답 | 12/12 | **12/12** | 11/12 |
+| 속도 | 20.3 tok/s | **29.6 tok/s** | 29.3 tok/s |
+| 크기 | 119 GB | **115 GB** | 228 GB |
+| Swap 필요 | No | **No** | Yes (200GB) |
+| 검열 해제 | ❌ | **✅** | ✅ |
+Runtime FP8(BF16 원본 + `--quantization fp8`)과 동일한 출력 품질이면서,
+체크포인트 크기 50% 감소 + swap 불필요 + 빠른 로딩.
+## 하드웨어 요구사항
+| 구성 | GPU 메모리 | 비고 |
+|------|-----------|------|
+| TP=1 | ~115 GB | H100 80GB 불가, GB200 등 |
+| **TP=2** | **~58 GB/GPU** | DGX Spark, H100×2, A100 80GB×2 |
+| TP=4 | ~29 GB/GPU | A100 40GB×4 |
+## 양자화 방법
+calibration 없이 BF16 weight를 128×128 block 단위로 직접 FP8 E4M3 변환합니다.
+[fp8-quantizer](https://github.com/JungkwanBan/fp8-quantizer)로 생성.
+```bash
+python convert_bf16_to_fp8.py \
+    --model wangzhang/Qwen3.5-122B-A10B-abliterated \
+    --output ./Qwen3.5-122B-A10B-abliterated-FP8
+```
+## 원본 모델 정보
+[wangzhang/Qwen3.5-122B-A10B-abliterated](https://huggingface.co/wangzhang/Qwen3.5-122B-A10B-abliterated):
+[Prometheus](https://github.com/wuwangzhang1216/prometheus)를 사용한 검열 해제 버전.
+Refusal rate 0.5% (200개 테스트 중 1개), KL divergence 0.0115.
+## 라이선스
+원본 모델의 라이선스를 따릅니다.

chat_template.jinja ADDED Viewed

	@@ -0,0 +1,154 @@

+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

config.json ADDED Viewed

	@@ -0,0 +1,367 @@

+{
+  "architectures": [
+    "Qwen3_5MoeForCausalLM"
+  ],
+  "model_type": "qwen3_5_moe",
+  "text_config": {
+    "architectures": [
+      "Qwen3_5MoeForCausalLM"
+    ],
+    "attention_bias": false,
+    "attention_dropout": 0.0,
+    "attn_output_gate": true,
+    "bos_token_id": null,
+    "dtype": "bfloat16",
+    "eos_token_id": 248044,
+    "full_attention_interval": 4,
+    "head_dim": 256,
+    "hidden_act": "silu",
+    "hidden_size": 3072,
+    "initializer_range": 0.02,
+    "layer_types": [
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention",
+      "linear_attention",
+      "linear_attention",
+      "linear_attention",
+      "full_attention"
+    ],
+    "linear_conv_kernel_dim": 4,
+    "linear_key_head_dim": 128,
+    "linear_num_key_heads": 16,
+    "linear_num_value_heads": 64,
+    "linear_value_head_dim": 128,
+    "mamba_ssm_dtype": "float32",
+    "max_position_embeddings": 262144,
+    "mlp_only_layers": [],
+    "model_type": "qwen3_5_moe_text",
+    "moe_intermediate_size": 1024,
+    "mtp_num_hidden_layers": 1,
+    "mtp_use_dedicated_embeddings": false,
+    "num_attention_heads": 32,
+    "num_experts": 256,
+    "num_experts_per_tok": 8,
+    "num_hidden_layers": 48,
+    "num_key_value_heads": 2,
+    "output_router_logits": false,
+    "pad_token_id": null,
+    "partial_rotary_factor": 0.25,
+    "rms_norm_eps": 1e-06,
+    "rope_parameters": {
+      "mrope_interleaved": true,
+      "mrope_section": [
+        11,
+        11,
+        10
+      ],
+      "partial_rotary_factor": 0.25,
+      "rope_theta": 10000000,
+      "rope_type": "default"
+    },
+    "router_aux_loss_coef": 0.001,
+    "shared_expert_intermediate_size": 1024,
+    "tie_word_embeddings": false,
+    "transformers_version": "5.3.0.dev0",
+    "use_cache": true,
+    "vocab_size": 248320
+  },
+  "quantization_config": {
+    "quant_method": "fp8",
+    "activation_scheme": "dynamic",
+    "act_per_tensor": false,
+    "modules_to_not_convert": [
+      "lm_head",
+      "model.language_model.embed_tokens",
+      "model.language_model.layers.0.linear_attn.conv1d",
+      "model.language_model.layers.0.linear_attn.in_proj_a",
+      "model.language_model.layers.0.linear_attn.in_proj_b",
+      "model.language_model.layers.0.mlp.gate",
+      "model.language_model.layers.0.mlp.shared_expert_gate",
+      "model.language_model.layers.1.linear_attn.conv1d",
+      "model.language_model.layers.1.linear_attn.in_proj_a",
+      "model.language_model.layers.1.linear_attn.in_proj_b",
+      "model.language_model.layers.1.mlp.gate",
+      "model.language_model.layers.1.mlp.shared_expert_gate",
+      "model.language_model.layers.10.linear_attn.conv1d",
+      "model.language_model.layers.10.linear_attn.in_proj_a",
+      "model.language_model.layers.10.linear_attn.in_proj_b",
+      "model.language_model.layers.10.mlp.gate",
+      "model.language_model.layers.10.mlp.shared_expert_gate",
+      "model.language_model.layers.11.linear_attn.conv1d",
+      "model.language_model.layers.11.linear_attn.in_proj_a",
+      "model.language_model.layers.11.linear_attn.in_proj_b",
+      "model.language_model.layers.11.mlp.gate",
+      "model.language_model.layers.11.mlp.shared_expert_gate",
+      "model.language_model.layers.12.linear_attn.conv1d",
+      "model.language_model.layers.12.linear_attn.in_proj_a",
+      "model.language_model.layers.12.linear_attn.in_proj_b",
+      "model.language_model.layers.12.mlp.gate",
+      "model.language_model.layers.12.mlp.shared_expert_gate",
+      "model.language_model.layers.13.linear_attn.conv1d",
+      "model.language_model.layers.13.linear_attn.in_proj_a",
+      "model.language_model.layers.13.linear_attn.in_proj_b",
+      "model.language_model.layers.13.mlp.gate",
+      "model.language_model.layers.13.mlp.shared_expert_gate",
+      "model.language_model.layers.14.linear_attn.conv1d",
+      "model.language_model.layers.14.linear_attn.in_proj_a",
+      "model.language_model.layers.14.linear_attn.in_proj_b",
+      "model.language_model.layers.14.mlp.gate",
+      "model.language_model.layers.14.mlp.shared_expert_gate",
+      "model.language_model.layers.15.linear_attn.conv1d",
+      "model.language_model.layers.15.linear_attn.in_proj_a",
+      "model.language_model.layers.15.linear_attn.in_proj_b",
+      "model.language_model.layers.15.mlp.gate",
+      "model.language_model.layers.15.mlp.shared_expert_gate",
+      "model.language_model.layers.16.linear_attn.conv1d",
+      "model.language_model.layers.16.linear_attn.in_proj_a",
+      "model.language_model.layers.16.linear_attn.in_proj_b",
+      "model.language_model.layers.16.mlp.gate",
+      "model.language_model.layers.16.mlp.shared_expert_gate",
+      "model.language_model.layers.17.linear_attn.conv1d",
+      "model.language_model.layers.17.linear_attn.in_proj_a",
+      "model.language_model.layers.17.linear_attn.in_proj_b",
+      "model.language_model.layers.17.mlp.gate",
+      "model.language_model.layers.17.mlp.shared_expert_gate",
+      "model.language_model.layers.18.linear_attn.conv1d",
+      "model.language_model.layers.18.linear_attn.in_proj_a",
+      "model.language_model.layers.18.linear_attn.in_proj_b",
+      "model.language_model.layers.18.mlp.gate",
+      "model.language_model.layers.18.mlp.shared_expert_gate",
+      "model.language_model.layers.19.linear_attn.conv1d",
+      "model.language_model.layers.19.linear_attn.in_proj_a",
+      "model.language_model.layers.19.linear_attn.in_proj_b",
+      "model.language_model.layers.19.mlp.gate",
+      "model.language_model.layers.19.mlp.shared_expert_gate",
+      "model.language_model.layers.2.linear_attn.conv1d",
+      "model.language_model.layers.2.linear_attn.in_proj_a",
+      "model.language_model.layers.2.linear_attn.in_proj_b",
+      "model.language_model.layers.2.mlp.gate",
+      "model.language_model.layers.2.mlp.shared_expert_gate",
+      "model.language_model.layers.20.linear_attn.conv1d",
+      "model.language_model.layers.20.linear_attn.in_proj_a",
+      "model.language_model.layers.20.linear_attn.in_proj_b",
+      "model.language_model.layers.20.mlp.gate",
+      "model.language_model.layers.20.mlp.shared_expert_gate",
+      "model.language_model.layers.21.linear_attn.conv1d",
+      "model.language_model.layers.21.linear_attn.in_proj_a",
+      "model.language_model.layers.21.linear_attn.in_proj_b",
+      "model.language_model.layers.21.mlp.gate",
+      "model.language_model.layers.21.mlp.shared_expert_gate",
+      "model.language_model.layers.22.linear_attn.conv1d",
+      "model.language_model.layers.22.linear_attn.in_proj_a",
+      "model.language_model.layers.22.linear_attn.in_proj_b",
+      "model.language_model.layers.22.mlp.gate",
+      "model.language_model.layers.22.mlp.shared_expert_gate",
+      "model.language_model.layers.23.linear_attn.conv1d",
+      "model.language_model.layers.23.linear_attn.in_proj_a",
+      "model.language_model.layers.23.linear_attn.in_proj_b",
+      "model.language_model.layers.23.mlp.gate",
+      "model.language_model.layers.23.mlp.shared_expert_gate",
+      "model.language_model.layers.24.linear_attn.conv1d",
+      "model.language_model.layers.24.linear_attn.in_proj_a",
+      "model.language_model.layers.24.linear_attn.in_proj_b",
+      "model.language_model.layers.24.mlp.gate",
+      "model.language_model.layers.24.mlp.shared_expert_gate",
+      "model.language_model.layers.25.linear_attn.conv1d",
+      "model.language_model.layers.25.linear_attn.in_proj_a",
+      "model.language_model.layers.25.linear_attn.in_proj_b",
+      "model.language_model.layers.25.mlp.gate",
+      "model.language_model.layers.25.mlp.shared_expert_gate",
+      "model.language_model.layers.26.linear_attn.conv1d",
+      "model.language_model.layers.26.linear_attn.in_proj_a",
+      "model.language_model.layers.26.linear_attn.in_proj_b",
+      "model.language_model.layers.26.mlp.gate",
+      "model.language_model.layers.26.mlp.shared_expert_gate",
+      "model.language_model.layers.27.linear_attn.conv1d",
+      "model.language_model.layers.27.linear_attn.in_proj_a",
+      "model.language_model.layers.27.linear_attn.in_proj_b",
+      "model.language_model.layers.27.mlp.gate",
+      "model.language_model.layers.27.mlp.shared_expert_gate",
+      "model.language_model.layers.28.linear_attn.conv1d",
+      "model.language_model.layers.28.linear_attn.in_proj_a",
+      "model.language_model.layers.28.linear_attn.in_proj_b",
+      "model.language_model.layers.28.mlp.gate",
+      "model.language_model.layers.28.mlp.shared_expert_gate",
+      "model.language_model.layers.29.linear_attn.conv1d",
+      "model.language_model.layers.29.linear_attn.in_proj_a",
+      "model.language_model.layers.29.linear_attn.in_proj_b",
+      "model.language_model.layers.29.mlp.gate",
+      "model.language_model.layers.29.mlp.shared_expert_gate",
+      "model.language_model.layers.3.linear_attn.conv1d",
+      "model.language_model.layers.3.linear_attn.in_proj_a",
+      "model.language_model.layers.3.linear_attn.in_proj_b",
+      "model.language_model.layers.3.mlp.gate",
+      "model.language_model.layers.3.mlp.shared_expert_gate",
+      "model.language_model.layers.30.linear_attn.conv1d",
+      "model.language_model.layers.30.linear_attn.in_proj_a",
+      "model.language_model.layers.30.linear_attn.in_proj_b",
+      "model.language_model.layers.30.mlp.gate",
+      "model.language_model.layers.30.mlp.shared_expert_gate",
+      "model.language_model.layers.31.linear_attn.conv1d",
+      "model.language_model.layers.31.linear_attn.in_proj_a",
+      "model.language_model.layers.31.linear_attn.in_proj_b",
+      "model.language_model.layers.31.mlp.gate",
+      "model.language_model.layers.31.mlp.shared_expert_gate",
+      "model.language_model.layers.32.linear_attn.conv1d",
+      "model.language_model.layers.32.linear_attn.in_proj_a",
+      "model.language_model.layers.32.linear_attn.in_proj_b",
+      "model.language_model.layers.32.mlp.gate",
+      "model.language_model.layers.32.mlp.shared_expert_gate",
+      "model.language_model.layers.33.linear_attn.conv1d",
+      "model.language_model.layers.33.linear_attn.in_proj_a",
+      "model.language_model.layers.33.linear_attn.in_proj_b",
+      "model.language_model.layers.33.mlp.gate",
+      "model.language_model.layers.33.mlp.shared_expert_gate",
+      "model.language_model.layers.34.linear_attn.conv1d",
+      "model.language_model.layers.34.linear_attn.in_proj_a",
+      "model.language_model.layers.34.linear_attn.in_proj_b",
+      "model.language_model.layers.34.mlp.gate",
+      "model.language_model.layers.34.mlp.shared_expert_gate",
+      "model.language_model.layers.35.linear_attn.conv1d",
+      "model.language_model.layers.35.linear_attn.in_proj_a",
+      "model.language_model.layers.35.linear_attn.in_proj_b",
+      "model.language_model.layers.35.mlp.gate",
+      "model.language_model.layers.35.mlp.shared_expert_gate",
+      "model.language_model.layers.36.linear_attn.conv1d",
+      "model.language_model.layers.36.linear_attn.in_proj_a",
+      "model.language_model.layers.36.linear_attn.in_proj_b",
+      "model.language_model.layers.36.mlp.gate",
+      "model.language_model.layers.36.mlp.shared_expert_gate",
+      "model.language_model.layers.37.linear_attn.conv1d",
+      "model.language_model.layers.37.linear_attn.in_proj_a",
+      "model.language_model.layers.37.linear_attn.in_proj_b",
+      "model.language_model.layers.37.mlp.gate",
+      "model.language_model.layers.37.mlp.shared_expert_gate",
+      "model.language_model.layers.38.linear_attn.conv1d",
+      "model.language_model.layers.38.linear_attn.in_proj_a",
+      "model.language_model.layers.38.linear_attn.in_proj_b",
+      "model.language_model.layers.38.mlp.gate",
+      "model.language_model.layers.38.mlp.shared_expert_gate",
+      "model.language_model.layers.39.linear_attn.conv1d",
+      "model.language_model.layers.39.linear_attn.in_proj_a",
+      "model.language_model.layers.39.linear_attn.in_proj_b",
+      "model.language_model.layers.39.mlp.gate",
+      "model.language_model.layers.39.mlp.shared_expert_gate",
+      "model.language_model.layers.4.linear_attn.conv1d",
+      "model.language_model.layers.4.linear_attn.in_proj_a",
+      "model.language_model.layers.4.linear_attn.in_proj_b",
+      "model.language_model.layers.4.mlp.gate",
+      "model.language_model.layers.4.mlp.shared_expert_gate",
+      "model.language_model.layers.40.linear_attn.conv1d",
+      "model.language_model.layers.40.linear_attn.in_proj_a",
+      "model.language_model.layers.40.linear_attn.in_proj_b",
+      "model.language_model.layers.40.mlp.gate",
+      "model.language_model.layers.40.mlp.shared_expert_gate",
+      "model.language_model.layers.41.linear_attn.conv1d",
+      "model.language_model.layers.41.linear_attn.in_proj_a",
+      "model.language_model.layers.41.linear_attn.in_proj_b",
+      "model.language_model.layers.41.mlp.gate",
+      "model.language_model.layers.41.mlp.shared_expert_gate",
+      "model.language_model.layers.42.linear_attn.conv1d",
+      "model.language_model.layers.42.linear_attn.in_proj_a",
+      "model.language_model.layers.42.linear_attn.in_proj_b",
+      "model.language_model.layers.42.mlp.gate",
+      "model.language_model.layers.42.mlp.shared_expert_gate",
+      "model.language_model.layers.43.linear_attn.conv1d",
+      "model.language_model.layers.43.linear_attn.in_proj_a",
+      "model.language_model.layers.43.linear_attn.in_proj_b",
+      "model.language_model.layers.43.mlp.gate",
+      "model.language_model.layers.43.mlp.shared_expert_gate",
+      "model.language_model.layers.44.linear_attn.conv1d",
+      "model.language_model.layers.44.linear_attn.in_proj_a",
+      "model.language_model.layers.44.linear_attn.in_proj_b",
+      "model.language_model.layers.44.mlp.gate",
+      "model.language_model.layers.44.mlp.shared_expert_gate",
+      "model.language_model.layers.45.linear_attn.conv1d",
+      "model.language_model.layers.45.linear_attn.in_proj_a",
+      "model.language_model.layers.45.linear_attn.in_proj_b",
+      "model.language_model.layers.45.mlp.gate",
+      "model.language_model.layers.45.mlp.shared_expert_gate",
+      "model.language_model.layers.46.linear_attn.conv1d",
+      "model.language_model.layers.46.linear_attn.in_proj_a",
+      "model.language_model.layers.46.linear_attn.in_proj_b",
+      "model.language_model.layers.46.mlp.gate",
+      "model.language_model.layers.46.mlp.shared_expert_gate",
+      "model.language_model.layers.47.linear_attn.conv1d",
+      "model.language_model.layers.47.linear_attn.in_proj_a",
+      "model.language_model.layers.47.linear_attn.in_proj_b",
+      "model.language_model.layers.47.mlp.gate",
+      "model.language_model.layers.47.mlp.shared_expert_gate",
+      "model.language_model.layers.5.linear_attn.conv1d",
+      "model.language_model.layers.5.linear_attn.in_proj_a",
+      "model.language_model.layers.5.linear_attn.in_proj_b",
+      "model.language_model.layers.5.mlp.gate",
+      "model.language_model.layers.5.mlp.shared_expert_gate",
+      "model.language_model.layers.6.linear_attn.conv1d",
+      "model.language_model.layers.6.linear_attn.in_proj_a",
+      "model.language_model.layers.6.linear_attn.in_proj_b",
+      "model.language_model.layers.6.mlp.gate",
+      "model.language_model.layers.6.mlp.shared_expert_gate",
+      "model.language_model.layers.7.linear_attn.conv1d",
+      "model.language_model.layers.7.linear_attn.in_proj_a",
+      "model.language_model.layers.7.linear_attn.in_proj_b",
+      "model.language_model.layers.7.mlp.gate",
+      "model.language_model.layers.7.mlp.shared_expert_gate",
+      "model.language_model.layers.8.linear_attn.conv1d",
+      "model.language_model.layers.8.linear_attn.in_proj_a",
+      "model.language_model.layers.8.linear_attn.in_proj_b",
+      "model.language_model.layers.8.mlp.gate",
+      "model.language_model.layers.8.mlp.shared_expert_gate",
+      "model.language_model.layers.9.linear_attn.conv1d",
+      "model.language_model.layers.9.linear_attn.in_proj_a",
+      "model.language_model.layers.9.linear_attn.in_proj_b",
+      "model.language_model.layers.9.mlp.gate",
+      "model.language_model.layers.9.mlp.shared_expert_gate"
+    ],
+    "weight_block_size": [
+      128,
+      128
+    ],
+    "weight_per_tensor": false
+  },
+  "tie_word_embeddings": false,
+  "transformers_version": "5.3.0.dev0",
+  "vision_config": null
+}

generation_config.json ADDED Viewed

	@@ -0,0 +1,13 @@

+{
+  "bos_token_id": 248044,
+  "do_sample": true,
+  "eos_token_id": [
+    248046,
+    248044
+  ],
+  "pad_token_id": 248044,
+  "temperature": 0.6,
+  "top_k": 20,
+  "top_p": 0.95,
+  "transformers_version": "5.3.0.dev0"
+}

model-00001-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7fdcd33aa62a4ae22e09e796539f22f9e53bf25c5b1fb7c84ce5ccbcd2ee5bc0
+size 3222147048

model-00002-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:30345eb0ec75088a0d4a889a5a958323e849124472c16df67081511bf96517d6
+size 3224375392

model-00003-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1315dc3ee572f5015b8d49fba600321556ee73024f15b30bfdbeb47040b9cc38
+size 3224375824

model-00004-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:fb607e422a16b3c1c64a9c6fc55e4883df88207d6da5db90d58c666a08929af2
+size 3223496976

model-00005-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:20cfa407781784c0075a564cae1534db8789f5141b82dd279f59dd0d8ed886f7
+size 3223696816

model-00006-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3d767d67648781841f1e39440e02709f91418f9f1a885027106925d4cc493ebb
+size 3224375584

model-00007-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:545ddf9f33b5985e08cdba889513b9eebfe9be52da5af2e65bd40f45c264ea73
+size 3223496968

model-00008-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:758d4356adee57a428aeaddf1748c2b7e62eb1e60269ac94f760b75ba29ef0a6
+size 3240317256

model-00009-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:90866e0e7fe0d1e687dd3448747de481db9f74a5446fce6077483e3bd3e235bd
+size 3223494952

model-00010-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:3b4de049c020965ffdbde382e09ee7b9b8ba707ca7b2f6416c2bd94fec59012e
+size 3224373040

model-00011-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:031208eec81debe188426e60b61ac9309ebe893e32532ee126801662318be473
+size 3224377944

model-00012-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:97239be745514439521ecfa4de03039546e655ade4335c0737694438fe51f98c
+size 3222824200

model-00013-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:579c5f95dfa2e0881b229283495cc3213e49af8bfb26dddafc494dbb22225fbf
+size 3224372800

model-00014-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b75530fcf7c15047e462ac41430b2eb17912f451bcef1ba30d2721dc7a0cafe1
+size 3224377936

model-00015-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e509ee70d39bba553e52715fe23b876aa0103aa04d1865c7fa1339f9c817a66d
+size 3224377720

model-00016-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:1ce643775786e39c9c4aa9c60e40c439ce9df5f2cf1a103c12e849610381520d
+size 3221936936

model-00017-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:d23b56ebf8d9df041850cb679f92b43f8e0346941e182760e52b16817cb7b504
+size 3224373056

model-00018-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:99581fbe17564558720457dc9501b5e39fc4e58fda9b30673a0a469ccf2d1add
+size 3224377944

model-00019-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:2d37884a4cb0d5406e906cc543cd03c94f037a81a0301a5947933169bdc10850
+size 3222824256

model-00020-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:da5a4b406183759d275d31925ed457f8506ab87d05195302e1c381e689ea6501
+size 3224372824

model-00021-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:86f155aa069e31c750158286434371df19096f330909a0805479ededbbfb91ba
+size 3224377936

model-00022-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a2484d8027f746da494839533110258680cb50a0641548fc72e9cffad5ab92b2
+size 3222823552

model-00023-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:65cba12b6a60a1c5d0c04321e3b33462deb4cd5c80d42a5036dc3db8521055ff
+size 3224372560

model-00024-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:8501d30258d824336ba7c7d2b9ab16ebd58da778b41c77a9246104f5a6909847
+size 3224377736

model-00025-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:414ffa4f6b12c44a3ac23909e2bcc6fa7c422d349736693ce7f8a27b87fe09b4
+size 3224377944

model-00026-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6c1632893165ec76e1a8e088847df0e2b021e0f6325efb0012cbb1d15ae64ebb
+size 3222819320

model-00027-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:6ab61a92de7cca327405b872df1f689b1609f26eac3b80497762e543ae408d28
+size 3224377504

model-00028-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:a1a8f82367eab9a68c40bf45257028c85331e37cdf4154c4a61b40024d1c543d
+size 3224377936

model-00029-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9a505ea3535f549a3039fdd244a6386acee315faa5ef16178497259db442247f
+size 3264409712

model-00030-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4ca4367e0017a719db470b4390c9b63a51ea661ccb1a39e420ca76d199a99320
+size 3223691272

model-00031-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c1c759b411efc03efe79da02eda01ecb59d8643607f941c62e438a0c94fef7bf
+size 3224377496

model-00032-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:81862d90448b6369090c58eac26084db2ab1dd054006e580e1e018fbf90bdfcd
+size 3223499024

model-00033-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:69d9d070460b75122934a936b7ef0490b1a5da658dbcd005bfad026c53a886cf
+size 3223698408

model-00034-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:4e777d95d3e43fb39a8817adfc89b6b9936cdba093f058a2810d5b77f4a93e52
+size 3224377496

model-00035-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:94fb043926dd2ff3e1f6f9ac62deef6c46e0ec20d4661febc8d6a43962d26894
+size 3223499008

model-00036-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e720a35a937f310daf4fdb760185b9167d4667a2e8e1f25edc184eda204e1674
+size 3224373464

model-00037-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:70523c3e00e96e0164bbe1b357c33348d7a37e353234ac4e52ff42578d803e43
+size 3223703648

model-00038-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c9d8eae78544bdde2a7dbad55a58c693d2bc15a1c1811f0d06c6ca07f9bcdfec
+size 3223498824

model-00039-of-00039.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9bbd1b92cb0befef58c847070212581ca588e3f43265a1d07fc8b21091990fd5
+size 1153165696

model.safetensors.index.json ADDED Viewed

The diff for this file is too large to render. See raw diff

preprocessor_config.json ADDED Viewed

	@@ -0,0 +1,21 @@

+{
+    "size": {
+        "longest_edge": 16777216,
+        "shortest_edge": 65536
+    },
+    "patch_size": 16,
+    "temporal_patch_size": 2,
+    "merge_size": 2,
+    "image_mean": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "image_std": [
+        0.5,
+        0.5,
+        0.5
+    ],
+    "processor_class": "Qwen3VLProcessor",
+    "image_processor_type": "Qwen2VLImageProcessorFast"
+}

tokenizer.json ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:639e352c0f904c1875d448ebed6f6faac005fd3eb58393b7f1fb3ff044e5ca03
+size 19989510

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,31 @@

+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}

vllm_patches/patch_qwen35_moe_text.py ADDED Viewed

	@@ -0,0 +1,290 @@

+#!/usr/bin/env python3
+"""
+vLLM patch: Qwen3.5 MoE text-only compatibility shim.
+Creates a text-only subclass of Qwen3_5MoeForConditionalGeneration that:
+- Reuses the wrapper's hybrid cache-spec calculation (fixes page-size bug)
+- Skips vision encoder initialization entirely
+- Sets supports_multimodal = False (prevents multimodal warmup)
+- Registers as Qwen3_5MoeForCausalLM in the model registry
+"""
+import re
+import textwrap
+REGISTRY_PATH = "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/registry.py"
+QWEN35_PATH = "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen3_5.py"
+def patch_registry():
+    """Map Qwen3_5MoeForCausalLM -> Qwen3_5MoeTextOnlyShim."""
+    with open(REGISTRY_PATH) as f:
+        content = f.read()
+    entry = '"Qwen3_5MoeForCausalLM"'
+    if entry in content:
+        # Update existing entry
+        content = re.sub(
+            r'"Qwen3_5MoeForCausalLM": \(\s*"qwen3_5",\s*"[^"]+",\s*\)',
+            '"Qwen3_5MoeForCausalLM": (\n        "qwen3_5",\n        "Qwen3_5MoeTextOnlyShim",\n    )',
+            content,
+        )
+        print("[patch] registry: Updated Qwen3_5MoeForCausalLM -> TextOnlyShim")
+    else:
+        # Add new entry
+        target = '"Qwen3_5MoeForConditionalGeneration": (\n        "qwen3_5",\n        "Qwen3_5MoeForConditionalGeneration",\n    ),'
+        insert = target + '\n    "Qwen3_5MoeForCausalLM": (\n        "qwen3_5",\n        "Qwen3_5MoeTextOnlyShim",\n    ),'
+        if target in content:
+            content = content.replace(target, insert)
+        else:
+            lines = content.split('\n')
+            for i, line in enumerate(lines):
+                if '"Qwen3_5MoeForConditionalGeneration"' in line:
+                    for j in range(i, min(i+5, len(lines))):
+                        if lines[j].strip() == '),':
+                            lines.insert(j+1, '    "Qwen3_5MoeForCausalLM": (')
+                            lines.insert(j+2, '        "qwen3_5",')
+                            lines.insert(j+3, '        "Qwen3_5MoeTextOnlyShim",')
+                            lines.insert(j+4, '    ),')
+                            content = '\n'.join(lines)
+                            break
+                    break
+        print("[patch] registry: Added Qwen3_5MoeForCausalLM -> TextOnlyShim")
+    with open(REGISTRY_PATH, 'w') as f:
+        f.write(content)
+def patch_add_text_only_shim():
+    """Add Qwen3_5MoeTextOnlyShim class to qwen3_5.py."""
+    with open(QWEN35_PATH) as f:
+        content = f.read()
+    if "Qwen3_5MoeTextOnlyShim" in content:
+        print("[patch] qwen3_5: TextOnlyShim already exists")
+        return
+    # The shim class: inherits ConditionalGeneration for cache-spec,
+    # but overrides __init__ to skip vision, and sets supports_multimodal = False
+    shim_code = '''
+########################################################
+# Text-only compatibility shim
+# Reuses ConditionalGeneration's cache-spec but skips vision
+########################################################
+class Qwen3_5MoeTextOnlyShim(Qwen3_5MoeForConditionalGeneration):
+    """Text-only shim for Qwen3.5 MoE CausalLM checkpoints.
+    Inherits Qwen3_5MoeForConditionalGeneration for hybrid cache-spec
+    calculation (fixing the page-size bug in CausalLM path), but:
+    - Does NOT initialize vision encoder
+    - Does NOT register as multimodal
+    - Rejects multimodal input at forward time
+    """
+    # Override: NOT a multimodal model
+    supports_multimodal = False
+    def __init__(self, *, vllm_config, prefix: str = "model"):
+        import logging
+        log = logging.getLogger("qwen3_5_text_only_shim")
+        # Skip the parent's multimodal __init__ entirely
+        # Go directly to nn.Module.__init__
+        nn.Module.__init__(self)
+        config = vllm_config.model_config.hf_config
+        self.config = config
+        # vision_config is now a dummy with safe values (hidden_size=128)
+        # created by the patched Qwen3_5MoeConfig.__init__
+        vc = getattr(config, "vision_config", None)
+        if vc is not None:
+            log.info(f"vision_config present: hidden_size={getattr(vc, 'hidden_size', '?')}")
+        # Inject dummy MultiModalConfig so _mark_language_model works
+        # All defaults are safe for text-only (mm_encoder_only=False, etc.)
+        from vllm.config.multimodal import MultiModalConfig
+        if vllm_config.model_config.multimodal_config is None:
+            vllm_config.model_config.multimodal_config = MultiModalConfig()
+            log.info("Injected dummy MultiModalConfig for text-only shim")
+        self.multimodal_config = vllm_config.model_config.multimodal_config
+        self.visual = None
+        self.use_data_parallel = False
+        self.is_multimodal_pruning_enabled = False
+        self._text_only_mode = True
+        log.info("Qwen3_5MoeTextOnlyShim: text-only mode, vision encoder skipped")
+        # Use _mark_language_model to preserve wrapper's cache-spec path
+        with self._mark_language_model(vllm_config):
+            self.language_model = Qwen3_5MoeForCausalLM(
+                vllm_config=vllm_config,
+                prefix=maybe_prefix(prefix, "language_model"),
+            )
+        self.make_empty_intermediate_tensors = (
+            self.language_model.make_empty_intermediate_tensors
+        )
+        # set MoE hyperparameters
+        self.set_moe_parameters()
+    def forward(self, *args, **kwargs):
+        """Forward: delegate to language_model, reject multimodal input."""
+        if kwargs.get("pixel_values") is not None or kwargs.get("image_grid_thw") is not None:
+            raise ValueError(
+                "Qwen3_5MoeTextOnlyShim does not support multimodal input. "
+                "This model was loaded as text-only."
+            )
+        return self.language_model(*args, **kwargs)
+    def load_weights(self, weights):
+        """Load weights with key remapping for text-only checkpoints."""
+        import logging
+        log = logging.getLogger("qwen3_5_text_only_shim")
+        def _remap(weights_iter):
+            remapped = False
+            for name, tensor in weights_iter:
+                new_name = name
+                # ModelOpt export uses model.language_model.* prefix
+                # which matches our module tree (self.language_model.model.*)
+                # No remapping needed for ConditionalGeneration path
+                # since self.language_model prefix is already "language_model"
+                yield new_name, tensor
+        loader = AutoWeightsLoader(self, skip_prefixes=["mtp.", "visual."])
+        return loader.load_weights(_remap(weights), mapper=self.hf_to_vllm_mapper)
+'''
+    # Insert before the final class or at the end of file
+    # Find the last class definition to insert after
+    insert_pos = content.rfind('\nclass Qwen3_5MoeForConditionalGeneration')
+    if insert_pos == -1:
+        # Append at end
+        content += shim_code
+    else:
+        # Find the end of Qwen3_5MoeForConditionalGeneration class (next class or EOF)
+        # Insert after the entire ConditionalGeneration class
+        # Find the set_moe_parameters() call which is the last line of __init__
+        end_of_class = content.find('\nclass ', insert_pos + 10)
+        if end_of_class == -1:
+            content += shim_code
+        else:
+            # Actually, insert at the very end of the file
+            content += shim_code
+    with open(QWEN35_PATH, 'w') as f:
+        f.write(content)
+    print("[patch] qwen3_5: Added Qwen3_5MoeTextOnlyShim class")
+def patch_processing_info():
+    """Patch ProcessingInfo to handle text-only config gracefully."""
+    with open(QWEN35_PATH) as f:
+        content = f.read()
+    if "text_only_shim_processing" in content:
+        print("[patch] qwen3_5: ProcessingInfo already patched")
+        return
+    old = '''class Qwen3_5MoeProcessingInfo(Qwen3VLProcessingInfo):
+    def get_hf_config(self):
+        return self.ctx.get_hf_config(Qwen3_5MoeConfig)'''
+    new = '''class Qwen3_5MoeProcessingInfo(Qwen3VLProcessingInfo):
+    # text_only_shim_processing
+    def get_hf_config(self):
+        try:
+            return self.ctx.get_hf_config(Qwen3_5MoeConfig)
+        except TypeError:
+            return self.ctx.model_config.hf_config
+    def get_data_parser(self):
+        config = self.get_hf_config()
+        if not hasattr(config, "vision_config") or config.vision_config is None:
+            from vllm.multimodal.parse import MultiModalDataParser
+            return MultiModalDataParser()
+        return super().get_data_parser()
+    def get_max_image_tokens(self):
+        config = self.get_hf_config()
+        if not hasattr(config, "vision_config") or config.vision_config is None:
+            return 0
+        return super().get_max_image_tokens()
+    def get_max_video_tokens(self, seq_len, mm_counts=None):
+        config = self.get_hf_config()
+        if not hasattr(config, "vision_config") or config.vision_config is None:
+            return 0
+        return super().get_max_video_tokens(seq_len, mm_counts)
+    def get_image_size_with_most_features(self):
+        config = self.get_hf_config()
+        if not hasattr(config, "vision_config") or config.vision_config is None:
+            return (0, 0)
+        return super().get_image_size_with_most_features()'''
+    if old in content:
+        content = content.replace(old, new)
+        with open(QWEN35_PATH, 'w') as f:
+            f.write(content)
+        print("[patch] qwen3_5: ProcessingInfo patched")
+    else:
+        print("[patch] qwen3_5: ProcessingInfo already modified or not found")
+def patch_config_vision_default():
+    """Prevent Qwen3_5MoeConfig from auto-creating vision_config when None.
+    The original code: if vision_config is None -> create default VisionConfig.
+    We change it: if vision_config is None -> keep as None.
+    This prevents vision hidden_size=1152 from leaking into FP8 TP2 validation.
+    """
+    CONFIG_PATH = "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/configs/qwen3_5_moe.py"
+    with open(CONFIG_PATH) as f:
+        content = f.read()
+    if "text_only_shim_config" in content:
+        print("[patch] config: vision_config default already patched")
+        return
+    old = '        elif vision_config is None:\n            self.vision_config = self.sub_configs["vision_config"]()'
+    # Instead of None, use a dummy with minimal safe values
+    # This prevents NoneType errors in multimodal processing code
+    # while keeping hidden_size small enough for TP2 block validation
+    new = '''        elif vision_config is None:
+            # text_only_shim_config: create minimal dummy vision config
+            # with safe values that pass TP2 block-wise FP8 validation
+            # hidden_size=128 is divisible by block_size=128 and any TP
+            self.vision_config = self.sub_configs["vision_config"](
+                hidden_size=128, intermediate_size=256, depth=0,
+                num_heads=1, patch_size=16, spatial_merge_size=2,
+                temporal_patch_size=2, in_channels=3,
+            )'''
+    if old in content:
+        content = content.replace(old, new)
+        with open(CONFIG_PATH, 'w') as f:
+            f.write(content)
+        print("[patch] config: vision_config default -> None (text-only safe)")
+    else:
+        print("[patch] config: Could not find vision_config default pattern")
+if __name__ == "__main__":
+    print("=" * 55)
+    print("vLLM Patch: Qwen3.5 MoE text-only shim v3")
+    print("=" * 55)
+    patch_registry()
+    patch_add_text_only_shim()
+    patch_processing_info()
+    patch_config_vision_default()
+    print("=" * 55)
+    print("Patch complete")