Hugging Face's logo

nota-ai
/
Solar-Open2-250B-Nota-INT4

Text Generation
Safetensors
English
Korean
Japanese
vllm
solar_open2
quantization
int4
Mixture of Experts
nota
conversational
custom_code
4-bit precision
auto-round
Model card Files
xet
 Community
hancheolp commited on
Commit
ab5ee86
·
verified ·
1 Parent(s): 9de90d1

Upload Solar Open2 250B Nota INT4

Browse files
Files changed (39) hide show
  1. .gitattributes +2 -0
  2. README.md +104 -0
  3. chat_template.jinja +229 -0
  4. config.json +2734 -0
  5. configuration_solar_open2.py +277 -0
  6. generation_config.json +9 -0
  7. model-00001-of-00027.safetensors +3 -0
  8. model-00002-of-00027.safetensors +3 -0
  9. model-00003-of-00027.safetensors +3 -0
  10. model-00004-of-00027.safetensors +3 -0
  11. model-00005-of-00027.safetensors +3 -0
  12. model-00006-of-00027.safetensors +3 -0
  13. model-00007-of-00027.safetensors +3 -0
  14. model-00008-of-00027.safetensors +3 -0
  15. model-00009-of-00027.safetensors +3 -0
  16. model-00010-of-00027.safetensors +3 -0
  17. model-00011-of-00027.safetensors +3 -0
  18. model-00012-of-00027.safetensors +3 -0
  19. model-00013-of-00027.safetensors +3 -0
  20. model-00014-of-00027.safetensors +3 -0
  21. model-00015-of-00027.safetensors +3 -0
  22. model-00016-of-00027.safetensors +3 -0
  23. model-00017-of-00027.safetensors +3 -0
  24. model-00018-of-00027.safetensors +3 -0
  25. model-00019-of-00027.safetensors +3 -0
  26. model-00020-of-00027.safetensors +3 -0
  27. model-00021-of-00027.safetensors +3 -0
  28. model-00022-of-00027.safetensors +3 -0
  29. model-00023-of-00027.safetensors +3 -0
  30. model-00024-of-00027.safetensors +3 -0
  31. model-00025-of-00027.safetensors +3 -0
  32. model-00026-of-00027.safetensors +3 -0
  33. model-00027-of-00027.safetensors +3 -0
  34. model.safetensors.index.json +3 -0
  35. modeling_solar_open2.py +1083 -0
  36. quantization_config.json +2659 -0
  37. special_tokens_map.json +4008 -0
  38. tokenizer.json +3 -0
  39. tokenizer_config.json +0 -0
.gitattributes CHANGED
@@ -33,3 +33,5 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ model.safetensors.index.json filter=lfs diff=lfs merge=lfs -text
37
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,104 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ base_model: upstage/Solar-Open2-250B
3
+ base_model_relation: quantized
4
+ library_name: vllm
5
+ pipeline_tag: text-generation
6
+ language:
7
+ - en
8
+ - ko
9
+ - ja
10
+ tags:
11
+ - quantization
12
+ - int4
13
+ - vllm
14
+ - moe
15
+ - nota
16
+ ---
17
+
18
+ # Solar Open2 250B — Nota INT4
19
+
20
+ [Nota AI](https://www.nota.ai/) presents a 4-bit quantized release of
21
+ [Upstage](https://www.upstage.ai/)'s
22
+ [Solar Open2 250B](https://huggingface.co/upstage/Solar-Open2-250B), produced with
23
+ Nota AI's proprietary quantization technology specialized for Mixture-of-Experts (MoE)
24
+ large language models.
25
+
26
+ ## Highlights
27
+
28
+ - **W4A16 (weight-only INT4)** — `group_size=128`, packed in the `auto_round`
29
+ (GPTQ-compatible) format. Formatted with AutoRound for direct, out-of-the-box
30
+ serving in [vLLM](https://github.com/vllm-project/vllm).
31
+ - **Nota AI's proprietary MoE quantization framework.** This release is built upon a
32
+ suite of techniques developed by Nota AI to preserve model quality under aggressive
33
+ low-bit quantization of MoE architectures:
34
+ - A MoE-specialized calibration-dataset construction method, which achieved
35
+ **1st place across all tracks** at the NVIDIA Nemotron Hackathon.
36
+ - [**DREAM-MoE**](https://openreview.net/pdf?id=Wyhqwjl51A) and
37
+ [**SRA-MoE**](https://openreview.net/pdf?id=H0NoX02erJ), two quantization algorithms
38
+ proposed by Nota AI (published at the *ICML 2026 Workshop on AdaptFM*), which
39
+ preserve MoE routing decisions and align expert-routing behavior throughout the
40
+ quantization process.
41
+
42
+
43
+ ## Performance
44
+
45
+ ### Weight footprint
46
+
47
+ | Precision | Weight footprint |
48
+ | ----------------- | :--------------: |
49
+ | BF16 | 500.6 GB |
50
+ | W4A16 (INT4-Nota) | 142.9 GB |
51
+
52
+ ### Benchmarks
53
+
54
+ | Benchmark | BF16 | W4A16 (INT4-Nota) |
55
+ | ----------------------- | :--------------: | :---------------: |
56
+ | Tau2-Bench | 75.20 | 74.04 |
57
+ | HLE | 27.88 | 27.15 |
58
+ | GPQA Diamond | 86.26 | 85.86 |
59
+ | IFBench | 80.00 | 80.54 |
60
+ | LiveCodeBench (v5–v6) | 87.03 | 85.50 |
61
+ | MMLU-Pro | 86.19 | 86.29 |
62
+ | AIME 2026 (EN) | 95.67 | 96.00 |
63
+ | IFEval (EN) | 94.09 | 93.35 |
64
+ | HMMT | 92.05 | 90.91 |
65
+ | KMMLU-Pro | 78.38 | 78.79 |
66
+ | HAE-RAE Bench v1.1 | 73.84 | 72.53 |
67
+ | AIME (KO) | 97.67 | 97.67 |
68
+ | KBL | 75.51 | 75.20 |
69
+ | KBank-MMLU | 80.80 | 80.88 |
70
+ | KorMedMCQA | 92.99 | 92.85 |
71
+ | **Avg.** | **81.57** | **81.17** |
72
+
73
+ ## Usage
74
+
75
+ This model is packed in the AutoRound (GPTQ-compatible) INT4 format and can be served
76
+ directly with vLLM:
77
+
78
+ ```bash
79
+ vllm serve nota-ai/Solar-Open2-250B-Nota-INT4 \
80
+ --tensor-parallel-size 8 \
81
+ --trust-remote-code
82
+ ```
83
+
84
+ See the original model card for prompt format and further details:
85
+ [upstage/Solar-Open2-250B](https://huggingface.co/upstage/Solar-Open2-250B).
86
+
87
+ ## Citation
88
+ ```bibtex
89
+ @inproceedings{park2026dreammoe,
90
+ title = {{DREAM-MoE}: Downstream Routing Error-Aware Margin-Preserving Quantization for Mixture-of-Experts Large Language Models},
91
+ author = {Park, Hancheol and Lee, Geonho and Kim, Tae-Ho},
92
+ booktitle = {ICML 2026 Workshop on Adaptive Foundation Models (AdaptFM)},
93
+ year = {2026},
94
+ url = {https://openreview.net/forum?id=Wyhqwjl51A},
95
+ }
96
+
97
+ @inproceedings{lee2026sramoe,
98
+ title = {{SRA-MoE}: Output-Aware Selective Router Alignment for MoE Quantization},
99
+ author = {Lee, Geonho and Park, Hancheol and Lee, Seunghyun and Choi, Jungwook and Kim, Tae-Ho},
100
+ booktitle = {ICML 2026 Workshop on Adaptive Foundation Models (AdaptFM)},
101
+ year = {2026},
102
+ url = {https://openreview.net/forum?id=H0NoX02erJ},
103
+ }
104
+ ```
chat_template.jinja ADDED
@@ -0,0 +1,229 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {#- ======== Template Parameters ======== #}
2
+ {%- set add_generation_prompt = add_generation_prompt if add_generation_prompt is defined else true %}
3
+ {%- set provider_system_prompt = provider_system_prompt if provider_system_prompt else false %}
4
+ {%- set reasoning_effort = reasoning_effort if reasoning_effort is defined else "high" %}
5
+ {%- set think_render_option = think_render_option if think_render_option is defined else "interleaved" %}
6
+
7
+ {#- ======== System Block State ======== #}
8
+ {%- set sys_ns = namespace(is_first_block=true) -%}
9
+
10
+ {#- ======== Find last user message index ======== #}
11
+ {%- set last_user_idx = namespace(value=-1) -%}
12
+ {%- for message in messages -%}
13
+ {%- if message.role == 'user' -%}
14
+ {%- set last_user_idx.value = loop.index0 -%}
15
+ {%- endif -%}
16
+ {%- endfor -%}
17
+
18
+ {#- ======== System messages renderers ======== #}
19
+ {%- macro render_system_message(user_system_messages) %}
20
+ {%- if provider_system_prompt %}
21
+ {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
22
+ {%- set sys_ns.is_first_block = false %}
23
+ {{- "## Provider System Prompt\n\nYou are Solar Open2 250B, a large language model trained by Upstage AI, a Korean startup. Your knowledge cutoff is 2026-02. The current date is " + strftime_now("%Y-%m-%d") + "." }}
24
+ {%- endif -%}
25
+ {%- if user_system_messages %}
26
+ {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
27
+ {%- set sys_ns.is_first_block = false %}
28
+ {{- "## System Prompt" }}
29
+ {%- for system_message in user_system_messages %}
30
+ {{- "\n\n" }}
31
+ {{- system_message }}
32
+ {%- endfor %}
33
+ {%- endif -%}
34
+ {%- endmacro %}
35
+
36
+ {%- macro render_tool_instruction(tools, is_last_block) %}
37
+ {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
38
+ {%- set sys_ns.is_first_block = false %}
39
+ {{- "## Tools\n- You may invoke one or more tools to assist with the user's query." }}
40
+ {{- "\n\n### Available Tools\n" }}
41
+ {%- for tool in tools %}
42
+ {{- "<|tool:start|>" }}
43
+ {{- tool.function | tojson }}
44
+ {{- "<|tool:end|>\n" }}
45
+ {%- endfor %}
46
+ {{- "\n### Tool Call Instruction\n" }}
47
+ {{- "- If using a tool, any reasoning must strictly precede the call. Do not append any text after the tool call.\n" }}
48
+ {{- "- If no tool is required, answer directly from your knowledge without ever mentioning the availability or absence of tools.\n" }}
49
+ {{- "- Each tool call MUST use this following format: <|tool_call:start|>{example-tool-name}\n<|tool_arg:start|>{example-key-name-1}<|tool_arg:value|>{example-value-1}<|tool_arg:end|>\n<|tool_arg:start|>{example-key-name-2}<|tool_arg:value|>{example-value-2}<|tool_arg:end|>\n<|tool_call:end|>\n" }}
50
+ {%- endmacro %}
51
+
52
+ {#-
53
+ Input convention for `response_format`:
54
+ - Per the OpenAI Chat Completions API, `response_format` is an object of one of:
55
+ {"type": "json_schema", "json_schema": {"name": ..., "schema": {...}, "strict": ...}}
56
+ {"type": "json_object"}
57
+ - Only `response_format.json_schema.schema` (the inner JSON Schema) is injected
58
+ into the prompt; the OpenAI wrapper fields (`name`, `strict`, ...) are dropped.
59
+ - For `json_object`, no schema exists, so we emit a generic JSON-only instruction.
60
+ - vLLM does NOT pass `response_format` to chat templates by default; it routes it
61
+ to guided decoding instead. This template/whale opts into prompt injection,
62
+ so callers serving the model directly via vLLM must opt in via
63
+ `chat_template_kwargs` (or rely on whale's normalization).
64
+ -#}
65
+ {%- macro render_json_response_format_instruction(response_format) %}
66
+ {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
67
+ {%- set sys_ns.is_first_block = false %}
68
+ {{- "## Output Format Constraint" }}
69
+ {%- if response_format.type == "json_schema" %}
70
+ {{- "\n\n- Your final response should follow the JSON schema:\n```json\n" }}
71
+ {{- response_format.json_schema.schema | tojson }}
72
+ {{- "\n```\n- Please ensure your answers adhere to this format and do not contain any unnecessary text." }}
73
+ {%- elif response_format.type == "json_object" %}
74
+ {{- "\n\n- Your final response must be a valid JSON object." }}
75
+ {{- "\n- Do not include any text outside the JSON object." }}
76
+ {%- endif %}
77
+ {%- endmacro %}
78
+
79
+ {#-
80
+ Input convention for `tool_arguments`:
81
+ - Must be a mapping (dict), NOT a JSON-encoded string.
82
+ - The OpenAI API spec defines `tool_calls[].function.arguments` as a JSON string,
83
+ but vLLM, HuggingFace transformers tool-use templates (Qwen, Hermes, Llama, ...),
84
+ and this template all expect the value to be parsed into a dict before rendering.
85
+ - vLLM normalizes this in `_postprocess_messages()` (vllm/entrypoints/chat_utils.py).
86
+ - Callers (whale, training pipelines that call `tokenizer.apply_chat_template`
87
+ directly, ...) are responsible for parsing the JSON string into a dict before
88
+ invoking the template.
89
+ -#}
90
+ {%- macro render_tool_arguments(tool_arguments) %}
91
+ {%- for args_name, args_value in tool_arguments|items %}
92
+ {{- "<|tool_arg:start|>"+ args_name + "<|tool_arg:value|>"}}
93
+ {%- set args_value = args_value if args_value is string else (args_value | tojson | safe) %}
94
+ {{- args_value }}
95
+ {{- "<|tool_arg:end|>\n" }}
96
+ {%- endfor %}
97
+ {%- endmacro %}
98
+
99
+ {#- ======== Render system message ======== #}
100
+ {%- set ns = namespace(system_messages=[]) -%}
101
+ {%- for message in messages -%}
102
+ {%- if message.role == 'system' -%}
103
+ {%- set ns.system_messages = ns.system_messages + [message.content] -%}
104
+ {%- endif -%}
105
+ {%- endfor -%}
106
+
107
+ {%- if ns.system_messages or provider_system_prompt or tools or response_format -%}
108
+ {{- "<|im:start|>system<|im:content|>" }}
109
+ {{- render_system_message(ns.system_messages) }}
110
+ {%- if tools -%}
111
+ {{- render_tool_instruction(tools, not response_format) }}
112
+ {%- endif %}
113
+ {%- if response_format -%}
114
+ {{- render_json_response_format_instruction(response_format) }}
115
+ {%- endif %}
116
+ {{- "<|im:end|>\n" }}
117
+ {%- endif -%}
118
+
119
+ {#- ======== Render main messages ======== #}
120
+ {%- for message in messages -%}
121
+ {%- if message.role == 'user' -%}
122
+ {{- "<|im:start|>user<|im:content|>" + message.content + "<|im:end|>\n" }}
123
+ {%- elif message.role == 'tool' -%}
124
+ {%- set prev_is_tool = loop.index0 > 0 and messages[loop.index0 - 1].role == 'tool' -%}
125
+ {#- Only render at the start of a contiguous tool block; subsequent tool messages -#}
126
+ {#- are already emitted (reordered) within the first one. -#}
127
+ {%- if not prev_is_tool -%}
128
+ {%- set start_idx = loop.index0 -%}
129
+ {#- Collect contiguous tool messages starting at start_idx -#}
130
+ {%- set tool_ns = namespace(items=[], scanning=true) -%}
131
+ {%- for j in range(start_idx, messages | length) -%}
132
+ {%- if tool_ns.scanning -%}
133
+ {%- if messages[j].role == 'tool' -%}
134
+ {%- set tool_ns.items = tool_ns.items + [messages[j]] -%}
135
+ {%- else -%}
136
+ {%- set tool_ns.scanning = false -%}
137
+ {%- endif -%}
138
+ {%- endif -%}
139
+ {%- endfor -%}
140
+
141
+ {{- "<|im:start|>tool<|im:content|>" }}
142
+
143
+ {%- set prev_msg = messages[start_idx - 1] if start_idx > 0 else none -%}
144
+ {%- if prev_msg and prev_msg.role == 'assistant' and prev_msg.tool_calls -%}
145
+ {#- Render tool responses in the order of the preceding assistant's tool_calls ids. -#}
146
+ {%- set sep_ns = namespace(first=true, rendered_ids=[]) -%}
147
+ {%- for tool_call in prev_msg.tool_calls -%}
148
+ {%- for tool_msg in tool_ns.items -%}
149
+ {%- if tool_msg.tool_call_id == tool_call.id -%}
150
+ {%- if not sep_ns.first -%}{{- "\n" }}{%- endif -%}
151
+ {%- set sep_ns.first = false -%}
152
+ {%- set sep_ns.rendered_ids = sep_ns.rendered_ids + [tool_msg.tool_call_id] -%}
153
+ {{- "<|tool_response:start|>" + tool_msg.content + "<|tool_response:end|>" }}
154
+ {%- endif -%}
155
+ {%- endfor -%}
156
+ {%- endfor -%}
157
+ {#- Append any orphan tool messages (no matching id) in their original order. -#}
158
+ {%- for tool_msg in tool_ns.items -%}
159
+ {%- if tool_msg.tool_call_id not in sep_ns.rendered_ids -%}
160
+ {%- if not sep_ns.first -%}{{- "\n" }}{%- endif -%}
161
+ {%- set sep_ns.first = false -%}
162
+ {{- "<|tool_response:start|>" + tool_msg.content + "<|tool_response:end|>" }}
163
+ {%- endif -%}
164
+ {%- endfor -%}
165
+ {%- else -%}
166
+ {%- for tool_msg in tool_ns.items -%}
167
+ {%- if not loop.first -%}{{- "\n" }}{%- endif -%}
168
+ {{- "<|tool_response:start|>" + tool_msg.content + "<|tool_response:end|>" }}
169
+ {%- endfor -%}
170
+ {%- endif -%}
171
+
172
+ {{- "\n<|im:end|>\n" }}
173
+ {%- endif -%}
174
+ {%- elif message.role == 'assistant' -%}
175
+ {{- "<|im:start|>assistant<|im:content|>" }}
176
+
177
+ {#- ======== Resolve reasoning field (reasoning or reasoning_content) ======== #}
178
+ {%- if message.reasoning is defined -%}
179
+ {%- set reasoning = message.reasoning -%}
180
+ {%- elif message.reasoning_content is defined -%}
181
+ {%- set reasoning = message.reasoning_content -%}
182
+ {%- else -%}
183
+ {%- set reasoning = none -%}
184
+ {%- endif -%}
185
+
186
+ {#- ======== Assistant Thinking ======== #}
187
+ {%- if think_render_option == "preserved" -%}
188
+ {%- if reasoning -%}
189
+ {{- "<|think:start|>" + reasoning + "<|think:end|>" }}
190
+ {%- else -%}
191
+ {{- "<|think:start|><|think:end|>" }}
192
+ {%- endif -%}
193
+ {%- elif think_render_option == "interleaved" -%}
194
+ {%- if reasoning and loop.index0 > last_user_idx.value -%}
195
+ {{- "<|think:start|>" + reasoning + "<|think:end|>" }}
196
+ {%- else -%}
197
+ {{- "<|think:start|><|think:end|>" }}
198
+ {%- endif -%}
199
+ {%- endif -%}
200
+
201
+ {#- ======== Assistant Messages ======== #}
202
+ {%- if message.content -%}
203
+ {{- message.content }}
204
+ {%- endif -%}
205
+
206
+ {#- ======== Assistant Tool calls ======== #}
207
+ {%- if message.tool_calls -%}
208
+ {%- for tool_call in message.tool_calls -%}
209
+ {%- if not loop.first -%}{{- "\n" }}{%- endif -%}
210
+ {{- "<|tool_call:start|>" + tool_call.function.name + "\n" }}
211
+ {#- ======== FIXME: QWEN3.5 Method ======== #}
212
+ {%- if tool_call.function.arguments is defined %}
213
+ {{- render_tool_arguments(tool_call.function.arguments) }}
214
+ {%- endif %}
215
+ {{- "<|tool_call:end|>" }}
216
+ {%- endfor -%}
217
+ {{- "\n" }}
218
+ {%- endif -%}
219
+ {{- "<|im:end|>\n" }}
220
+ {%- endif -%}
221
+ {%- endfor -%}
222
+
223
+ {%- if add_generation_prompt -%}
224
+ {%- if reasoning_effort in ["medium", "high"] -%}
225
+ {{- "<|im:start|>assistant<|im:content|><|think:start|>" }}
226
+ {%- else -%}
227
+ {{- "<|im:start|>assistant<|im:content|><|think:start|><|think:end|>" }}
228
+ {%- endif -%}
229
+ {%- endif -%}
config.json ADDED
@@ -0,0 +1,2734 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "SolarOpen2ForCausalLM"
4
+ ],
5
+ "attention_bias": false,
6
+ "attention_dropout": 0.0,
7
+ "auto_map": {
8
+ "AutoConfig": "configuration_solar_open2.SolarOpen2Config",
9
+ "AutoModel": "modeling_solar_open2.SolarOpen2Model",
10
+ "AutoModelForCausalLM": "modeling_solar_open2.SolarOpen2ForCausalLM"
11
+ },
12
+ "bos_token_id": 1,
13
+ "dtype": "bfloat16",
14
+ "eos_token_id": 2,
15
+ "first_k_dense_replace": 0,
16
+ "gqa_interval": 3,
17
+ "gqa_layers": [
18
+ 0,
19
+ 4,
20
+ 8,
21
+ 12,
22
+ 16,
23
+ 20,
24
+ 24,
25
+ 28,
26
+ 32,
27
+ 36,
28
+ 40,
29
+ 44
30
+ ],
31
+ "hc_dynamic": true,
32
+ "hc_rate": 0,
33
+ "hc_use_sigmoid": false,
34
+ "hc_use_tanh": true,
35
+ "head_dim": 128,
36
+ "hidden_act": "silu",
37
+ "hidden_size": 4096,
38
+ "initializer_range": 0.02,
39
+ "intermediate_size": 10240,
40
+ "kda_allow_neg_eigval": true,
41
+ "kda_gate_lower_bound": -5.0,
42
+ "kda_use_full_proj": false,
43
+ "linear_attn_config": {
44
+ "head_dim": 128,
45
+ "num_heads": 64,
46
+ "num_kv_heads": null,
47
+ "short_conv_kernel_size": 4
48
+ },
49
+ "max_position_embeddings": 1048576,
50
+ "model_type": "solar_open2",
51
+ "moe_intermediate_size": 1280,
52
+ "n_group": 1,
53
+ "n_routed_experts": 320,
54
+ "n_shared_experts": 1,
55
+ "norm_topk_prob": true,
56
+ "num_attention_heads": 64,
57
+ "num_experts_per_tok": 8,
58
+ "num_hidden_layers": 48,
59
+ "num_key_value_heads": 8,
60
+ "pad_token_id": 2,
61
+ "partial_rotary_factor": 1.0,
62
+ "quantization_config": {
63
+ "autoround_version": "0.13.1",
64
+ "bits": 4,
65
+ "block_name_to_quantize": "model.layers",
66
+ "data_type": "int",
67
+ "extra_config": {
68
+ ".*model\\.layers\\.[0-47]\\.mlp\\.gate.*": {
69
+ "bits": 16,
70
+ "data_type": "float"
71
+ },
72
+ "model.layers.0.mlp.shared_experts.down_proj": {
73
+ "act_data_type": "fp",
74
+ "bits": 16,
75
+ "data_type": "fp"
76
+ },
77
+ "model.layers.0.mlp.shared_experts.gate_proj": {
78
+ "act_data_type": "fp",
79
+ "bits": 16,
80
+ "data_type": "fp"
81
+ },
82
+ "model.layers.0.mlp.shared_experts.up_proj": {
83
+ "act_data_type": "fp",
84
+ "bits": 16,
85
+ "data_type": "fp"
86
+ },
87
+ "model.layers.0.self_attn.g_proj": {
88
+ "act_data_type": "fp",
89
+ "bits": 16,
90
+ "data_type": "fp"
91
+ },
92
+ "model.layers.0.self_attn.k_proj": {
93
+ "act_data_type": "fp",
94
+ "bits": 16,
95
+ "data_type": "fp"
96
+ },
97
+ "model.layers.0.self_attn.o_proj": {
98
+ "act_data_type": "fp",
99
+ "bits": 16,
100
+ "data_type": "fp"
101
+ },
102
+ "model.layers.0.self_attn.q_proj": {
103
+ "act_data_type": "fp",
104
+ "bits": 16,
105
+ "data_type": "fp"
106
+ },
107
+ "model.layers.0.self_attn.v_proj": {
108
+ "act_data_type": "fp",
109
+ "bits": 16,
110
+ "data_type": "fp"
111
+ },
112
+ "model.layers.1.mlp.shared_experts.down_proj": {
113
+ "act_data_type": "fp",
114
+ "bits": 16,
115
+ "data_type": "fp"
116
+ },
117
+ "model.layers.1.mlp.shared_experts.gate_proj": {
118
+ "act_data_type": "fp",
119
+ "bits": 16,
120
+ "data_type": "fp"
121
+ },
122
+ "model.layers.1.mlp.shared_experts.up_proj": {
123
+ "act_data_type": "fp",
124
+ "bits": 16,
125
+ "data_type": "fp"
126
+ },
127
+ "model.layers.1.self_attn.b_proj": {
128
+ "act_data_type": "fp",
129
+ "bits": 16,
130
+ "data_type": "fp"
131
+ },
132
+ "model.layers.1.self_attn.f_a_proj": {
133
+ "act_data_type": "fp",
134
+ "bits": 16,
135
+ "data_type": "fp"
136
+ },
137
+ "model.layers.1.self_attn.f_b_proj": {
138
+ "act_data_type": "fp",
139
+ "bits": 16,
140
+ "data_type": "fp"
141
+ },
142
+ "model.layers.1.self_attn.g_a_proj": {
143
+ "act_data_type": "fp",
144
+ "bits": 16,
145
+ "data_type": "fp"
146
+ },
147
+ "model.layers.1.self_attn.g_b_proj": {
148
+ "act_data_type": "fp",
149
+ "bits": 16,
150
+ "data_type": "fp"
151
+ },
152
+ "model.layers.1.self_attn.k_proj": {
153
+ "act_data_type": "fp",
154
+ "bits": 16,
155
+ "data_type": "fp"
156
+ },
157
+ "model.layers.1.self_attn.o_proj": {
158
+ "act_data_type": "fp",
159
+ "bits": 16,
160
+ "data_type": "fp"
161
+ },
162
+ "model.layers.1.self_attn.q_proj": {
163
+ "act_data_type": "fp",
164
+ "bits": 16,
165
+ "data_type": "fp"
166
+ },
167
+ "model.layers.1.self_attn.v_proj": {
168
+ "act_data_type": "fp",
169
+ "bits": 16,
170
+ "data_type": "fp"
171
+ },
172
+ "model.layers.10.mlp.shared_experts.down_proj": {
173
+ "act_data_type": "fp",
174
+ "bits": 16,
175
+ "data_type": "fp"
176
+ },
177
+ "model.layers.10.mlp.shared_experts.gate_proj": {
178
+ "act_data_type": "fp",
179
+ "bits": 16,
180
+ "data_type": "fp"
181
+ },
182
+ "model.layers.10.mlp.shared_experts.up_proj": {
183
+ "act_data_type": "fp",
184
+ "bits": 16,
185
+ "data_type": "fp"
186
+ },
187
+ "model.layers.10.self_attn.b_proj": {
188
+ "act_data_type": "fp",
189
+ "bits": 16,
190
+ "data_type": "fp"
191
+ },
192
+ "model.layers.10.self_attn.f_a_proj": {
193
+ "act_data_type": "fp",
194
+ "bits": 16,
195
+ "data_type": "fp"
196
+ },
197
+ "model.layers.10.self_attn.f_b_proj": {
198
+ "act_data_type": "fp",
199
+ "bits": 16,
200
+ "data_type": "fp"
201
+ },
202
+ "model.layers.10.self_attn.g_a_proj": {
203
+ "act_data_type": "fp",
204
+ "bits": 16,
205
+ "data_type": "fp"
206
+ },
207
+ "model.layers.10.self_attn.g_b_proj": {
208
+ "act_data_type": "fp",
209
+ "bits": 16,
210
+ "data_type": "fp"
211
+ },
212
+ "model.layers.10.self_attn.k_proj": {
213
+ "act_data_type": "fp",
214
+ "bits": 16,
215
+ "data_type": "fp"
216
+ },
217
+ "model.layers.10.self_attn.o_proj": {
218
+ "act_data_type": "fp",
219
+ "bits": 16,
220
+ "data_type": "fp"
221
+ },
222
+ "model.layers.10.self_attn.q_proj": {
223
+ "act_data_type": "fp",
224
+ "bits": 16,
225
+ "data_type": "fp"
226
+ },
227
+ "model.layers.10.self_attn.v_proj": {
228
+ "act_data_type": "fp",
229
+ "bits": 16,
230
+ "data_type": "fp"
231
+ },
232
+ "model.layers.11.mlp.shared_experts.down_proj": {
233
+ "act_data_type": "fp",
234
+ "bits": 16,
235
+ "data_type": "fp"
236
+ },
237
+ "model.layers.11.mlp.shared_experts.gate_proj": {
238
+ "act_data_type": "fp",
239
+ "bits": 16,
240
+ "data_type": "fp"
241
+ },
242
+ "model.layers.11.mlp.shared_experts.up_proj": {
243
+ "act_data_type": "fp",
244
+ "bits": 16,
245
+ "data_type": "fp"
246
+ },
247
+ "model.layers.11.self_attn.b_proj": {
248
+ "act_data_type": "fp",
249
+ "bits": 16,
250
+ "data_type": "fp"
251
+ },
252
+ "model.layers.11.self_attn.f_a_proj": {
253
+ "act_data_type": "fp",
254
+ "bits": 16,
255
+ "data_type": "fp"
256
+ },
257
+ "model.layers.11.self_attn.f_b_proj": {
258
+ "act_data_type": "fp",
259
+ "bits": 16,
260
+ "data_type": "fp"
261
+ },
262
+ "model.layers.11.self_attn.g_a_proj": {
263
+ "act_data_type": "fp",
264
+ "bits": 16,
265
+ "data_type": "fp"
266
+ },
267
+ "model.layers.11.self_attn.g_b_proj": {
268
+ "act_data_type": "fp",
269
+ "bits": 16,
270
+ "data_type": "fp"
271
+ },
272
+ "model.layers.11.self_attn.k_proj": {
273
+ "act_data_type": "fp",
274
+ "bits": 16,
275
+ "data_type": "fp"
276
+ },
277
+ "model.layers.11.self_attn.o_proj": {
278
+ "act_data_type": "fp",
279
+ "bits": 16,
280
+ "data_type": "fp"
281
+ },
282
+ "model.layers.11.self_attn.q_proj": {
283
+ "act_data_type": "fp",
284
+ "bits": 16,
285
+ "data_type": "fp"
286
+ },
287
+ "model.layers.11.self_attn.v_proj": {
288
+ "act_data_type": "fp",
289
+ "bits": 16,
290
+ "data_type": "fp"
291
+ },
292
+ "model.layers.12.mlp.shared_experts.down_proj": {
293
+ "act_data_type": "fp",
294
+ "bits": 16,
295
+ "data_type": "fp"
296
+ },
297
+ "model.layers.12.mlp.shared_experts.gate_proj": {
298
+ "act_data_type": "fp",
299
+ "bits": 16,
300
+ "data_type": "fp"
301
+ },
302
+ "model.layers.12.mlp.shared_experts.up_proj": {
303
+ "act_data_type": "fp",
304
+ "bits": 16,
305
+ "data_type": "fp"
306
+ },
307
+ "model.layers.12.self_attn.g_proj": {
308
+ "act_data_type": "fp",
309
+ "bits": 16,
310
+ "data_type": "fp"
311
+ },
312
+ "model.layers.12.self_attn.k_proj": {
313
+ "act_data_type": "fp",
314
+ "bits": 16,
315
+ "data_type": "fp"
316
+ },
317
+ "model.layers.12.self_attn.o_proj": {
318
+ "act_data_type": "fp",
319
+ "bits": 16,
320
+ "data_type": "fp"
321
+ },
322
+ "model.layers.12.self_attn.q_proj": {
323
+ "act_data_type": "fp",
324
+ "bits": 16,
325
+ "data_type": "fp"
326
+ },
327
+ "model.layers.12.self_attn.v_proj": {
328
+ "act_data_type": "fp",
329
+ "bits": 16,
330
+ "data_type": "fp"
331
+ },
332
+ "model.layers.13.mlp.shared_experts.down_proj": {
333
+ "act_data_type": "fp",
334
+ "bits": 16,
335
+ "data_type": "fp"
336
+ },
337
+ "model.layers.13.mlp.shared_experts.gate_proj": {
338
+ "act_data_type": "fp",
339
+ "bits": 16,
340
+ "data_type": "fp"
341
+ },
342
+ "model.layers.13.mlp.shared_experts.up_proj": {
343
+ "act_data_type": "fp",
344
+ "bits": 16,
345
+ "data_type": "fp"
346
+ },
347
+ "model.layers.13.self_attn.b_proj": {
348
+ "act_data_type": "fp",
349
+ "bits": 16,
350
+ "data_type": "fp"
351
+ },
352
+ "model.layers.13.self_attn.f_a_proj": {
353
+ "act_data_type": "fp",
354
+ "bits": 16,
355
+ "data_type": "fp"
356
+ },
357
+ "model.layers.13.self_attn.f_b_proj": {
358
+ "act_data_type": "fp",
359
+ "bits": 16,
360
+ "data_type": "fp"
361
+ },
362
+ "model.layers.13.self_attn.g_a_proj": {
363
+ "act_data_type": "fp",
364
+ "bits": 16,
365
+ "data_type": "fp"
366
+ },
367
+ "model.layers.13.self_attn.g_b_proj": {
368
+ "act_data_type": "fp",
369
+ "bits": 16,
370
+ "data_type": "fp"
371
+ },
372
+ "model.layers.13.self_attn.k_proj": {
373
+ "act_data_type": "fp",
374
+ "bits": 16,
375
+ "data_type": "fp"
376
+ },
377
+ "model.layers.13.self_attn.o_proj": {
378
+ "act_data_type": "fp",
379
+ "bits": 16,
380
+ "data_type": "fp"
381
+ },
382
+ "model.layers.13.self_attn.q_proj": {
383
+ "act_data_type": "fp",
384
+ "bits": 16,
385
+ "data_type": "fp"
386
+ },
387
+ "model.layers.13.self_attn.v_proj": {
388
+ "act_data_type": "fp",
389
+ "bits": 16,
390
+ "data_type": "fp"
391
+ },
392
+ "model.layers.14.mlp.shared_experts.down_proj": {
393
+ "act_data_type": "fp",
394
+ "bits": 16,
395
+ "data_type": "fp"
396
+ },
397
+ "model.layers.14.mlp.shared_experts.gate_proj": {
398
+ "act_data_type": "fp",
399
+ "bits": 16,
400
+ "data_type": "fp"
401
+ },
402
+ "model.layers.14.mlp.shared_experts.up_proj": {
403
+ "act_data_type": "fp",
404
+ "bits": 16,
405
+ "data_type": "fp"
406
+ },
407
+ "model.layers.14.self_attn.b_proj": {
408
+ "act_data_type": "fp",
409
+ "bits": 16,
410
+ "data_type": "fp"
411
+ },
412
+ "model.layers.14.self_attn.f_a_proj": {
413
+ "act_data_type": "fp",
414
+ "bits": 16,
415
+ "data_type": "fp"
416
+ },
417
+ "model.layers.14.self_attn.f_b_proj": {
418
+ "act_data_type": "fp",
419
+ "bits": 16,
420
+ "data_type": "fp"
421
+ },
422
+ "model.layers.14.self_attn.g_a_proj": {
423
+ "act_data_type": "fp",
424
+ "bits": 16,
425
+ "data_type": "fp"
426
+ },
427
+ "model.layers.14.self_attn.g_b_proj": {
428
+ "act_data_type": "fp",
429
+ "bits": 16,
430
+ "data_type": "fp"
431
+ },
432
+ "model.layers.14.self_attn.k_proj": {
433
+ "act_data_type": "fp",
434
+ "bits": 16,
435
+ "data_type": "fp"
436
+ },
437
+ "model.layers.14.self_attn.o_proj": {
438
+ "act_data_type": "fp",
439
+ "bits": 16,
440
+ "data_type": "fp"
441
+ },
442
+ "model.layers.14.self_attn.q_proj": {
443
+ "act_data_type": "fp",
444
+ "bits": 16,
445
+ "data_type": "fp"
446
+ },
447
+ "model.layers.14.self_attn.v_proj": {
448
+ "act_data_type": "fp",
449
+ "bits": 16,
450
+ "data_type": "fp"
451
+ },
452
+ "model.layers.15.mlp.shared_experts.down_proj": {
453
+ "act_data_type": "fp",
454
+ "bits": 16,
455
+ "data_type": "fp"
456
+ },
457
+ "model.layers.15.mlp.shared_experts.gate_proj": {
458
+ "act_data_type": "fp",
459
+ "bits": 16,
460
+ "data_type": "fp"
461
+ },
462
+ "model.layers.15.mlp.shared_experts.up_proj": {
463
+ "act_data_type": "fp",
464
+ "bits": 16,
465
+ "data_type": "fp"
466
+ },
467
+ "model.layers.15.self_attn.b_proj": {
468
+ "act_data_type": "fp",
469
+ "bits": 16,
470
+ "data_type": "fp"
471
+ },
472
+ "model.layers.15.self_attn.f_a_proj": {
473
+ "act_data_type": "fp",
474
+ "bits": 16,
475
+ "data_type": "fp"
476
+ },
477
+ "model.layers.15.self_attn.f_b_proj": {
478
+ "act_data_type": "fp",
479
+ "bits": 16,
480
+ "data_type": "fp"
481
+ },
482
+ "model.layers.15.self_attn.g_a_proj": {
483
+ "act_data_type": "fp",
484
+ "bits": 16,
485
+ "data_type": "fp"
486
+ },
487
+ "model.layers.15.self_attn.g_b_proj": {
488
+ "act_data_type": "fp",
489
+ "bits": 16,
490
+ "data_type": "fp"
491
+ },
492
+ "model.layers.15.self_attn.k_proj": {
493
+ "act_data_type": "fp",
494
+ "bits": 16,
495
+ "data_type": "fp"
496
+ },
497
+ "model.layers.15.self_attn.o_proj": {
498
+ "act_data_type": "fp",
499
+ "bits": 16,
500
+ "data_type": "fp"
501
+ },
502
+ "model.layers.15.self_attn.q_proj": {
503
+ "act_data_type": "fp",
504
+ "bits": 16,
505
+ "data_type": "fp"
506
+ },
507
+ "model.layers.15.self_attn.v_proj": {
508
+ "act_data_type": "fp",
509
+ "bits": 16,
510
+ "data_type": "fp"
511
+ },
512
+ "model.layers.16.mlp.shared_experts.down_proj": {
513
+ "act_data_type": "fp",
514
+ "bits": 16,
515
+ "data_type": "fp"
516
+ },
517
+ "model.layers.16.mlp.shared_experts.gate_proj": {
518
+ "act_data_type": "fp",
519
+ "bits": 16,
520
+ "data_type": "fp"
521
+ },
522
+ "model.layers.16.mlp.shared_experts.up_proj": {
523
+ "act_data_type": "fp",
524
+ "bits": 16,
525
+ "data_type": "fp"
526
+ },
527
+ "model.layers.16.self_attn.g_proj": {
528
+ "act_data_type": "fp",
529
+ "bits": 16,
530
+ "data_type": "fp"
531
+ },
532
+ "model.layers.16.self_attn.k_proj": {
533
+ "act_data_type": "fp",
534
+ "bits": 16,
535
+ "data_type": "fp"
536
+ },
537
+ "model.layers.16.self_attn.o_proj": {
538
+ "act_data_type": "fp",
539
+ "bits": 16,
540
+ "data_type": "fp"
541
+ },
542
+ "model.layers.16.self_attn.q_proj": {
543
+ "act_data_type": "fp",
544
+ "bits": 16,
545
+ "data_type": "fp"
546
+ },
547
+ "model.layers.16.self_attn.v_proj": {
548
+ "act_data_type": "fp",
549
+ "bits": 16,
550
+ "data_type": "fp"
551
+ },
552
+ "model.layers.17.mlp.shared_experts.down_proj": {
553
+ "act_data_type": "fp",
554
+ "bits": 16,
555
+ "data_type": "fp"
556
+ },
557
+ "model.layers.17.mlp.shared_experts.gate_proj": {
558
+ "act_data_type": "fp",
559
+ "bits": 16,
560
+ "data_type": "fp"
561
+ },
562
+ "model.layers.17.mlp.shared_experts.up_proj": {
563
+ "act_data_type": "fp",
564
+ "bits": 16,
565
+ "data_type": "fp"
566
+ },
567
+ "model.layers.17.self_attn.b_proj": {
568
+ "act_data_type": "fp",
569
+ "bits": 16,
570
+ "data_type": "fp"
571
+ },
572
+ "model.layers.17.self_attn.f_a_proj": {
573
+ "act_data_type": "fp",
574
+ "bits": 16,
575
+ "data_type": "fp"
576
+ },
577
+ "model.layers.17.self_attn.f_b_proj": {
578
+ "act_data_type": "fp",
579
+ "bits": 16,
580
+ "data_type": "fp"
581
+ },
582
+ "model.layers.17.self_attn.g_a_proj": {
583
+ "act_data_type": "fp",
584
+ "bits": 16,
585
+ "data_type": "fp"
586
+ },
587
+ "model.layers.17.self_attn.g_b_proj": {
588
+ "act_data_type": "fp",
589
+ "bits": 16,
590
+ "data_type": "fp"
591
+ },
592
+ "model.layers.17.self_attn.k_proj": {
593
+ "act_data_type": "fp",
594
+ "bits": 16,
595
+ "data_type": "fp"
596
+ },
597
+ "model.layers.17.self_attn.o_proj": {
598
+ "act_data_type": "fp",
599
+ "bits": 16,
600
+ "data_type": "fp"
601
+ },
602
+ "model.layers.17.self_attn.q_proj": {
603
+ "act_data_type": "fp",
604
+ "bits": 16,
605
+ "data_type": "fp"
606
+ },
607
+ "model.layers.17.self_attn.v_proj": {
608
+ "act_data_type": "fp",
609
+ "bits": 16,
610
+ "data_type": "fp"
611
+ },
612
+ "model.layers.18.mlp.shared_experts.down_proj": {
613
+ "act_data_type": "fp",
614
+ "bits": 16,
615
+ "data_type": "fp"
616
+ },
617
+ "model.layers.18.mlp.shared_experts.gate_proj": {
618
+ "act_data_type": "fp",
619
+ "bits": 16,
620
+ "data_type": "fp"
621
+ },
622
+ "model.layers.18.mlp.shared_experts.up_proj": {
623
+ "act_data_type": "fp",
624
+ "bits": 16,
625
+ "data_type": "fp"
626
+ },
627
+ "model.layers.18.self_attn.b_proj": {
628
+ "act_data_type": "fp",
629
+ "bits": 16,
630
+ "data_type": "fp"
631
+ },
632
+ "model.layers.18.self_attn.f_a_proj": {
633
+ "act_data_type": "fp",
634
+ "bits": 16,
635
+ "data_type": "fp"
636
+ },
637
+ "model.layers.18.self_attn.f_b_proj": {
638
+ "act_data_type": "fp",
639
+ "bits": 16,
640
+ "data_type": "fp"
641
+ },
642
+ "model.layers.18.self_attn.g_a_proj": {
643
+ "act_data_type": "fp",
644
+ "bits": 16,
645
+ "data_type": "fp"
646
+ },
647
+ "model.layers.18.self_attn.g_b_proj": {
648
+ "act_data_type": "fp",
649
+ "bits": 16,
650
+ "data_type": "fp"
651
+ },
652
+ "model.layers.18.self_attn.k_proj": {
653
+ "act_data_type": "fp",
654
+ "bits": 16,
655
+ "data_type": "fp"
656
+ },
657
+ "model.layers.18.self_attn.o_proj": {
658
+ "act_data_type": "fp",
659
+ "bits": 16,
660
+ "data_type": "fp"
661
+ },
662
+ "model.layers.18.self_attn.q_proj": {
663
+ "act_data_type": "fp",
664
+ "bits": 16,
665
+ "data_type": "fp"
666
+ },
667
+ "model.layers.18.self_attn.v_proj": {
668
+ "act_data_type": "fp",
669
+ "bits": 16,
670
+ "data_type": "fp"
671
+ },
672
+ "model.layers.19.mlp.shared_experts.down_proj": {
673
+ "act_data_type": "fp",
674
+ "bits": 16,
675
+ "data_type": "fp"
676
+ },
677
+ "model.layers.19.mlp.shared_experts.gate_proj": {
678
+ "act_data_type": "fp",
679
+ "bits": 16,
680
+ "data_type": "fp"
681
+ },
682
+ "model.layers.19.mlp.shared_experts.up_proj": {
683
+ "act_data_type": "fp",
684
+ "bits": 16,
685
+ "data_type": "fp"
686
+ },
687
+ "model.layers.19.self_attn.b_proj": {
688
+ "act_data_type": "fp",
689
+ "bits": 16,
690
+ "data_type": "fp"
691
+ },
692
+ "model.layers.19.self_attn.f_a_proj": {
693
+ "act_data_type": "fp",
694
+ "bits": 16,
695
+ "data_type": "fp"
696
+ },
697
+ "model.layers.19.self_attn.f_b_proj": {
698
+ "act_data_type": "fp",
699
+ "bits": 16,
700
+ "data_type": "fp"
701
+ },
702
+ "model.layers.19.self_attn.g_a_proj": {
703
+ "act_data_type": "fp",
704
+ "bits": 16,
705
+ "data_type": "fp"
706
+ },
707
+ "model.layers.19.self_attn.g_b_proj": {
708
+ "act_data_type": "fp",
709
+ "bits": 16,
710
+ "data_type": "fp"
711
+ },
712
+ "model.layers.19.self_attn.k_proj": {
713
+ "act_data_type": "fp",
714
+ "bits": 16,
715
+ "data_type": "fp"
716
+ },
717
+ "model.layers.19.self_attn.o_proj": {
718
+ "act_data_type": "fp",
719
+ "bits": 16,
720
+ "data_type": "fp"
721
+ },
722
+ "model.layers.19.self_attn.q_proj": {
723
+ "act_data_type": "fp",
724
+ "bits": 16,
725
+ "data_type": "fp"
726
+ },
727
+ "model.layers.19.self_attn.v_proj": {
728
+ "act_data_type": "fp",
729
+ "bits": 16,
730
+ "data_type": "fp"
731
+ },
732
+ "model.layers.2.mlp.shared_experts.down_proj": {
733
+ "act_data_type": "fp",
734
+ "bits": 16,
735
+ "data_type": "fp"
736
+ },
737
+ "model.layers.2.mlp.shared_experts.gate_proj": {
738
+ "act_data_type": "fp",
739
+ "bits": 16,
740
+ "data_type": "fp"
741
+ },
742
+ "model.layers.2.mlp.shared_experts.up_proj": {
743
+ "act_data_type": "fp",
744
+ "bits": 16,
745
+ "data_type": "fp"
746
+ },
747
+ "model.layers.2.self_attn.b_proj": {
748
+ "act_data_type": "fp",
749
+ "bits": 16,
750
+ "data_type": "fp"
751
+ },
752
+ "model.layers.2.self_attn.f_a_proj": {
753
+ "act_data_type": "fp",
754
+ "bits": 16,
755
+ "data_type": "fp"
756
+ },
757
+ "model.layers.2.self_attn.f_b_proj": {
758
+ "act_data_type": "fp",
759
+ "bits": 16,
760
+ "data_type": "fp"
761
+ },
762
+ "model.layers.2.self_attn.g_a_proj": {
763
+ "act_data_type": "fp",
764
+ "bits": 16,
765
+ "data_type": "fp"
766
+ },
767
+ "model.layers.2.self_attn.g_b_proj": {
768
+ "act_data_type": "fp",
769
+ "bits": 16,
770
+ "data_type": "fp"
771
+ },
772
+ "model.layers.2.self_attn.k_proj": {
773
+ "act_data_type": "fp",
774
+ "bits": 16,
775
+ "data_type": "fp"
776
+ },
777
+ "model.layers.2.self_attn.o_proj": {
778
+ "act_data_type": "fp",
779
+ "bits": 16,
780
+ "data_type": "fp"
781
+ },
782
+ "model.layers.2.self_attn.q_proj": {
783
+ "act_data_type": "fp",
784
+ "bits": 16,
785
+ "data_type": "fp"
786
+ },
787
+ "model.layers.2.self_attn.v_proj": {
788
+ "act_data_type": "fp",
789
+ "bits": 16,
790
+ "data_type": "fp"
791
+ },
792
+ "model.layers.20.mlp.shared_experts.down_proj": {
793
+ "act_data_type": "fp",
794
+ "bits": 16,
795
+ "data_type": "fp"
796
+ },
797
+ "model.layers.20.mlp.shared_experts.gate_proj": {
798
+ "act_data_type": "fp",
799
+ "bits": 16,
800
+ "data_type": "fp"
801
+ },
802
+ "model.layers.20.mlp.shared_experts.up_proj": {
803
+ "act_data_type": "fp",
804
+ "bits": 16,
805
+ "data_type": "fp"
806
+ },
807
+ "model.layers.20.self_attn.g_proj": {
808
+ "act_data_type": "fp",
809
+ "bits": 16,
810
+ "data_type": "fp"
811
+ },
812
+ "model.layers.20.self_attn.k_proj": {
813
+ "act_data_type": "fp",
814
+ "bits": 16,
815
+ "data_type": "fp"
816
+ },
817
+ "model.layers.20.self_attn.o_proj": {
818
+ "act_data_type": "fp",
819
+ "bits": 16,
820
+ "data_type": "fp"
821
+ },
822
+ "model.layers.20.self_attn.q_proj": {
823
+ "act_data_type": "fp",
824
+ "bits": 16,
825
+ "data_type": "fp"
826
+ },
827
+ "model.layers.20.self_attn.v_proj": {
828
+ "act_data_type": "fp",
829
+ "bits": 16,
830
+ "data_type": "fp"
831
+ },
832
+ "model.layers.21.mlp.shared_experts.down_proj": {
833
+ "act_data_type": "fp",
834
+ "bits": 16,
835
+ "data_type": "fp"
836
+ },
837
+ "model.layers.21.mlp.shared_experts.gate_proj": {
838
+ "act_data_type": "fp",
839
+ "bits": 16,
840
+ "data_type": "fp"
841
+ },
842
+ "model.layers.21.mlp.shared_experts.up_proj": {
843
+ "act_data_type": "fp",
844
+ "bits": 16,
845
+ "data_type": "fp"
846
+ },
847
+ "model.layers.21.self_attn.b_proj": {
848
+ "act_data_type": "fp",
849
+ "bits": 16,
850
+ "data_type": "fp"
851
+ },
852
+ "model.layers.21.self_attn.f_a_proj": {
853
+ "act_data_type": "fp",
854
+ "bits": 16,
855
+ "data_type": "fp"
856
+ },
857
+ "model.layers.21.self_attn.f_b_proj": {
858
+ "act_data_type": "fp",
859
+ "bits": 16,
860
+ "data_type": "fp"
861
+ },
862
+ "model.layers.21.self_attn.g_a_proj": {
863
+ "act_data_type": "fp",
864
+ "bits": 16,
865
+ "data_type": "fp"
866
+ },
867
+ "model.layers.21.self_attn.g_b_proj": {
868
+ "act_data_type": "fp",
869
+ "bits": 16,
870
+ "data_type": "fp"
871
+ },
872
+ "model.layers.21.self_attn.k_proj": {
873
+ "act_data_type": "fp",
874
+ "bits": 16,
875
+ "data_type": "fp"
876
+ },
877
+ "model.layers.21.self_attn.o_proj": {
878
+ "act_data_type": "fp",
879
+ "bits": 16,
880
+ "data_type": "fp"
881
+ },
882
+ "model.layers.21.self_attn.q_proj": {
883
+ "act_data_type": "fp",
884
+ "bits": 16,
885
+ "data_type": "fp"
886
+ },
887
+ "model.layers.21.self_attn.v_proj": {
888
+ "act_data_type": "fp",
889
+ "bits": 16,
890
+ "data_type": "fp"
891
+ },
892
+ "model.layers.22.mlp.shared_experts.down_proj": {
893
+ "act_data_type": "fp",
894
+ "bits": 16,
895
+ "data_type": "fp"
896
+ },
897
+ "model.layers.22.mlp.shared_experts.gate_proj": {
898
+ "act_data_type": "fp",
899
+ "bits": 16,
900
+ "data_type": "fp"
901
+ },
902
+ "model.layers.22.mlp.shared_experts.up_proj": {
903
+ "act_data_type": "fp",
904
+ "bits": 16,
905
+ "data_type": "fp"
906
+ },
907
+ "model.layers.22.self_attn.b_proj": {
908
+ "act_data_type": "fp",
909
+ "bits": 16,
910
+ "data_type": "fp"
911
+ },
912
+ "model.layers.22.self_attn.f_a_proj": {
913
+ "act_data_type": "fp",
914
+ "bits": 16,
915
+ "data_type": "fp"
916
+ },
917
+ "model.layers.22.self_attn.f_b_proj": {
918
+ "act_data_type": "fp",
919
+ "bits": 16,
920
+ "data_type": "fp"
921
+ },
922
+ "model.layers.22.self_attn.g_a_proj": {
923
+ "act_data_type": "fp",
924
+ "bits": 16,
925
+ "data_type": "fp"
926
+ },
927
+ "model.layers.22.self_attn.g_b_proj": {
928
+ "act_data_type": "fp",
929
+ "bits": 16,
930
+ "data_type": "fp"
931
+ },
932
+ "model.layers.22.self_attn.k_proj": {
933
+ "act_data_type": "fp",
934
+ "bits": 16,
935
+ "data_type": "fp"
936
+ },
937
+ "model.layers.22.self_attn.o_proj": {
938
+ "act_data_type": "fp",
939
+ "bits": 16,
940
+ "data_type": "fp"
941
+ },
942
+ "model.layers.22.self_attn.q_proj": {
943
+ "act_data_type": "fp",
944
+ "bits": 16,
945
+ "data_type": "fp"
946
+ },
947
+ "model.layers.22.self_attn.v_proj": {
948
+ "act_data_type": "fp",
949
+ "bits": 16,
950
+ "data_type": "fp"
951
+ },
952
+ "model.layers.23.mlp.shared_experts.down_proj": {
953
+ "act_data_type": "fp",
954
+ "bits": 16,
955
+ "data_type": "fp"
956
+ },
957
+ "model.layers.23.mlp.shared_experts.gate_proj": {
958
+ "act_data_type": "fp",
959
+ "bits": 16,
960
+ "data_type": "fp"
961
+ },
962
+ "model.layers.23.mlp.shared_experts.up_proj": {
963
+ "act_data_type": "fp",
964
+ "bits": 16,
965
+ "data_type": "fp"
966
+ },
967
+ "model.layers.23.self_attn.b_proj": {
968
+ "act_data_type": "fp",
969
+ "bits": 16,
970
+ "data_type": "fp"
971
+ },
972
+ "model.layers.23.self_attn.f_a_proj": {
973
+ "act_data_type": "fp",
974
+ "bits": 16,
975
+ "data_type": "fp"
976
+ },
977
+ "model.layers.23.self_attn.f_b_proj": {
978
+ "act_data_type": "fp",
979
+ "bits": 16,
980
+ "data_type": "fp"
981
+ },
982
+ "model.layers.23.self_attn.g_a_proj": {
983
+ "act_data_type": "fp",
984
+ "bits": 16,
985
+ "data_type": "fp"
986
+ },
987
+ "model.layers.23.self_attn.g_b_proj": {
988
+ "act_data_type": "fp",
989
+ "bits": 16,
990
+ "data_type": "fp"
991
+ },
992
+ "model.layers.23.self_attn.k_proj": {
993
+ "act_data_type": "fp",
994
+ "bits": 16,
995
+ "data_type": "fp"
996
+ },
997
+ "model.layers.23.self_attn.o_proj": {
998
+ "act_data_type": "fp",
999
+ "bits": 16,
1000
+ "data_type": "fp"
1001
+ },
1002
+ "model.layers.23.self_attn.q_proj": {
1003
+ "act_data_type": "fp",
1004
+ "bits": 16,
1005
+ "data_type": "fp"
1006
+ },
1007
+ "model.layers.23.self_attn.v_proj": {
1008
+ "act_data_type": "fp",
1009
+ "bits": 16,
1010
+ "data_type": "fp"
1011
+ },
1012
+ "model.layers.24.mlp.shared_experts.down_proj": {
1013
+ "act_data_type": "fp",
1014
+ "bits": 16,
1015
+ "data_type": "fp"
1016
+ },
1017
+ "model.layers.24.mlp.shared_experts.gate_proj": {
1018
+ "act_data_type": "fp",
1019
+ "bits": 16,
1020
+ "data_type": "fp"
1021
+ },
1022
+ "model.layers.24.mlp.shared_experts.up_proj": {
1023
+ "act_data_type": "fp",
1024
+ "bits": 16,
1025
+ "data_type": "fp"
1026
+ },
1027
+ "model.layers.24.self_attn.g_proj": {
1028
+ "act_data_type": "fp",
1029
+ "bits": 16,
1030
+ "data_type": "fp"
1031
+ },
1032
+ "model.layers.24.self_attn.k_proj": {
1033
+ "act_data_type": "fp",
1034
+ "bits": 16,
1035
+ "data_type": "fp"
1036
+ },
1037
+ "model.layers.24.self_attn.o_proj": {
1038
+ "act_data_type": "fp",
1039
+ "bits": 16,
1040
+ "data_type": "fp"
1041
+ },
1042
+ "model.layers.24.self_attn.q_proj": {
1043
+ "act_data_type": "fp",
1044
+ "bits": 16,
1045
+ "data_type": "fp"
1046
+ },
1047
+ "model.layers.24.self_attn.v_proj": {
1048
+ "act_data_type": "fp",
1049
+ "bits": 16,
1050
+ "data_type": "fp"
1051
+ },
1052
+ "model.layers.25.mlp.shared_experts.down_proj": {
1053
+ "act_data_type": "fp",
1054
+ "bits": 16,
1055
+ "data_type": "fp"
1056
+ },
1057
+ "model.layers.25.mlp.shared_experts.gate_proj": {
1058
+ "act_data_type": "fp",
1059
+ "bits": 16,
1060
+ "data_type": "fp"
1061
+ },
1062
+ "model.layers.25.mlp.shared_experts.up_proj": {
1063
+ "act_data_type": "fp",
1064
+ "bits": 16,
1065
+ "data_type": "fp"
1066
+ },
1067
+ "model.layers.25.self_attn.b_proj": {
1068
+ "act_data_type": "fp",
1069
+ "bits": 16,
1070
+ "data_type": "fp"
1071
+ },
1072
+ "model.layers.25.self_attn.f_a_proj": {
1073
+ "act_data_type": "fp",
1074
+ "bits": 16,
1075
+ "data_type": "fp"
1076
+ },
1077
+ "model.layers.25.self_attn.f_b_proj": {
1078
+ "act_data_type": "fp",
1079
+ "bits": 16,
1080
+ "data_type": "fp"
1081
+ },
1082
+ "model.layers.25.self_attn.g_a_proj": {
1083
+ "act_data_type": "fp",
1084
+ "bits": 16,
1085
+ "data_type": "fp"
1086
+ },
1087
+ "model.layers.25.self_attn.g_b_proj": {
1088
+ "act_data_type": "fp",
1089
+ "bits": 16,
1090
+ "data_type": "fp"
1091
+ },
1092
+ "model.layers.25.self_attn.k_proj": {
1093
+ "act_data_type": "fp",
1094
+ "bits": 16,
1095
+ "data_type": "fp"
1096
+ },
1097
+ "model.layers.25.self_attn.o_proj": {
1098
+ "act_data_type": "fp",
1099
+ "bits": 16,
1100
+ "data_type": "fp"
1101
+ },
1102
+ "model.layers.25.self_attn.q_proj": {
1103
+ "act_data_type": "fp",
1104
+ "bits": 16,
1105
+ "data_type": "fp"
1106
+ },
1107
+ "model.layers.25.self_attn.v_proj": {
1108
+ "act_data_type": "fp",
1109
+ "bits": 16,
1110
+ "data_type": "fp"
1111
+ },
1112
+ "model.layers.26.mlp.shared_experts.down_proj": {
1113
+ "act_data_type": "fp",
1114
+ "bits": 16,
1115
+ "data_type": "fp"
1116
+ },
1117
+ "model.layers.26.mlp.shared_experts.gate_proj": {
1118
+ "act_data_type": "fp",
1119
+ "bits": 16,
1120
+ "data_type": "fp"
1121
+ },
1122
+ "model.layers.26.mlp.shared_experts.up_proj": {
1123
+ "act_data_type": "fp",
1124
+ "bits": 16,
1125
+ "data_type": "fp"
1126
+ },
1127
+ "model.layers.26.self_attn.b_proj": {
1128
+ "act_data_type": "fp",
1129
+ "bits": 16,
1130
+ "data_type": "fp"
1131
+ },
1132
+ "model.layers.26.self_attn.f_a_proj": {
1133
+ "act_data_type": "fp",
1134
+ "bits": 16,
1135
+ "data_type": "fp"
1136
+ },
1137
+ "model.layers.26.self_attn.f_b_proj": {
1138
+ "act_data_type": "fp",
1139
+ "bits": 16,
1140
+ "data_type": "fp"
1141
+ },
1142
+ "model.layers.26.self_attn.g_a_proj": {
1143
+ "act_data_type": "fp",
1144
+ "bits": 16,
1145
+ "data_type": "fp"
1146
+ },
1147
+ "model.layers.26.self_attn.g_b_proj": {
1148
+ "act_data_type": "fp",
1149
+ "bits": 16,
1150
+ "data_type": "fp"
1151
+ },
1152
+ "model.layers.26.self_attn.k_proj": {
1153
+ "act_data_type": "fp",
1154
+ "bits": 16,
1155
+ "data_type": "fp"
1156
+ },
1157
+ "model.layers.26.self_attn.o_proj": {
1158
+ "act_data_type": "fp",
1159
+ "bits": 16,
1160
+ "data_type": "fp"
1161
+ },
1162
+ "model.layers.26.self_attn.q_proj": {
1163
+ "act_data_type": "fp",
1164
+ "bits": 16,
1165
+ "data_type": "fp"
1166
+ },
1167
+ "model.layers.26.self_attn.v_proj": {
1168
+ "act_data_type": "fp",
1169
+ "bits": 16,
1170
+ "data_type": "fp"
1171
+ },
1172
+ "model.layers.27.mlp.shared_experts.down_proj": {
1173
+ "act_data_type": "fp",
1174
+ "bits": 16,
1175
+ "data_type": "fp"
1176
+ },
1177
+ "model.layers.27.mlp.shared_experts.gate_proj": {
1178
+ "act_data_type": "fp",
1179
+ "bits": 16,
1180
+ "data_type": "fp"
1181
+ },
1182
+ "model.layers.27.mlp.shared_experts.up_proj": {
1183
+ "act_data_type": "fp",
1184
+ "bits": 16,
1185
+ "data_type": "fp"
1186
+ },
1187
+ "model.layers.27.self_attn.b_proj": {
1188
+ "act_data_type": "fp",
1189
+ "bits": 16,
1190
+ "data_type": "fp"
1191
+ },
1192
+ "model.layers.27.self_attn.f_a_proj": {
1193
+ "act_data_type": "fp",
1194
+ "bits": 16,
1195
+ "data_type": "fp"
1196
+ },
1197
+ "model.layers.27.self_attn.f_b_proj": {
1198
+ "act_data_type": "fp",
1199
+ "bits": 16,
1200
+ "data_type": "fp"
1201
+ },
1202
+ "model.layers.27.self_attn.g_a_proj": {
1203
+ "act_data_type": "fp",
1204
+ "bits": 16,
1205
+ "data_type": "fp"
1206
+ },
1207
+ "model.layers.27.self_attn.g_b_proj": {
1208
+ "act_data_type": "fp",
1209
+ "bits": 16,
1210
+ "data_type": "fp"
1211
+ },
1212
+ "model.layers.27.self_attn.k_proj": {
1213
+ "act_data_type": "fp",
1214
+ "bits": 16,
1215
+ "data_type": "fp"
1216
+ },
1217
+ "model.layers.27.self_attn.o_proj": {
1218
+ "act_data_type": "fp",
1219
+ "bits": 16,
1220
+ "data_type": "fp"
1221
+ },
1222
+ "model.layers.27.self_attn.q_proj": {
1223
+ "act_data_type": "fp",
1224
+ "bits": 16,
1225
+ "data_type": "fp"
1226
+ },
1227
+ "model.layers.27.self_attn.v_proj": {
1228
+ "act_data_type": "fp",
1229
+ "bits": 16,
1230
+ "data_type": "fp"
1231
+ },
1232
+ "model.layers.28.mlp.shared_experts.down_proj": {
1233
+ "act_data_type": "fp",
1234
+ "bits": 16,
1235
+ "data_type": "fp"
1236
+ },
1237
+ "model.layers.28.mlp.shared_experts.gate_proj": {
1238
+ "act_data_type": "fp",
1239
+ "bits": 16,
1240
+ "data_type": "fp"
1241
+ },
1242
+ "model.layers.28.mlp.shared_experts.up_proj": {
1243
+ "act_data_type": "fp",
1244
+ "bits": 16,
1245
+ "data_type": "fp"
1246
+ },
1247
+ "model.layers.28.self_attn.g_proj": {
1248
+ "act_data_type": "fp",
1249
+ "bits": 16,
1250
+ "data_type": "fp"
1251
+ },
1252
+ "model.layers.28.self_attn.k_proj": {
1253
+ "act_data_type": "fp",
1254
+ "bits": 16,
1255
+ "data_type": "fp"
1256
+ },
1257
+ "model.layers.28.self_attn.o_proj": {
1258
+ "act_data_type": "fp",
1259
+ "bits": 16,
1260
+ "data_type": "fp"
1261
+ },
1262
+ "model.layers.28.self_attn.q_proj": {
1263
+ "act_data_type": "fp",
1264
+ "bits": 16,
1265
+ "data_type": "fp"
1266
+ },
1267
+ "model.layers.28.self_attn.v_proj": {
1268
+ "act_data_type": "fp",
1269
+ "bits": 16,
1270
+ "data_type": "fp"
1271
+ },
1272
+ "model.layers.29.mlp.shared_experts.down_proj": {
1273
+ "act_data_type": "fp",
1274
+ "bits": 16,
1275
+ "data_type": "fp"
1276
+ },
1277
+ "model.layers.29.mlp.shared_experts.gate_proj": {
1278
+ "act_data_type": "fp",
1279
+ "bits": 16,
1280
+ "data_type": "fp"
1281
+ },
1282
+ "model.layers.29.mlp.shared_experts.up_proj": {
1283
+ "act_data_type": "fp",
1284
+ "bits": 16,
1285
+ "data_type": "fp"
1286
+ },
1287
+ "model.layers.29.self_attn.b_proj": {
1288
+ "act_data_type": "fp",
1289
+ "bits": 16,
1290
+ "data_type": "fp"
1291
+ },
1292
+ "model.layers.29.self_attn.f_a_proj": {
1293
+ "act_data_type": "fp",
1294
+ "bits": 16,
1295
+ "data_type": "fp"
1296
+ },
1297
+ "model.layers.29.self_attn.f_b_proj": {
1298
+ "act_data_type": "fp",
1299
+ "bits": 16,
1300
+ "data_type": "fp"
1301
+ },
1302
+ "model.layers.29.self_attn.g_a_proj": {
1303
+ "act_data_type": "fp",
1304
+ "bits": 16,
1305
+ "data_type": "fp"
1306
+ },
1307
+ "model.layers.29.self_attn.g_b_proj": {
1308
+ "act_data_type": "fp",
1309
+ "bits": 16,
1310
+ "data_type": "fp"
1311
+ },
1312
+ "model.layers.29.self_attn.k_proj": {
1313
+ "act_data_type": "fp",
1314
+ "bits": 16,
1315
+ "data_type": "fp"
1316
+ },
1317
+ "model.layers.29.self_attn.o_proj": {
1318
+ "act_data_type": "fp",
1319
+ "bits": 16,
1320
+ "data_type": "fp"
1321
+ },
1322
+ "model.layers.29.self_attn.q_proj": {
1323
+ "act_data_type": "fp",
1324
+ "bits": 16,
1325
+ "data_type": "fp"
1326
+ },
1327
+ "model.layers.29.self_attn.v_proj": {
1328
+ "act_data_type": "fp",
1329
+ "bits": 16,
1330
+ "data_type": "fp"
1331
+ },
1332
+ "model.layers.3.mlp.shared_experts.down_proj": {
1333
+ "act_data_type": "fp",
1334
+ "bits": 16,
1335
+ "data_type": "fp"
1336
+ },
1337
+ "model.layers.3.mlp.shared_experts.gate_proj": {
1338
+ "act_data_type": "fp",
1339
+ "bits": 16,
1340
+ "data_type": "fp"
1341
+ },
1342
+ "model.layers.3.mlp.shared_experts.up_proj": {
1343
+ "act_data_type": "fp",
1344
+ "bits": 16,
1345
+ "data_type": "fp"
1346
+ },
1347
+ "model.layers.3.self_attn.b_proj": {
1348
+ "act_data_type": "fp",
1349
+ "bits": 16,
1350
+ "data_type": "fp"
1351
+ },
1352
+ "model.layers.3.self_attn.f_a_proj": {
1353
+ "act_data_type": "fp",
1354
+ "bits": 16,
1355
+ "data_type": "fp"
1356
+ },
1357
+ "model.layers.3.self_attn.f_b_proj": {
1358
+ "act_data_type": "fp",
1359
+ "bits": 16,
1360
+ "data_type": "fp"
1361
+ },
1362
+ "model.layers.3.self_attn.g_a_proj": {
1363
+ "act_data_type": "fp",
1364
+ "bits": 16,
1365
+ "data_type": "fp"
1366
+ },
1367
+ "model.layers.3.self_attn.g_b_proj": {
1368
+ "act_data_type": "fp",
1369
+ "bits": 16,
1370
+ "data_type": "fp"
1371
+ },
1372
+ "model.layers.3.self_attn.k_proj": {
1373
+ "act_data_type": "fp",
1374
+ "bits": 16,
1375
+ "data_type": "fp"
1376
+ },
1377
+ "model.layers.3.self_attn.o_proj": {
1378
+ "act_data_type": "fp",
1379
+ "bits": 16,
1380
+ "data_type": "fp"
1381
+ },
1382
+ "model.layers.3.self_attn.q_proj": {
1383
+ "act_data_type": "fp",
1384
+ "bits": 16,
1385
+ "data_type": "fp"
1386
+ },
1387
+ "model.layers.3.self_attn.v_proj": {
1388
+ "act_data_type": "fp",
1389
+ "bits": 16,
1390
+ "data_type": "fp"
1391
+ },
1392
+ "model.layers.30.mlp.shared_experts.down_proj": {
1393
+ "act_data_type": "fp",
1394
+ "bits": 16,
1395
+ "data_type": "fp"
1396
+ },
1397
+ "model.layers.30.mlp.shared_experts.gate_proj": {
1398
+ "act_data_type": "fp",
1399
+ "bits": 16,
1400
+ "data_type": "fp"
1401
+ },
1402
+ "model.layers.30.mlp.shared_experts.up_proj": {
1403
+ "act_data_type": "fp",
1404
+ "bits": 16,
1405
+ "data_type": "fp"
1406
+ },
1407
+ "model.layers.30.self_attn.b_proj": {
1408
+ "act_data_type": "fp",
1409
+ "bits": 16,
1410
+ "data_type": "fp"
1411
+ },
1412
+ "model.layers.30.self_attn.f_a_proj": {
1413
+ "act_data_type": "fp",
1414
+ "bits": 16,
1415
+ "data_type": "fp"
1416
+ },
1417
+ "model.layers.30.self_attn.f_b_proj": {
1418
+ "act_data_type": "fp",
1419
+ "bits": 16,
1420
+ "data_type": "fp"
1421
+ },
1422
+ "model.layers.30.self_attn.g_a_proj": {
1423
+ "act_data_type": "fp",
1424
+ "bits": 16,
1425
+ "data_type": "fp"
1426
+ },
1427
+ "model.layers.30.self_attn.g_b_proj": {
1428
+ "act_data_type": "fp",
1429
+ "bits": 16,
1430
+ "data_type": "fp"
1431
+ },
1432
+ "model.layers.30.self_attn.k_proj": {
1433
+ "act_data_type": "fp",
1434
+ "bits": 16,
1435
+ "data_type": "fp"
1436
+ },
1437
+ "model.layers.30.self_attn.o_proj": {
1438
+ "act_data_type": "fp",
1439
+ "bits": 16,
1440
+ "data_type": "fp"
1441
+ },
1442
+ "model.layers.30.self_attn.q_proj": {
1443
+ "act_data_type": "fp",
1444
+ "bits": 16,
1445
+ "data_type": "fp"
1446
+ },
1447
+ "model.layers.30.self_attn.v_proj": {
1448
+ "act_data_type": "fp",
1449
+ "bits": 16,
1450
+ "data_type": "fp"
1451
+ },
1452
+ "model.layers.31.mlp.shared_experts.down_proj": {
1453
+ "act_data_type": "fp",
1454
+ "bits": 16,
1455
+ "data_type": "fp"
1456
+ },
1457
+ "model.layers.31.mlp.shared_experts.gate_proj": {
1458
+ "act_data_type": "fp",
1459
+ "bits": 16,
1460
+ "data_type": "fp"
1461
+ },
1462
+ "model.layers.31.mlp.shared_experts.up_proj": {
1463
+ "act_data_type": "fp",
1464
+ "bits": 16,
1465
+ "data_type": "fp"
1466
+ },
1467
+ "model.layers.31.self_attn.b_proj": {
1468
+ "act_data_type": "fp",
1469
+ "bits": 16,
1470
+ "data_type": "fp"
1471
+ },
1472
+ "model.layers.31.self_attn.f_a_proj": {
1473
+ "act_data_type": "fp",
1474
+ "bits": 16,
1475
+ "data_type": "fp"
1476
+ },
1477
+ "model.layers.31.self_attn.f_b_proj": {
1478
+ "act_data_type": "fp",
1479
+ "bits": 16,
1480
+ "data_type": "fp"
1481
+ },
1482
+ "model.layers.31.self_attn.g_a_proj": {
1483
+ "act_data_type": "fp",
1484
+ "bits": 16,
1485
+ "data_type": "fp"
1486
+ },
1487
+ "model.layers.31.self_attn.g_b_proj": {
1488
+ "act_data_type": "fp",
1489
+ "bits": 16,
1490
+ "data_type": "fp"
1491
+ },
1492
+ "model.layers.31.self_attn.k_proj": {
1493
+ "act_data_type": "fp",
1494
+ "bits": 16,
1495
+ "data_type": "fp"
1496
+ },
1497
+ "model.layers.31.self_attn.o_proj": {
1498
+ "act_data_type": "fp",
1499
+ "bits": 16,
1500
+ "data_type": "fp"
1501
+ },
1502
+ "model.layers.31.self_attn.q_proj": {
1503
+ "act_data_type": "fp",
1504
+ "bits": 16,
1505
+ "data_type": "fp"
1506
+ },
1507
+ "model.layers.31.self_attn.v_proj": {
1508
+ "act_data_type": "fp",
1509
+ "bits": 16,
1510
+ "data_type": "fp"
1511
+ },
1512
+ "model.layers.32.mlp.shared_experts.down_proj": {
1513
+ "act_data_type": "fp",
1514
+ "bits": 16,
1515
+ "data_type": "fp"
1516
+ },
1517
+ "model.layers.32.mlp.shared_experts.gate_proj": {
1518
+ "act_data_type": "fp",
1519
+ "bits": 16,
1520
+ "data_type": "fp"
1521
+ },
1522
+ "model.layers.32.mlp.shared_experts.up_proj": {
1523
+ "act_data_type": "fp",
1524
+ "bits": 16,
1525
+ "data_type": "fp"
1526
+ },
1527
+ "model.layers.32.self_attn.g_proj": {
1528
+ "act_data_type": "fp",
1529
+ "bits": 16,
1530
+ "data_type": "fp"
1531
+ },
1532
+ "model.layers.32.self_attn.k_proj": {
1533
+ "act_data_type": "fp",
1534
+ "bits": 16,
1535
+ "data_type": "fp"
1536
+ },
1537
+ "model.layers.32.self_attn.o_proj": {
1538
+ "act_data_type": "fp",
1539
+ "bits": 16,
1540
+ "data_type": "fp"
1541
+ },
1542
+ "model.layers.32.self_attn.q_proj": {
1543
+ "act_data_type": "fp",
1544
+ "bits": 16,
1545
+ "data_type": "fp"
1546
+ },
1547
+ "model.layers.32.self_attn.v_proj": {
1548
+ "act_data_type": "fp",
1549
+ "bits": 16,
1550
+ "data_type": "fp"
1551
+ },
1552
+ "model.layers.33.mlp.shared_experts.down_proj": {
1553
+ "act_data_type": "fp",
1554
+ "bits": 16,
1555
+ "data_type": "fp"
1556
+ },
1557
+ "model.layers.33.mlp.shared_experts.gate_proj": {
1558
+ "act_data_type": "fp",
1559
+ "bits": 16,
1560
+ "data_type": "fp"
1561
+ },
1562
+ "model.layers.33.mlp.shared_experts.up_proj": {
1563
+ "act_data_type": "fp",
1564
+ "bits": 16,
1565
+ "data_type": "fp"
1566
+ },
1567
+ "model.layers.33.self_attn.b_proj": {
1568
+ "act_data_type": "fp",
1569
+ "bits": 16,
1570
+ "data_type": "fp"
1571
+ },
1572
+ "model.layers.33.self_attn.f_a_proj": {
1573
+ "act_data_type": "fp",
1574
+ "bits": 16,
1575
+ "data_type": "fp"
1576
+ },
1577
+ "model.layers.33.self_attn.f_b_proj": {
1578
+ "act_data_type": "fp",
1579
+ "bits": 16,
1580
+ "data_type": "fp"
1581
+ },
1582
+ "model.layers.33.self_attn.g_a_proj": {
1583
+ "act_data_type": "fp",
1584
+ "bits": 16,
1585
+ "data_type": "fp"
1586
+ },
1587
+ "model.layers.33.self_attn.g_b_proj": {
1588
+ "act_data_type": "fp",
1589
+ "bits": 16,
1590
+ "data_type": "fp"
1591
+ },
1592
+ "model.layers.33.self_attn.k_proj": {
1593
+ "act_data_type": "fp",
1594
+ "bits": 16,
1595
+ "data_type": "fp"
1596
+ },
1597
+ "model.layers.33.self_attn.o_proj": {
1598
+ "act_data_type": "fp",
1599
+ "bits": 16,
1600
+ "data_type": "fp"
1601
+ },
1602
+ "model.layers.33.self_attn.q_proj": {
1603
+ "act_data_type": "fp",
1604
+ "bits": 16,
1605
+ "data_type": "fp"
1606
+ },
1607
+ "model.layers.33.self_attn.v_proj": {
1608
+ "act_data_type": "fp",
1609
+ "bits": 16,
1610
+ "data_type": "fp"
1611
+ },
1612
+ "model.layers.34.mlp.shared_experts.down_proj": {
1613
+ "act_data_type": "fp",
1614
+ "bits": 16,
1615
+ "data_type": "fp"
1616
+ },
1617
+ "model.layers.34.mlp.shared_experts.gate_proj": {
1618
+ "act_data_type": "fp",
1619
+ "bits": 16,
1620
+ "data_type": "fp"
1621
+ },
1622
+ "model.layers.34.mlp.shared_experts.up_proj": {
1623
+ "act_data_type": "fp",
1624
+ "bits": 16,
1625
+ "data_type": "fp"
1626
+ },
1627
+ "model.layers.34.self_attn.b_proj": {
1628
+ "act_data_type": "fp",
1629
+ "bits": 16,
1630
+ "data_type": "fp"
1631
+ },
1632
+ "model.layers.34.self_attn.f_a_proj": {
1633
+ "act_data_type": "fp",
1634
+ "bits": 16,
1635
+ "data_type": "fp"
1636
+ },
1637
+ "model.layers.34.self_attn.f_b_proj": {
1638
+ "act_data_type": "fp",
1639
+ "bits": 16,
1640
+ "data_type": "fp"
1641
+ },
1642
+ "model.layers.34.self_attn.g_a_proj": {
1643
+ "act_data_type": "fp",
1644
+ "bits": 16,
1645
+ "data_type": "fp"
1646
+ },
1647
+ "model.layers.34.self_attn.g_b_proj": {
1648
+ "act_data_type": "fp",
1649
+ "bits": 16,
1650
+ "data_type": "fp"
1651
+ },
1652
+ "model.layers.34.self_attn.k_proj": {
1653
+ "act_data_type": "fp",
1654
+ "bits": 16,
1655
+ "data_type": "fp"
1656
+ },
1657
+ "model.layers.34.self_attn.o_proj": {
1658
+ "act_data_type": "fp",
1659
+ "bits": 16,
1660
+ "data_type": "fp"
1661
+ },
1662
+ "model.layers.34.self_attn.q_proj": {
1663
+ "act_data_type": "fp",
1664
+ "bits": 16,
1665
+ "data_type": "fp"
1666
+ },
1667
+ "model.layers.34.self_attn.v_proj": {
1668
+ "act_data_type": "fp",
1669
+ "bits": 16,
1670
+ "data_type": "fp"
1671
+ },
1672
+ "model.layers.35.mlp.shared_experts.down_proj": {
1673
+ "act_data_type": "fp",
1674
+ "bits": 16,
1675
+ "data_type": "fp"
1676
+ },
1677
+ "model.layers.35.mlp.shared_experts.gate_proj": {
1678
+ "act_data_type": "fp",
1679
+ "bits": 16,
1680
+ "data_type": "fp"
1681
+ },
1682
+ "model.layers.35.mlp.shared_experts.up_proj": {
1683
+ "act_data_type": "fp",
1684
+ "bits": 16,
1685
+ "data_type": "fp"
1686
+ },
1687
+ "model.layers.35.self_attn.b_proj": {
1688
+ "act_data_type": "fp",
1689
+ "bits": 16,
1690
+ "data_type": "fp"
1691
+ },
1692
+ "model.layers.35.self_attn.f_a_proj": {
1693
+ "act_data_type": "fp",
1694
+ "bits": 16,
1695
+ "data_type": "fp"
1696
+ },
1697
+ "model.layers.35.self_attn.f_b_proj": {
1698
+ "act_data_type": "fp",
1699
+ "bits": 16,
1700
+ "data_type": "fp"
1701
+ },
1702
+ "model.layers.35.self_attn.g_a_proj": {
1703
+ "act_data_type": "fp",
1704
+ "bits": 16,
1705
+ "data_type": "fp"
1706
+ },
1707
+ "model.layers.35.self_attn.g_b_proj": {
1708
+ "act_data_type": "fp",
1709
+ "bits": 16,
1710
+ "data_type": "fp"
1711
+ },
1712
+ "model.layers.35.self_attn.k_proj": {
1713
+ "act_data_type": "fp",
1714
+ "bits": 16,
1715
+ "data_type": "fp"
1716
+ },
1717
+ "model.layers.35.self_attn.o_proj": {
1718
+ "act_data_type": "fp",
1719
+ "bits": 16,
1720
+ "data_type": "fp"
1721
+ },
1722
+ "model.layers.35.self_attn.q_proj": {
1723
+ "act_data_type": "fp",
1724
+ "bits": 16,
1725
+ "data_type": "fp"
1726
+ },
1727
+ "model.layers.35.self_attn.v_proj": {
1728
+ "act_data_type": "fp",
1729
+ "bits": 16,
1730
+ "data_type": "fp"
1731
+ },
1732
+ "model.layers.36.mlp.shared_experts.down_proj": {
1733
+ "act_data_type": "fp",
1734
+ "bits": 16,
1735
+ "data_type": "fp"
1736
+ },
1737
+ "model.layers.36.mlp.shared_experts.gate_proj": {
1738
+ "act_data_type": "fp",
1739
+ "bits": 16,
1740
+ "data_type": "fp"
1741
+ },
1742
+ "model.layers.36.mlp.shared_experts.up_proj": {
1743
+ "act_data_type": "fp",
1744
+ "bits": 16,
1745
+ "data_type": "fp"
1746
+ },
1747
+ "model.layers.36.self_attn.g_proj": {
1748
+ "act_data_type": "fp",
1749
+ "bits": 16,
1750
+ "data_type": "fp"
1751
+ },
1752
+ "model.layers.36.self_attn.k_proj": {
1753
+ "act_data_type": "fp",
1754
+ "bits": 16,
1755
+ "data_type": "fp"
1756
+ },
1757
+ "model.layers.36.self_attn.o_proj": {
1758
+ "act_data_type": "fp",
1759
+ "bits": 16,
1760
+ "data_type": "fp"
1761
+ },
1762
+ "model.layers.36.self_attn.q_proj": {
1763
+ "act_data_type": "fp",
1764
+ "bits": 16,
1765
+ "data_type": "fp"
1766
+ },
1767
+ "model.layers.36.self_attn.v_proj": {
1768
+ "act_data_type": "fp",
1769
+ "bits": 16,
1770
+ "data_type": "fp"
1771
+ },
1772
+ "model.layers.37.mlp.shared_experts.down_proj": {
1773
+ "act_data_type": "fp",
1774
+ "bits": 16,
1775
+ "data_type": "fp"
1776
+ },
1777
+ "model.layers.37.mlp.shared_experts.gate_proj": {
1778
+ "act_data_type": "fp",
1779
+ "bits": 16,
1780
+ "data_type": "fp"
1781
+ },
1782
+ "model.layers.37.mlp.shared_experts.up_proj": {
1783
+ "act_data_type": "fp",
1784
+ "bits": 16,
1785
+ "data_type": "fp"
1786
+ },
1787
+ "model.layers.37.self_attn.b_proj": {
1788
+ "act_data_type": "fp",
1789
+ "bits": 16,
1790
+ "data_type": "fp"
1791
+ },
1792
+ "model.layers.37.self_attn.f_a_proj": {
1793
+ "act_data_type": "fp",
1794
+ "bits": 16,
1795
+ "data_type": "fp"
1796
+ },
1797
+ "model.layers.37.self_attn.f_b_proj": {
1798
+ "act_data_type": "fp",
1799
+ "bits": 16,
1800
+ "data_type": "fp"
1801
+ },
1802
+ "model.layers.37.self_attn.g_a_proj": {
1803
+ "act_data_type": "fp",
1804
+ "bits": 16,
1805
+ "data_type": "fp"
1806
+ },
1807
+ "model.layers.37.self_attn.g_b_proj": {
1808
+ "act_data_type": "fp",
1809
+ "bits": 16,
1810
+ "data_type": "fp"
1811
+ },
1812
+ "model.layers.37.self_attn.k_proj": {
1813
+ "act_data_type": "fp",
1814
+ "bits": 16,
1815
+ "data_type": "fp"
1816
+ },
1817
+ "model.layers.37.self_attn.o_proj": {
1818
+ "act_data_type": "fp",
1819
+ "bits": 16,
1820
+ "data_type": "fp"
1821
+ },
1822
+ "model.layers.37.self_attn.q_proj": {
1823
+ "act_data_type": "fp",
1824
+ "bits": 16,
1825
+ "data_type": "fp"
1826
+ },
1827
+ "model.layers.37.self_attn.v_proj": {
1828
+ "act_data_type": "fp",
1829
+ "bits": 16,
1830
+ "data_type": "fp"
1831
+ },
1832
+ "model.layers.38.mlp.shared_experts.down_proj": {
1833
+ "act_data_type": "fp",
1834
+ "bits": 16,
1835
+ "data_type": "fp"
1836
+ },
1837
+ "model.layers.38.mlp.shared_experts.gate_proj": {
1838
+ "act_data_type": "fp",
1839
+ "bits": 16,
1840
+ "data_type": "fp"
1841
+ },
1842
+ "model.layers.38.mlp.shared_experts.up_proj": {
1843
+ "act_data_type": "fp",
1844
+ "bits": 16,
1845
+ "data_type": "fp"
1846
+ },
1847
+ "model.layers.38.self_attn.b_proj": {
1848
+ "act_data_type": "fp",
1849
+ "bits": 16,
1850
+ "data_type": "fp"
1851
+ },
1852
+ "model.layers.38.self_attn.f_a_proj": {
1853
+ "act_data_type": "fp",
1854
+ "bits": 16,
1855
+ "data_type": "fp"
1856
+ },
1857
+ "model.layers.38.self_attn.f_b_proj": {
1858
+ "act_data_type": "fp",
1859
+ "bits": 16,
1860
+ "data_type": "fp"
1861
+ },
1862
+ "model.layers.38.self_attn.g_a_proj": {
1863
+ "act_data_type": "fp",
1864
+ "bits": 16,
1865
+ "data_type": "fp"
1866
+ },
1867
+ "model.layers.38.self_attn.g_b_proj": {
1868
+ "act_data_type": "fp",
1869
+ "bits": 16,
1870
+ "data_type": "fp"
1871
+ },
1872
+ "model.layers.38.self_attn.k_proj": {
1873
+ "act_data_type": "fp",
1874
+ "bits": 16,
1875
+ "data_type": "fp"
1876
+ },
1877
+ "model.layers.38.self_attn.o_proj": {
1878
+ "act_data_type": "fp",
1879
+ "bits": 16,
1880
+ "data_type": "fp"
1881
+ },
1882
+ "model.layers.38.self_attn.q_proj": {
1883
+ "act_data_type": "fp",
1884
+ "bits": 16,
1885
+ "data_type": "fp"
1886
+ },
1887
+ "model.layers.38.self_attn.v_proj": {
1888
+ "act_data_type": "fp",
1889
+ "bits": 16,
1890
+ "data_type": "fp"
1891
+ },
1892
+ "model.layers.39.mlp.shared_experts.down_proj": {
1893
+ "act_data_type": "fp",
1894
+ "bits": 16,
1895
+ "data_type": "fp"
1896
+ },
1897
+ "model.layers.39.mlp.shared_experts.gate_proj": {
1898
+ "act_data_type": "fp",
1899
+ "bits": 16,
1900
+ "data_type": "fp"
1901
+ },
1902
+ "model.layers.39.mlp.shared_experts.up_proj": {
1903
+ "act_data_type": "fp",
1904
+ "bits": 16,
1905
+ "data_type": "fp"
1906
+ },
1907
+ "model.layers.39.self_attn.b_proj": {
1908
+ "act_data_type": "fp",
1909
+ "bits": 16,
1910
+ "data_type": "fp"
1911
+ },
1912
+ "model.layers.39.self_attn.f_a_proj": {
1913
+ "act_data_type": "fp",
1914
+ "bits": 16,
1915
+ "data_type": "fp"
1916
+ },
1917
+ "model.layers.39.self_attn.f_b_proj": {
1918
+ "act_data_type": "fp",
1919
+ "bits": 16,
1920
+ "data_type": "fp"
1921
+ },
1922
+ "model.layers.39.self_attn.g_a_proj": {
1923
+ "act_data_type": "fp",
1924
+ "bits": 16,
1925
+ "data_type": "fp"
1926
+ },
1927
+ "model.layers.39.self_attn.g_b_proj": {
1928
+ "act_data_type": "fp",
1929
+ "bits": 16,
1930
+ "data_type": "fp"
1931
+ },
1932
+ "model.layers.39.self_attn.k_proj": {
1933
+ "act_data_type": "fp",
1934
+ "bits": 16,
1935
+ "data_type": "fp"
1936
+ },
1937
+ "model.layers.39.self_attn.o_proj": {
1938
+ "act_data_type": "fp",
1939
+ "bits": 16,
1940
+ "data_type": "fp"
1941
+ },
1942
+ "model.layers.39.self_attn.q_proj": {
1943
+ "act_data_type": "fp",
1944
+ "bits": 16,
1945
+ "data_type": "fp"
1946
+ },
1947
+ "model.layers.39.self_attn.v_proj": {
1948
+ "act_data_type": "fp",
1949
+ "bits": 16,
1950
+ "data_type": "fp"
1951
+ },
1952
+ "model.layers.4.mlp.shared_experts.down_proj": {
1953
+ "act_data_type": "fp",
1954
+ "bits": 16,
1955
+ "data_type": "fp"
1956
+ },
1957
+ "model.layers.4.mlp.shared_experts.gate_proj": {
1958
+ "act_data_type": "fp",
1959
+ "bits": 16,
1960
+ "data_type": "fp"
1961
+ },
1962
+ "model.layers.4.mlp.shared_experts.up_proj": {
1963
+ "act_data_type": "fp",
1964
+ "bits": 16,
1965
+ "data_type": "fp"
1966
+ },
1967
+ "model.layers.4.self_attn.g_proj": {
1968
+ "act_data_type": "fp",
1969
+ "bits": 16,
1970
+ "data_type": "fp"
1971
+ },
1972
+ "model.layers.4.self_attn.k_proj": {
1973
+ "act_data_type": "fp",
1974
+ "bits": 16,
1975
+ "data_type": "fp"
1976
+ },
1977
+ "model.layers.4.self_attn.o_proj": {
1978
+ "act_data_type": "fp",
1979
+ "bits": 16,
1980
+ "data_type": "fp"
1981
+ },
1982
+ "model.layers.4.self_attn.q_proj": {
1983
+ "act_data_type": "fp",
1984
+ "bits": 16,
1985
+ "data_type": "fp"
1986
+ },
1987
+ "model.layers.4.self_attn.v_proj": {
1988
+ "act_data_type": "fp",
1989
+ "bits": 16,
1990
+ "data_type": "fp"
1991
+ },
1992
+ "model.layers.40.mlp.shared_experts.down_proj": {
1993
+ "act_data_type": "fp",
1994
+ "bits": 16,
1995
+ "data_type": "fp"
1996
+ },
1997
+ "model.layers.40.mlp.shared_experts.gate_proj": {
1998
+ "act_data_type": "fp",
1999
+ "bits": 16,
2000
+ "data_type": "fp"
2001
+ },
2002
+ "model.layers.40.mlp.shared_experts.up_proj": {
2003
+ "act_data_type": "fp",
2004
+ "bits": 16,
2005
+ "data_type": "fp"
2006
+ },
2007
+ "model.layers.40.self_attn.g_proj": {
2008
+ "act_data_type": "fp",
2009
+ "bits": 16,
2010
+ "data_type": "fp"
2011
+ },
2012
+ "model.layers.40.self_attn.k_proj": {
2013
+ "act_data_type": "fp",
2014
+ "bits": 16,
2015
+ "data_type": "fp"
2016
+ },
2017
+ "model.layers.40.self_attn.o_proj": {
2018
+ "act_data_type": "fp",
2019
+ "bits": 16,
2020
+ "data_type": "fp"
2021
+ },
2022
+ "model.layers.40.self_attn.q_proj": {
2023
+ "act_data_type": "fp",
2024
+ "bits": 16,
2025
+ "data_type": "fp"
2026
+ },
2027
+ "model.layers.40.self_attn.v_proj": {
2028
+ "act_data_type": "fp",
2029
+ "bits": 16,
2030
+ "data_type": "fp"
2031
+ },
2032
+ "model.layers.41.mlp.shared_experts.down_proj": {
2033
+ "act_data_type": "fp",
2034
+ "bits": 16,
2035
+ "data_type": "fp"
2036
+ },
2037
+ "model.layers.41.mlp.shared_experts.gate_proj": {
2038
+ "act_data_type": "fp",
2039
+ "bits": 16,
2040
+ "data_type": "fp"
2041
+ },
2042
+ "model.layers.41.mlp.shared_experts.up_proj": {
2043
+ "act_data_type": "fp",
2044
+ "bits": 16,
2045
+ "data_type": "fp"
2046
+ },
2047
+ "model.layers.41.self_attn.b_proj": {
2048
+ "act_data_type": "fp",
2049
+ "bits": 16,
2050
+ "data_type": "fp"
2051
+ },
2052
+ "model.layers.41.self_attn.f_a_proj": {
2053
+ "act_data_type": "fp",
2054
+ "bits": 16,
2055
+ "data_type": "fp"
2056
+ },
2057
+ "model.layers.41.self_attn.f_b_proj": {
2058
+ "act_data_type": "fp",
2059
+ "bits": 16,
2060
+ "data_type": "fp"
2061
+ },
2062
+ "model.layers.41.self_attn.g_a_proj": {
2063
+ "act_data_type": "fp",
2064
+ "bits": 16,
2065
+ "data_type": "fp"
2066
+ },
2067
+ "model.layers.41.self_attn.g_b_proj": {
2068
+ "act_data_type": "fp",
2069
+ "bits": 16,
2070
+ "data_type": "fp"
2071
+ },
2072
+ "model.layers.41.self_attn.k_proj": {
2073
+ "act_data_type": "fp",
2074
+ "bits": 16,
2075
+ "data_type": "fp"
2076
+ },
2077
+ "model.layers.41.self_attn.o_proj": {
2078
+ "act_data_type": "fp",
2079
+ "bits": 16,
2080
+ "data_type": "fp"
2081
+ },
2082
+ "model.layers.41.self_attn.q_proj": {
2083
+ "act_data_type": "fp",
2084
+ "bits": 16,
2085
+ "data_type": "fp"
2086
+ },
2087
+ "model.layers.41.self_attn.v_proj": {
2088
+ "act_data_type": "fp",
2089
+ "bits": 16,
2090
+ "data_type": "fp"
2091
+ },
2092
+ "model.layers.42.mlp.shared_experts.down_proj": {
2093
+ "act_data_type": "fp",
2094
+ "bits": 16,
2095
+ "data_type": "fp"
2096
+ },
2097
+ "model.layers.42.mlp.shared_experts.gate_proj": {
2098
+ "act_data_type": "fp",
2099
+ "bits": 16,
2100
+ "data_type": "fp"
2101
+ },
2102
+ "model.layers.42.mlp.shared_experts.up_proj": {
2103
+ "act_data_type": "fp",
2104
+ "bits": 16,
2105
+ "data_type": "fp"
2106
+ },
2107
+ "model.layers.42.self_attn.b_proj": {
2108
+ "act_data_type": "fp",
2109
+ "bits": 16,
2110
+ "data_type": "fp"
2111
+ },
2112
+ "model.layers.42.self_attn.f_a_proj": {
2113
+ "act_data_type": "fp",
2114
+ "bits": 16,
2115
+ "data_type": "fp"
2116
+ },
2117
+ "model.layers.42.self_attn.f_b_proj": {
2118
+ "act_data_type": "fp",
2119
+ "bits": 16,
2120
+ "data_type": "fp"
2121
+ },
2122
+ "model.layers.42.self_attn.g_a_proj": {
2123
+ "act_data_type": "fp",
2124
+ "bits": 16,
2125
+ "data_type": "fp"
2126
+ },
2127
+ "model.layers.42.self_attn.g_b_proj": {
2128
+ "act_data_type": "fp",
2129
+ "bits": 16,
2130
+ "data_type": "fp"
2131
+ },
2132
+ "model.layers.42.self_attn.k_proj": {
2133
+ "act_data_type": "fp",
2134
+ "bits": 16,
2135
+ "data_type": "fp"
2136
+ },
2137
+ "model.layers.42.self_attn.o_proj": {
2138
+ "act_data_type": "fp",
2139
+ "bits": 16,
2140
+ "data_type": "fp"
2141
+ },
2142
+ "model.layers.42.self_attn.q_proj": {
2143
+ "act_data_type": "fp",
2144
+ "bits": 16,
2145
+ "data_type": "fp"
2146
+ },
2147
+ "model.layers.42.self_attn.v_proj": {
2148
+ "act_data_type": "fp",
2149
+ "bits": 16,
2150
+ "data_type": "fp"
2151
+ },
2152
+ "model.layers.43.mlp.shared_experts.down_proj": {
2153
+ "act_data_type": "fp",
2154
+ "bits": 16,
2155
+ "data_type": "fp"
2156
+ },
2157
+ "model.layers.43.mlp.shared_experts.gate_proj": {
2158
+ "act_data_type": "fp",
2159
+ "bits": 16,
2160
+ "data_type": "fp"
2161
+ },
2162
+ "model.layers.43.mlp.shared_experts.up_proj": {
2163
+ "act_data_type": "fp",
2164
+ "bits": 16,
2165
+ "data_type": "fp"
2166
+ },
2167
+ "model.layers.43.self_attn.b_proj": {
2168
+ "act_data_type": "fp",
2169
+ "bits": 16,
2170
+ "data_type": "fp"
2171
+ },
2172
+ "model.layers.43.self_attn.f_a_proj": {
2173
+ "act_data_type": "fp",
2174
+ "bits": 16,
2175
+ "data_type": "fp"
2176
+ },
2177
+ "model.layers.43.self_attn.f_b_proj": {
2178
+ "act_data_type": "fp",
2179
+ "bits": 16,
2180
+ "data_type": "fp"
2181
+ },
2182
+ "model.layers.43.self_attn.g_a_proj": {
2183
+ "act_data_type": "fp",
2184
+ "bits": 16,
2185
+ "data_type": "fp"
2186
+ },
2187
+ "model.layers.43.self_attn.g_b_proj": {
2188
+ "act_data_type": "fp",
2189
+ "bits": 16,
2190
+ "data_type": "fp"
2191
+ },
2192
+ "model.layers.43.self_attn.k_proj": {
2193
+ "act_data_type": "fp",
2194
+ "bits": 16,
2195
+ "data_type": "fp"
2196
+ },
2197
+ "model.layers.43.self_attn.o_proj": {
2198
+ "act_data_type": "fp",
2199
+ "bits": 16,
2200
+ "data_type": "fp"
2201
+ },
2202
+ "model.layers.43.self_attn.q_proj": {
2203
+ "act_data_type": "fp",
2204
+ "bits": 16,
2205
+ "data_type": "fp"
2206
+ },
2207
+ "model.layers.43.self_attn.v_proj": {
2208
+ "act_data_type": "fp",
2209
+ "bits": 16,
2210
+ "data_type": "fp"
2211
+ },
2212
+ "model.layers.44.mlp.shared_experts.down_proj": {
2213
+ "act_data_type": "fp",
2214
+ "bits": 16,
2215
+ "data_type": "fp"
2216
+ },
2217
+ "model.layers.44.mlp.shared_experts.gate_proj": {
2218
+ "act_data_type": "fp",
2219
+ "bits": 16,
2220
+ "data_type": "fp"
2221
+ },
2222
+ "model.layers.44.mlp.shared_experts.up_proj": {
2223
+ "act_data_type": "fp",
2224
+ "bits": 16,
2225
+ "data_type": "fp"
2226
+ },
2227
+ "model.layers.44.self_attn.g_proj": {
2228
+ "act_data_type": "fp",
2229
+ "bits": 16,
2230
+ "data_type": "fp"
2231
+ },
2232
+ "model.layers.44.self_attn.k_proj": {
2233
+ "act_data_type": "fp",
2234
+ "bits": 16,
2235
+ "data_type": "fp"
2236
+ },
2237
+ "model.layers.44.self_attn.o_proj": {
2238
+ "act_data_type": "fp",
2239
+ "bits": 16,
2240
+ "data_type": "fp"
2241
+ },
2242
+ "model.layers.44.self_attn.q_proj": {
2243
+ "act_data_type": "fp",
2244
+ "bits": 16,
2245
+ "data_type": "fp"
2246
+ },
2247
+ "model.layers.44.self_attn.v_proj": {
2248
+ "act_data_type": "fp",
2249
+ "bits": 16,
2250
+ "data_type": "fp"
2251
+ },
2252
+ "model.layers.45.mlp.shared_experts.down_proj": {
2253
+ "act_data_type": "fp",
2254
+ "bits": 16,
2255
+ "data_type": "fp"
2256
+ },
2257
+ "model.layers.45.mlp.shared_experts.gate_proj": {
2258
+ "act_data_type": "fp",
2259
+ "bits": 16,
2260
+ "data_type": "fp"
2261
+ },
2262
+ "model.layers.45.mlp.shared_experts.up_proj": {
2263
+ "act_data_type": "fp",
2264
+ "bits": 16,
2265
+ "data_type": "fp"
2266
+ },
2267
+ "model.layers.45.self_attn.b_proj": {
2268
+ "act_data_type": "fp",
2269
+ "bits": 16,
2270
+ "data_type": "fp"
2271
+ },
2272
+ "model.layers.45.self_attn.f_a_proj": {
2273
+ "act_data_type": "fp",
2274
+ "bits": 16,
2275
+ "data_type": "fp"
2276
+ },
2277
+ "model.layers.45.self_attn.f_b_proj": {
2278
+ "act_data_type": "fp",
2279
+ "bits": 16,
2280
+ "data_type": "fp"
2281
+ },
2282
+ "model.layers.45.self_attn.g_a_proj": {
2283
+ "act_data_type": "fp",
2284
+ "bits": 16,
2285
+ "data_type": "fp"
2286
+ },
2287
+ "model.layers.45.self_attn.g_b_proj": {
2288
+ "act_data_type": "fp",
2289
+ "bits": 16,
2290
+ "data_type": "fp"
2291
+ },
2292
+ "model.layers.45.self_attn.k_proj": {
2293
+ "act_data_type": "fp",
2294
+ "bits": 16,
2295
+ "data_type": "fp"
2296
+ },
2297
+ "model.layers.45.self_attn.o_proj": {
2298
+ "act_data_type": "fp",
2299
+ "bits": 16,
2300
+ "data_type": "fp"
2301
+ },
2302
+ "model.layers.45.self_attn.q_proj": {
2303
+ "act_data_type": "fp",
2304
+ "bits": 16,
2305
+ "data_type": "fp"
2306
+ },
2307
+ "model.layers.45.self_attn.v_proj": {
2308
+ "act_data_type": "fp",
2309
+ "bits": 16,
2310
+ "data_type": "fp"
2311
+ },
2312
+ "model.layers.46.mlp.shared_experts.down_proj": {
2313
+ "act_data_type": "fp",
2314
+ "bits": 16,
2315
+ "data_type": "fp"
2316
+ },
2317
+ "model.layers.46.mlp.shared_experts.gate_proj": {
2318
+ "act_data_type": "fp",
2319
+ "bits": 16,
2320
+ "data_type": "fp"
2321
+ },
2322
+ "model.layers.46.mlp.shared_experts.up_proj": {
2323
+ "act_data_type": "fp",
2324
+ "bits": 16,
2325
+ "data_type": "fp"
2326
+ },
2327
+ "model.layers.46.self_attn.b_proj": {
2328
+ "act_data_type": "fp",
2329
+ "bits": 16,
2330
+ "data_type": "fp"
2331
+ },
2332
+ "model.layers.46.self_attn.f_a_proj": {
2333
+ "act_data_type": "fp",
2334
+ "bits": 16,
2335
+ "data_type": "fp"
2336
+ },
2337
+ "model.layers.46.self_attn.f_b_proj": {
2338
+ "act_data_type": "fp",
2339
+ "bits": 16,
2340
+ "data_type": "fp"
2341
+ },
2342
+ "model.layers.46.self_attn.g_a_proj": {
2343
+ "act_data_type": "fp",
2344
+ "bits": 16,
2345
+ "data_type": "fp"
2346
+ },
2347
+ "model.layers.46.self_attn.g_b_proj": {
2348
+ "act_data_type": "fp",
2349
+ "bits": 16,
2350
+ "data_type": "fp"
2351
+ },
2352
+ "model.layers.46.self_attn.k_proj": {
2353
+ "act_data_type": "fp",
2354
+ "bits": 16,
2355
+ "data_type": "fp"
2356
+ },
2357
+ "model.layers.46.self_attn.o_proj": {
2358
+ "act_data_type": "fp",
2359
+ "bits": 16,
2360
+ "data_type": "fp"
2361
+ },
2362
+ "model.layers.46.self_attn.q_proj": {
2363
+ "act_data_type": "fp",
2364
+ "bits": 16,
2365
+ "data_type": "fp"
2366
+ },
2367
+ "model.layers.46.self_attn.v_proj": {
2368
+ "act_data_type": "fp",
2369
+ "bits": 16,
2370
+ "data_type": "fp"
2371
+ },
2372
+ "model.layers.47.mlp.shared_experts.down_proj": {
2373
+ "act_data_type": "fp",
2374
+ "bits": 16,
2375
+ "data_type": "fp"
2376
+ },
2377
+ "model.layers.47.mlp.shared_experts.gate_proj": {
2378
+ "act_data_type": "fp",
2379
+ "bits": 16,
2380
+ "data_type": "fp"
2381
+ },
2382
+ "model.layers.47.mlp.shared_experts.up_proj": {
2383
+ "act_data_type": "fp",
2384
+ "bits": 16,
2385
+ "data_type": "fp"
2386
+ },
2387
+ "model.layers.47.self_attn.b_proj": {
2388
+ "act_data_type": "fp",
2389
+ "bits": 16,
2390
+ "data_type": "fp"
2391
+ },
2392
+ "model.layers.47.self_attn.f_a_proj": {
2393
+ "act_data_type": "fp",
2394
+ "bits": 16,
2395
+ "data_type": "fp"
2396
+ },
2397
+ "model.layers.47.self_attn.f_b_proj": {
2398
+ "act_data_type": "fp",
2399
+ "bits": 16,
2400
+ "data_type": "fp"
2401
+ },
2402
+ "model.layers.47.self_attn.g_a_proj": {
2403
+ "act_data_type": "fp",
2404
+ "bits": 16,
2405
+ "data_type": "fp"
2406
+ },
2407
+ "model.layers.47.self_attn.g_b_proj": {
2408
+ "act_data_type": "fp",
2409
+ "bits": 16,
2410
+ "data_type": "fp"
2411
+ },
2412
+ "model.layers.47.self_attn.k_proj": {
2413
+ "act_data_type": "fp",
2414
+ "bits": 16,
2415
+ "data_type": "fp"
2416
+ },
2417
+ "model.layers.47.self_attn.o_proj": {
2418
+ "act_data_type": "fp",
2419
+ "bits": 16,
2420
+ "data_type": "fp"
2421
+ },
2422
+ "model.layers.47.self_attn.q_proj": {
2423
+ "act_data_type": "fp",
2424
+ "bits": 16,
2425
+ "data_type": "fp"
2426
+ },
2427
+ "model.layers.47.self_attn.v_proj": {
2428
+ "act_data_type": "fp",
2429
+ "bits": 16,
2430
+ "data_type": "fp"
2431
+ },
2432
+ "model.layers.5.mlp.shared_experts.down_proj": {
2433
+ "act_data_type": "fp",
2434
+ "bits": 16,
2435
+ "data_type": "fp"
2436
+ },
2437
+ "model.layers.5.mlp.shared_experts.gate_proj": {
2438
+ "act_data_type": "fp",
2439
+ "bits": 16,
2440
+ "data_type": "fp"
2441
+ },
2442
+ "model.layers.5.mlp.shared_experts.up_proj": {
2443
+ "act_data_type": "fp",
2444
+ "bits": 16,
2445
+ "data_type": "fp"
2446
+ },
2447
+ "model.layers.5.self_attn.b_proj": {
2448
+ "act_data_type": "fp",
2449
+ "bits": 16,
2450
+ "data_type": "fp"
2451
+ },
2452
+ "model.layers.5.self_attn.f_a_proj": {
2453
+ "act_data_type": "fp",
2454
+ "bits": 16,
2455
+ "data_type": "fp"
2456
+ },
2457
+ "model.layers.5.self_attn.f_b_proj": {
2458
+ "act_data_type": "fp",
2459
+ "bits": 16,
2460
+ "data_type": "fp"
2461
+ },
2462
+ "model.layers.5.self_attn.g_a_proj": {
2463
+ "act_data_type": "fp",
2464
+ "bits": 16,
2465
+ "data_type": "fp"
2466
+ },
2467
+ "model.layers.5.self_attn.g_b_proj": {
2468
+ "act_data_type": "fp",
2469
+ "bits": 16,
2470
+ "data_type": "fp"
2471
+ },
2472
+ "model.layers.5.self_attn.k_proj": {
2473
+ "act_data_type": "fp",
2474
+ "bits": 16,
2475
+ "data_type": "fp"
2476
+ },
2477
+ "model.layers.5.self_attn.o_proj": {
2478
+ "act_data_type": "fp",
2479
+ "bits": 16,
2480
+ "data_type": "fp"
2481
+ },
2482
+ "model.layers.5.self_attn.q_proj": {
2483
+ "act_data_type": "fp",
2484
+ "bits": 16,
2485
+ "data_type": "fp"
2486
+ },
2487
+ "model.layers.5.self_attn.v_proj": {
2488
+ "act_data_type": "fp",
2489
+ "bits": 16,
2490
+ "data_type": "fp"
2491
+ },
2492
+ "model.layers.6.mlp.shared_experts.down_proj": {
2493
+ "act_data_type": "fp",
2494
+ "bits": 16,
2495
+ "data_type": "fp"
2496
+ },
2497
+ "model.layers.6.mlp.shared_experts.gate_proj": {
2498
+ "act_data_type": "fp",
2499
+ "bits": 16,
2500
+ "data_type": "fp"
2501
+ },
2502
+ "model.layers.6.mlp.shared_experts.up_proj": {
2503
+ "act_data_type": "fp",
2504
+ "bits": 16,
2505
+ "data_type": "fp"
2506
+ },
2507
+ "model.layers.6.self_attn.b_proj": {
2508
+ "act_data_type": "fp",
2509
+ "bits": 16,
2510
+ "data_type": "fp"
2511
+ },
2512
+ "model.layers.6.self_attn.f_a_proj": {
2513
+ "act_data_type": "fp",
2514
+ "bits": 16,
2515
+ "data_type": "fp"
2516
+ },
2517
+ "model.layers.6.self_attn.f_b_proj": {
2518
+ "act_data_type": "fp",
2519
+ "bits": 16,
2520
+ "data_type": "fp"
2521
+ },
2522
+ "model.layers.6.self_attn.g_a_proj": {
2523
+ "act_data_type": "fp",
2524
+ "bits": 16,
2525
+ "data_type": "fp"
2526
+ },
2527
+ "model.layers.6.self_attn.g_b_proj": {
2528
+ "act_data_type": "fp",
2529
+ "bits": 16,
2530
+ "data_type": "fp"
2531
+ },
2532
+ "model.layers.6.self_attn.k_proj": {
2533
+ "act_data_type": "fp",
2534
+ "bits": 16,
2535
+ "data_type": "fp"
2536
+ },
2537
+ "model.layers.6.self_attn.o_proj": {
2538
+ "act_data_type": "fp",
2539
+ "bits": 16,
2540
+ "data_type": "fp"
2541
+ },
2542
+ "model.layers.6.self_attn.q_proj": {
2543
+ "act_data_type": "fp",
2544
+ "bits": 16,
2545
+ "data_type": "fp"
2546
+ },
2547
+ "model.layers.6.self_attn.v_proj": {
2548
+ "act_data_type": "fp",
2549
+ "bits": 16,
2550
+ "data_type": "fp"
2551
+ },
2552
+ "model.layers.7.mlp.shared_experts.down_proj": {
2553
+ "act_data_type": "fp",
2554
+ "bits": 16,
2555
+ "data_type": "fp"
2556
+ },
2557
+ "model.layers.7.mlp.shared_experts.gate_proj": {
2558
+ "act_data_type": "fp",
2559
+ "bits": 16,
2560
+ "data_type": "fp"
2561
+ },
2562
+ "model.layers.7.mlp.shared_experts.up_proj": {
2563
+ "act_data_type": "fp",
2564
+ "bits": 16,
2565
+ "data_type": "fp"
2566
+ },
2567
+ "model.layers.7.self_attn.b_proj": {
2568
+ "act_data_type": "fp",
2569
+ "bits": 16,
2570
+ "data_type": "fp"
2571
+ },
2572
+ "model.layers.7.self_attn.f_a_proj": {
2573
+ "act_data_type": "fp",
2574
+ "bits": 16,
2575
+ "data_type": "fp"
2576
+ },
2577
+ "model.layers.7.self_attn.f_b_proj": {
2578
+ "act_data_type": "fp",
2579
+ "bits": 16,
2580
+ "data_type": "fp"
2581
+ },
2582
+ "model.layers.7.self_attn.g_a_proj": {
2583
+ "act_data_type": "fp",
2584
+ "bits": 16,
2585
+ "data_type": "fp"
2586
+ },
2587
+ "model.layers.7.self_attn.g_b_proj": {
2588
+ "act_data_type": "fp",
2589
+ "bits": 16,
2590
+ "data_type": "fp"
2591
+ },
2592
+ "model.layers.7.self_attn.k_proj": {
2593
+ "act_data_type": "fp",
2594
+ "bits": 16,
2595
+ "data_type": "fp"
2596
+ },
2597
+ "model.layers.7.self_attn.o_proj": {
2598
+ "act_data_type": "fp",
2599
+ "bits": 16,
2600
+ "data_type": "fp"
2601
+ },
2602
+ "model.layers.7.self_attn.q_proj": {
2603
+ "act_data_type": "fp",
2604
+ "bits": 16,
2605
+ "data_type": "fp"
2606
+ },
2607
+ "model.layers.7.self_attn.v_proj": {
2608
+ "act_data_type": "fp",
2609
+ "bits": 16,
2610
+ "data_type": "fp"
2611
+ },
2612
+ "model.layers.8.mlp.shared_experts.down_proj": {
2613
+ "act_data_type": "fp",
2614
+ "bits": 16,
2615
+ "data_type": "fp"
2616
+ },
2617
+ "model.layers.8.mlp.shared_experts.gate_proj": {
2618
+ "act_data_type": "fp",
2619
+ "bits": 16,
2620
+ "data_type": "fp"
2621
+ },
2622
+ "model.layers.8.mlp.shared_experts.up_proj": {
2623
+ "act_data_type": "fp",
2624
+ "bits": 16,
2625
+ "data_type": "fp"
2626
+ },
2627
+ "model.layers.8.self_attn.g_proj": {
2628
+ "act_data_type": "fp",
2629
+ "bits": 16,
2630
+ "data_type": "fp"
2631
+ },
2632
+ "model.layers.8.self_attn.k_proj": {
2633
+ "act_data_type": "fp",
2634
+ "bits": 16,
2635
+ "data_type": "fp"
2636
+ },
2637
+ "model.layers.8.self_attn.o_proj": {
2638
+ "act_data_type": "fp",
2639
+ "bits": 16,
2640
+ "data_type": "fp"
2641
+ },
2642
+ "model.layers.8.self_attn.q_proj": {
2643
+ "act_data_type": "fp",
2644
+ "bits": 16,
2645
+ "data_type": "fp"
2646
+ },
2647
+ "model.layers.8.self_attn.v_proj": {
2648
+ "act_data_type": "fp",
2649
+ "bits": 16,
2650
+ "data_type": "fp"
2651
+ },
2652
+ "model.layers.9.mlp.shared_experts.down_proj": {
2653
+ "act_data_type": "fp",
2654
+ "bits": 16,
2655
+ "data_type": "fp"
2656
+ },
2657
+ "model.layers.9.mlp.shared_experts.gate_proj": {
2658
+ "act_data_type": "fp",
2659
+ "bits": 16,
2660
+ "data_type": "fp"
2661
+ },
2662
+ "model.layers.9.mlp.shared_experts.up_proj": {
2663
+ "act_data_type": "fp",
2664
+ "bits": 16,
2665
+ "data_type": "fp"
2666
+ },
2667
+ "model.layers.9.self_attn.b_proj": {
2668
+ "act_data_type": "fp",
2669
+ "bits": 16,
2670
+ "data_type": "fp"
2671
+ },
2672
+ "model.layers.9.self_attn.f_a_proj": {
2673
+ "act_data_type": "fp",
2674
+ "bits": 16,
2675
+ "data_type": "fp"
2676
+ },
2677
+ "model.layers.9.self_attn.f_b_proj": {
2678
+ "act_data_type": "fp",
2679
+ "bits": 16,
2680
+ "data_type": "fp"
2681
+ },
2682
+ "model.layers.9.self_attn.g_a_proj": {
2683
+ "act_data_type": "fp",
2684
+ "bits": 16,
2685
+ "data_type": "fp"
2686
+ },
2687
+ "model.layers.9.self_attn.g_b_proj": {
2688
+ "act_data_type": "fp",
2689
+ "bits": 16,
2690
+ "data_type": "fp"
2691
+ },
2692
+ "model.layers.9.self_attn.k_proj": {
2693
+ "act_data_type": "fp",
2694
+ "bits": 16,
2695
+ "data_type": "fp"
2696
+ },
2697
+ "model.layers.9.self_attn.o_proj": {
2698
+ "act_data_type": "fp",
2699
+ "bits": 16,
2700
+ "data_type": "fp"
2701
+ },
2702
+ "model.layers.9.self_attn.q_proj": {
2703
+ "act_data_type": "fp",
2704
+ "bits": 16,
2705
+ "data_type": "fp"
2706
+ },
2707
+ "model.layers.9.self_attn.v_proj": {
2708
+ "act_data_type": "fp",
2709
+ "bits": 16,
2710
+ "data_type": "fp"
2711
+ }
2712
+ },
2713
+ "group_size": 128,
2714
+ "iters": 1000,
2715
+ "low_gpu_mem_usage": true,
2716
+ "nsamples": 1024,
2717
+ "packing_format": "auto_round:auto_gptq",
2718
+ "quant_method": "auto-round",
2719
+ "sym": true
2720
+ },
2721
+ "rms_norm_eps": 1e-05,
2722
+ "rope_scaling": null,
2723
+ "rope_theta": 10000,
2724
+ "routed_scaling_factor": 1.0,
2725
+ "tie_word_embeddings": false,
2726
+ "topk_group": 1,
2727
+ "transformers_version": "4.57.6",
2728
+ "use_cache": true,
2729
+ "use_gqa_gate": true,
2730
+ "use_gqa_gate_bias": false,
2731
+ "use_qk_norm": false,
2732
+ "use_rope": false,
2733
+ "vocab_size": 196608
2734
+ }
configuration_solar_open2.py ADDED
@@ -0,0 +1,277 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2025 Upstage AI.
3
+ # Copyright 2025 The ZhipuAI Inc. team and HuggingFace Inc. team.
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+ #
17
+ # This file has been modified by Upstage AI including
18
+ # - Hyperparameter Adjustments: Modified the model architecture by increasing vocab_size and num_hidden_layers, while decreasing num_attention_heads, intermediate_size, and moe_intermediate_size.
19
+ # RoPE Configuration: Replaced the generic rope_parameters argument with explicit rope_theta and rope_scaling parameters to define Rotary Positional Embeddings settings.
20
+ #
21
+ # Based on code from: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4_moe/configuration_glm4_moe.py
22
+
23
+ from transformers.configuration_utils import PretrainedConfig
24
+ from transformers.modeling_rope_utils import rope_config_validation
25
+
26
+
27
+ class SolarOpen2Config(PretrainedConfig):
28
+ r"""
29
+ This is the configuration class to store the configuration of a [`SolarOpen2Model`]. It is used to instantiate a
30
+ SolarOpen2 model according to the specified arguments, defining the model architecture.
31
+
32
+ Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
33
+ documentation from [`PretrainedConfig`] for more information.
34
+
35
+
36
+ Args:
37
+ vocab_size (`int`, *optional*, defaults to 196608):
38
+ Vocabulary size of the SolarOpen2 model. Defines the number of different tokens that can be represented by the
39
+ `inputs_ids` passed when calling [`SolarOpen2Model`]
40
+ hidden_size (`int`, *optional*, defaults to 4096):
41
+ Dimension of the hidden representations.
42
+ intermediate_size (`int`, *optional*, defaults to 10240):
43
+ Dimension of the MLP representations.
44
+ num_hidden_layers (`int`, *optional*, defaults to 48):
45
+ Number of hidden layers in the Transformer encoder.
46
+ num_attention_heads (`int`, *optional*, defaults to 64):
47
+ Number of attention heads for each attention layer in the Transformer encoder.
48
+ partial_rotary_factor (`float`, *optional*, defaults to 1.0):
49
+ The factor of the partial rotary position.
50
+ num_key_value_heads (`int`, *optional*, defaults to 8):
51
+ This is the number of key_value heads that should be used to implement Grouped Query Attention. If
52
+ `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
53
+ `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
54
+ converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
55
+ by meanpooling all the original heads within that group. For more details, check out [this
56
+ paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `32`.
57
+
58
+ hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
59
+ The non-linear activation function (function or string) in the decoder.
60
+ max_position_embeddings (`int`, *optional*, defaults to 131072):
61
+ The maximum sequence length that this model might ever be used with.
62
+ initializer_range (`float`, *optional*, defaults to 0.02):
63
+ The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
64
+ rms_norm_eps (`float`, *optional*, defaults to 1e-05):
65
+ The epsilon used by the rms normalization layers.
66
+ use_cache (`bool`, *optional*, defaults to `True`):
67
+ Whether or not the model should return the last key/values attentions (not used by all models). Only
68
+ relevant if `config.is_decoder=True`.
69
+ tie_word_embeddings (`bool`, *optional*, defaults to `False`):
70
+ Whether the model's input and output word embeddings should be tied.
71
+ rope_theta (`float`, *optional*, defaults to 1000000.0):
72
+ The base period of the RoPE embeddings.
73
+ rope_scaling (`Dict`, *optional*):
74
+ Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
75
+ and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
76
+ accordingly.
77
+ Expected contents:
78
+ `rope_type` (`str`):
79
+ The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
80
+ 'llama3'], with 'default' being the original RoPE implementation.
81
+ `factor` (`float`, *optional*):
82
+ Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
83
+ most scaling types, a `factor` of x will enable the model to handle sequences of length x *
84
+ original maximum pre-trained length.
85
+ `original_max_position_embeddings` (`int`, *optional*):
86
+ Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during
87
+ pretraining.
88
+ `attention_factor` (`float`, *optional*):
89
+ Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
90
+ computation. If unspecified, it defaults to value recommended by the implementation, using the
91
+ `factor` field to infer the suggested value.
92
+ `beta_fast` (`float`, *optional*):
93
+ Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
94
+ ramp function. If unspecified, it defaults to 32.
95
+ `beta_slow` (`float`, *optional*):
96
+ Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
97
+ ramp function. If unspecified, it defaults to 1.
98
+ `short_factor` (`list[float]`, *optional*):
99
+ Only used with 'longrope'. The scaling factor to be applied to short contexts (<
100
+ `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
101
+ size divided by the number of attention heads divided by 2
102
+ `long_factor` (`list[float]`, *optional*):
103
+ Only used with 'longrope'. The scaling factor to be applied to long contexts (<
104
+ `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
105
+ size divided by the number of attention heads divided by 2
106
+ `low_freq_factor` (`float`, *optional*):
107
+ Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
108
+ `high_freq_factor` (`float`, *optional*):
109
+ Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
110
+ attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
111
+ Whether to use a bias in the query, key, value and output projection layers during self-attention.
112
+ attention_dropout (`float`, *optional*, defaults to 0.0):
113
+ The dropout ratio for the attention probabilities.
114
+ moe_intermediate_size (`int`, *optional*, defaults to 1280):
115
+ Intermediate size of the routed expert.
116
+ num_experts_per_tok (`int`, *optional*, defaults to 8):
117
+ number of experts per token.
118
+ n_shared_experts (`int`, *optional*, defaults to 1):
119
+ Number of shared experts.
120
+ n_routed_experts (`int`, *optional*, defaults to 128):
121
+ Number of routed experts.
122
+ routed_scaling_factor (`float`, *optional*, defaults to 1.0):
123
+ Scaling factor or routed experts.
124
+ n_group (`int`, *optional*, defaults to 1):
125
+ Number of groups for routed experts.
126
+ topk_group (`int`, *optional*, defaults to 1):
127
+ Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
128
+ first_k_dense_replace (`int`, *optional*, defaults to 0):
129
+ Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
130
+ \--k dense layers--/
131
+ norm_topk_prob (`bool`, *optional*, defaults to `True`):
132
+ Whether to normalize the topk probabilities.
133
+ use_qk_norm (`bool`, *optional*, defaults to `False`):
134
+ Whether to use query-key normalization in the attention
135
+ ```python
136
+ >>> from transformers import SolarOpen2Model, SolarOpen2Config
137
+
138
+ >>> # Initializing a SolarOpen2 style configuration
139
+ >>> configuration = SolarOpen2Config()
140
+
141
+ >>> # Initializing a model from the SolarOpen2 style configuration
142
+ >>> model = SolarOpen2Model(configuration)
143
+
144
+ >>> # Accessing the model configuration
145
+ >>> configuration = model.config
146
+ ```"""
147
+
148
+ model_type = "solar_open2"
149
+ keys_to_ignore_at_inference = ["past_key_values"]
150
+
151
+ # Default tensor parallel plan for base model `SolarOpen2`
152
+ base_model_tp_plan = {
153
+ "layers.*.self_attn.q_proj": "colwise",
154
+ "layers.*.self_attn.k_proj": "colwise",
155
+ "layers.*.self_attn.v_proj": "colwise",
156
+ "layers.*.self_attn.o_proj": "rowwise",
157
+ "layers.*.mlp.experts.*.gate_proj": "colwise",
158
+ "layers.*.mlp.experts.*.up_proj": "colwise",
159
+ "layers.*.mlp.experts.*.down_proj": "rowwise",
160
+ "layers.*.mlp.gate_proj": "colwise",
161
+ "layers.*.mlp.up_proj": "colwise",
162
+ "layers.*.mlp.down_proj": "rowwise",
163
+ }
164
+ base_model_pp_plan = {
165
+ "embed_tokens": (["input_ids"], ["inputs_embeds"]),
166
+ "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
167
+ "norm": (["hidden_states"], ["hidden_states"]),
168
+ }
169
+
170
+ def __init__(
171
+ self,
172
+ vocab_size=196608,
173
+ hidden_size=4096,
174
+ intermediate_size=10240,
175
+ num_hidden_layers=48,
176
+ num_attention_heads=64,
177
+ partial_rotary_factor=1.0,
178
+ num_key_value_heads=8,
179
+ hidden_act="silu",
180
+ max_position_embeddings=131072,
181
+ initializer_range=0.02,
182
+ rms_norm_eps=1e-5,
183
+ use_cache=True,
184
+ tie_word_embeddings=False,
185
+ rope_theta=1000000.0,
186
+ rope_scaling=None,
187
+ attention_bias=False,
188
+ attention_dropout=0.0,
189
+ moe_intermediate_size=1280,
190
+ num_experts_per_tok=8,
191
+ n_shared_experts=1,
192
+ n_routed_experts=128,
193
+ routed_scaling_factor=1.0,
194
+ n_group=1,
195
+ topk_group=1,
196
+ first_k_dense_replace=0,
197
+ norm_topk_prob=True,
198
+ use_qk_norm=False,
199
+ use_rope=False,
200
+ gqa_interval=4,
201
+ gqa_layers=None,
202
+ use_gqa_gate=True,
203
+ use_gqa_gate_bias=False,
204
+ linear_attn_config={
205
+ "short_conv_kernel_size": 4,
206
+ "head_dim": 128,
207
+ "num_heads": 16,
208
+ "num_kv_heads": None,
209
+ },
210
+ kda_use_full_proj=False,
211
+ kda_gate_lower_bound=-5.0,
212
+ kda_allow_neg_eigval=False,
213
+ # Hyper-Connections
214
+ hc_rate=0,
215
+ hc_dynamic=True,
216
+ hc_use_tanh=True,
217
+ hc_use_sigmoid=False,
218
+ **kwargs,
219
+ ):
220
+ self.vocab_size = vocab_size
221
+ self.max_position_embeddings = max_position_embeddings
222
+ self.hidden_size = hidden_size
223
+ self.intermediate_size = intermediate_size
224
+ self.num_hidden_layers = num_hidden_layers
225
+ self.num_attention_heads = num_attention_heads
226
+ self.partial_rotary_factor = partial_rotary_factor
227
+
228
+ self.num_key_value_heads = num_key_value_heads
229
+ self.hidden_act = hidden_act
230
+ self.initializer_range = initializer_range
231
+ self.rms_norm_eps = rms_norm_eps
232
+ self.use_cache = use_cache
233
+ self.rope_theta = rope_theta
234
+ self.rope_scaling = rope_scaling
235
+ self.attention_bias = attention_bias
236
+ self.attention_dropout = attention_dropout
237
+ # Validate the correctness of rotary position embeddings parameters
238
+ # BC: if there is a 'type' field, move it to 'rope_type'.
239
+ if self.rope_scaling is not None and "type" in self.rope_scaling:
240
+ self.rope_scaling["rope_type"] = self.rope_scaling["type"]
241
+ rope_config_validation(self)
242
+
243
+ # MoE arguments
244
+ self.moe_intermediate_size = moe_intermediate_size
245
+ self.num_experts_per_tok = num_experts_per_tok
246
+ self.n_group = n_group
247
+ self.topk_group = topk_group
248
+ self.n_shared_experts = n_shared_experts
249
+ self.n_routed_experts = n_routed_experts
250
+ self.routed_scaling_factor = routed_scaling_factor
251
+ self.first_k_dense_replace = first_k_dense_replace
252
+ self.norm_topk_prob = norm_topk_prob
253
+ self.use_qk_norm = use_qk_norm
254
+
255
+ self.use_rope = use_rope
256
+ self.gqa_interval = gqa_interval
257
+ self.gqa_layers = gqa_layers
258
+ self.use_gqa_gate = use_gqa_gate
259
+ self.use_gqa_gate_bias = use_gqa_gate_bias
260
+ self.linear_attn_config = linear_attn_config
261
+ self.kda_use_full_proj = kda_use_full_proj
262
+ self.kda_gate_lower_bound = kda_gate_lower_bound
263
+ self.kda_allow_neg_eigval = kda_allow_neg_eigval
264
+
265
+ # Hyper-Connections
266
+ self.hc_rate = hc_rate
267
+ self.hc_dynamic = hc_dynamic
268
+ self.hc_use_tanh = hc_use_tanh
269
+ self.hc_use_sigmoid = hc_use_sigmoid
270
+
271
+ super().__init__(
272
+ tie_word_embeddings=tie_word_embeddings,
273
+ **kwargs,
274
+ )
275
+
276
+
277
+ __all__ = ["SolarOpen2Config"]
generation_config.json ADDED
@@ -0,0 +1,9 @@
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bos_token_id": 1,
3
+ "do_sample": true,
4
+ "eos_token_id": [
5
+ 2,
6
+ 129
7
+ ],
8
+ "transformers_version": "4.57.6"
9
+ }
model-00001-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:18cef87d5a1d358cdd90ef9c86c40cf4d13cc1289dfff7097aef49c9a477d420
3
+ size 5368598376
model-00002-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:018c49cb35f0aea579d38937fd8308cd595f54360698a51a1e54055494060717
3
+ size 5367450920
model-00003-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:eb4f74c8131d501423530aecddcecf9f3484aae7a050acf52a34a26a2b131f9f
3
+ size 5367280888
model-00004-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:470c95f6e3292c1ae53579e8634acb42630646554e2ab4ef0439193bc4b3831c
3
+ size 5367450920
model-00005-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:88cbc089b5aff97a7f9bcb0f0797c4c1d902d01cac8cc4cff5b76a495a699a9a
3
+ size 5367281328
model-00006-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4665d29a1149f848c857a8806337d6d670bf6eff9c4e73ed3312a0d1e3a502e8
3
+ size 5369077168
model-00007-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:99de25ea77e3982bd3d3f75cbc46d342a26a8470bd6a9e4249fdb333b8e94587
3
+ size 5369367576
model-00008-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5271afda3cdb14b59f523ecceb54d44fcc9ea3f208ba6b3860661b8d91871c31
3
+ size 5367456200
model-00009-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0899eedd0e33b0c31a7c31fdb87f7ae2fadadefc6700eb0e99b107111a2e23f1
3
+ size 5367286208
model-00010-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:66fad39877ab655c6eb949cc62b593e4a5eb0643a0ac385fc0e4d72617c97113
3
+ size 5367456200
model-00011-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ca63fc777bf337bfd08d807cd7d115267a0d7c1336d176977b3665ceda247e99
3
+ size 5367286432
model-00012-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:d6cd151be82838ec3cbf6a9dac96a741cb6091c46a7f37566a13f6d2a509d081
3
+ size 5367456728
model-00013-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f1fadee7a49c7b8082d1cf029281d318a3c8b4938865d45ad39127c68952401c
3
+ size 5368436456
model-00014-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:26a126dbc9464a3bdb2c9d2fc2e78efcd1e4ca32de4c11cfaa74ebc062b3aaff
3
+ size 5367286208
model-00015-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:19eac2ede0e1868b5e26a5ba0ebf31eab99dbb1e1d310f8fec255eb71320e4cb
3
+ size 5367456200
model-00016-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e05e06e9eb9659f65acca81cb6192da3fbcfa2535332454170a1a26241543e6a
3
+ size 5367286208
model-00017-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:ed78cbc350941ef435545610776418a409e0a6a823cc104a8300fb14dfa905be
3
+ size 5367456256
model-00018-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:14c333c47521edc1f607789af60ad95b921050ddd95bd55bee090de7f9278830
3
+ size 5367286680
model-00019-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5698b9f7538e022384e09cbc69411091d3da91cef5f0985c99ae6a964ae34680
3
+ size 5367884000
model-00020-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e666fde54483ec3e515803e3593844fd8f068a2199886eec4788efaeba76213a
3
+ size 5367838888
model-00021-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0a4638243698b6bfdafacfbadb293435e3a5a4bacaaec7e88a436cb940f5774e
3
+ size 5367456200
model-00022-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:4ce591f21b22c26aa561fc6f6e37b69e3ca36bba9aeae642d86464cebbf04a56
3
+ size 5367286208
model-00023-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:a5c43c90cf9148c1629e455d0c344377ae88bda7ae6ba4c6c26d1c60c18f1035
3
+ size 5367456200
model-00024-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:de37b1911fea1facac25c91c3da69d520c1643cf08305caf18bb0ec3bd5f6537
3
+ size 5367286512
model-00025-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:00238d2c1fcd7456b8a29688e2bbe63933fbc8d0eaee6f4634760282d476ada3
3
+ size 5367456752
model-00026-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b1118f8291f5f1883d57a6c263420a13b0578960a2dd17512c54d69f05bed97c
3
+ size 5368436352
model-00027-of-00027.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f0cb2668b4371e473e2648236c2f601554182c742ecab7b4a3cfd99ad7e1d5be
3
+ size 3364299304
model.safetensors.index.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:255b0cb9e82b5f564290bdd1c52734e2f9809d74ee80b056fdf3e3c601df1ae7
3
+ size 12675548
modeling_solar_open2.py ADDED
@@ -0,0 +1,1083 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ # coding=utf-8
2
+ # Copyright 2025 Upstage AI.
3
+ # Copyright 2025 The GLM4 & ZhipuAI team and HuggingFace Inc. team.
4
+ #
5
+ # Licensed under the Apache License, Version 2.0 (the "License");
6
+ # you may not use this file except in compliance with the License.
7
+ # You may obtain a copy of the License at
8
+ #
9
+ # http://www.apache.org/licenses/LICENSE-2.0
10
+ #
11
+ # Unless required by applicable law or agreed to in writing, software
12
+ # distributed under the License is distributed on an "AS IS" BASIS,
13
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14
+ # See the License for the specific language governing permissions and
15
+ # limitations under the License.
16
+ #
17
+ # This file has been modified by Upstage AI including:
18
+ # - Hybrid MoE Architecture: Replaced the standard dense structure with a depth-dependent Hybrid MoE, adding `SolarOpen2MoE` and `SolarOpen2TopkRouter` classes.
19
+ # - RoPE Strategy: Changed the rotary position embedding strategy from GLM4's interleaved rotation to Llama-style block rotation (via modified `rotate_half`).
20
+ # - Normalization Logic: Simplified the layer normalization structure by removing GLM4's extra post-operation norms and adding optional Query-Key Normalization (`use_qk_norm`).
21
+ #
22
+ # Based on code from: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4/modeling_glm4.py
23
+
24
+ from typing import Callable, Optional, Union, Any
25
+
26
+ import torch
27
+ import torch.nn.functional as F
28
+ from torch import nn
29
+ from einops import rearrange
30
+
31
+ from transformers.activations import ACT2FN
32
+ from transformers.cache_utils import Cache, DynamicCache
33
+ from transformers.generation import GenerationMixin
34
+ from transformers.integrations import use_kernel_forward_from_hub
35
+ from transformers.masking_utils import create_causal_mask
36
+ from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
37
+ from transformers.modeling_layers import GradientCheckpointingLayer
38
+ from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
39
+ from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
40
+ from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
41
+ from transformers.processing_utils import Unpack
42
+ from transformers.utils import TransformersKwargs, auto_docstring, can_return_tuple
43
+ from transformers.utils.deprecation import deprecate_kwarg
44
+ from transformers.utils.generic import check_model_inputs
45
+ from .configuration_solar_open2 import SolarOpen2Config
46
+
47
+ try:
48
+ from fla.modules import FusedRMSNormGated, ShortConvolution
49
+ from fla.ops.kda import chunk_kda, fused_recurrent_kda
50
+ from fla.ops.kda.gate import fused_kda_gate
51
+ from fla.layers.utils import get_unpad_data, index_first_axis, pad_input
52
+ except ImportError:
53
+ raise ImportError("Plese run `pip install -U fla-core`")
54
+
55
+ def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
56
+ """
57
+ This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
58
+ num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
59
+ """
60
+ batch, num_key_value_heads, slen, head_dim = hidden_states.shape
61
+ if n_rep == 1:
62
+ return hidden_states
63
+ hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
64
+ return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
65
+
66
+
67
+ def repeat_kv_linear(x: torch.Tensor, n_rep: int) -> torch.Tensor:
68
+ """torch.repeat_interleave(x, dim=-2, repeats=n_rep) for (..., n_kv_heads, head_dim) layout."""
69
+ if n_rep == 1:
70
+ return x
71
+ shape = x.shape
72
+ return (
73
+ x.unsqueeze(-2)
74
+ .expand(*shape[:-1], n_rep, shape[-1])
75
+ .reshape(*shape[:-2], shape[-2] * n_rep, shape[-1])
76
+ )
77
+
78
+
79
+ def eager_attention_forward(
80
+ module: nn.Module,
81
+ query: torch.Tensor,
82
+ key: torch.Tensor,
83
+ value: torch.Tensor,
84
+ attention_mask: Optional[torch.Tensor],
85
+ scaling: float,
86
+ dropout: float = 0.0,
87
+ **kwargs: Unpack[TransformersKwargs],
88
+ ):
89
+ key_states = repeat_kv(key, module.num_key_value_groups)
90
+ value_states = repeat_kv(value, module.num_key_value_groups)
91
+
92
+ attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
93
+ if attention_mask is not None:
94
+ causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
95
+ attn_weights = attn_weights + causal_mask
96
+
97
+ attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
98
+ attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
99
+ attn_output = torch.matmul(attn_weights, value_states)
100
+ attn_output = attn_output.transpose(1, 2).contiguous()
101
+
102
+ return attn_output, attn_weights
103
+
104
+
105
+ def rotate_half(x):
106
+ """Rotates half the hidden dims of the input."""
107
+ x1 = x[..., : x.shape[-1] // 2]
108
+ x2 = x[..., x.shape[-1] // 2 :]
109
+ return torch.cat((-x2, x1), dim=-1)
110
+
111
+
112
+ def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
113
+ """Applies Rotary Position Embedding to the query and key tensors.
114
+
115
+ Args:
116
+ q (`torch.Tensor`): The query tensor.
117
+ k (`torch.Tensor`): The key tensor.
118
+ cos (`torch.Tensor`): The cosine part of the rotary embedding.
119
+ sin (`torch.Tensor`): The sine part of the rotary embedding.
120
+ position_ids (`torch.Tensor`, *optional*):
121
+ Deprecated and unused.
122
+ unsqueeze_dim (`int`, *optional*, defaults to 1):
123
+ The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
124
+ sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
125
+ that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
126
+ k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
127
+ cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
128
+ the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
129
+ Returns:
130
+ `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
131
+ """
132
+ cos = cos.unsqueeze(unsqueeze_dim)
133
+ sin = sin.unsqueeze(unsqueeze_dim)
134
+
135
+ # Keep half or full tensor for later concatenation
136
+ rotary_dim = cos.shape[-1]
137
+ q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
138
+ k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
139
+
140
+ # Apply rotary embeddings on the first half or full tensor
141
+ q_embed = (q_rot * cos) + (rotate_half(q_rot) * sin)
142
+ k_embed = (k_rot * cos) + (rotate_half(k_rot) * sin)
143
+
144
+ # Concatenate back to full shape
145
+ q_embed = torch.cat([q_embed, q_pass], dim=-1)
146
+ k_embed = torch.cat([k_embed, k_pass], dim=-1)
147
+ return q_embed, k_embed
148
+
149
+
150
+ class SolarOpen2DynamicCache:
151
+ """
152
+ Dynamic cache for SolarOpen2 model.
153
+ Inspired by Kimi-Linear
154
+ """
155
+ is_compileable = False
156
+
157
+ def __init__(self, config: SolarOpen2Config):
158
+ super().__init__()
159
+ self.config = config
160
+
161
+ if config.linear_attn_config is not None:
162
+ self.layer_types = []
163
+ for i in range(config.num_hidden_layers):
164
+ # Determine if this layer uses GQA: gqa_layers takes priority over gqa_interval
165
+ if config.gqa_layers is not None:
166
+ use_gqa = i in config.gqa_layers
167
+ else:
168
+ use_gqa = (i + 1) % config.gqa_interval == 0
169
+ if use_gqa:
170
+ self.layer_types.append("full_attention")
171
+ else:
172
+ self.layer_types.append("linear_attention")
173
+ else:
174
+ self.layer_types = ["full_attention"] * config.num_hidden_layers
175
+
176
+ self.transformer_layers = [
177
+ i for i in range(config.num_hidden_layers) if self.layer_types[i] == "full_attention"
178
+ ]
179
+
180
+ linear_layers = [i for i in range(
181
+ config.num_hidden_layers) if self.layer_types[i] == "linear_attention"]
182
+ self.last_linear_layer = linear_layers[-1] if linear_layers else -1
183
+
184
+ self.conv_states = [None for _ in range(config.num_hidden_layers)]
185
+ self.recurrent_states = [None for _ in range(config.num_hidden_layers)]
186
+ self.key_cache = [None for _ in range(config.num_hidden_layers)]
187
+ self.value_cache = [None for _ in range(config.num_hidden_layers)]
188
+
189
+ def __len__(self):
190
+ return len(self.layer_types)
191
+
192
+ def update(
193
+ self,
194
+ key_states: torch.Tensor,
195
+ value_states: torch.Tensor,
196
+ layer_idx: int,
197
+ cache_kwargs: dict[str, Any] | None = None,
198
+ ) -> tuple[torch.Tensor, torch.Tensor]:
199
+ if self.key_cache[layer_idx] is None:
200
+ self.key_cache[layer_idx] = key_states
201
+ self.value_cache[layer_idx] = value_states
202
+ else:
203
+ self.key_cache[layer_idx] = torch.cat(
204
+ [self.key_cache[layer_idx], key_states], dim=2)
205
+ self.value_cache[layer_idx] = torch.cat(
206
+ [self.value_cache[layer_idx], value_states], dim=2)
207
+
208
+ return self.key_cache[layer_idx], self.value_cache[layer_idx]
209
+
210
+ def reorder_cache(self, beam_idx: torch.LongTensor):
211
+ """Reorders the cache for beam search, given the selected beam indices."""
212
+ for layer_idx in range(len(self.key_cache)):
213
+ if self.key_cache[layer_idx] is not None:
214
+ device = self.key_cache[layer_idx].device
215
+ beam_idx = beam_idx.to(device)
216
+ self.key_cache[layer_idx] = self.key_cache[layer_idx].index_select(
217
+ 0, beam_idx)
218
+ self.value_cache[layer_idx] = self.value_cache[layer_idx].index_select(
219
+ 0, beam_idx)
220
+
221
+ if self.conv_states[layer_idx] is not None:
222
+ device = self.conv_states[layer_idx][0].device
223
+ beam_idx = beam_idx.to(device)
224
+ q_conv, k_conv, v_conv = self.conv_states[layer_idx]
225
+ self.conv_states[layer_idx] = (
226
+ q_conv.index_select(0, beam_idx),
227
+ k_conv.index_select(0, beam_idx),
228
+ v_conv.index_select(0, beam_idx),
229
+ )
230
+ self.recurrent_states[layer_idx] = self.recurrent_states[layer_idx].index_select(
231
+ 0, beam_idx)
232
+
233
+ def get_seq_length(self, layer_idx: int | None = 0) -> int:
234
+ """Returns the sequence length of the cached states. A layer index can be optionally passed."""
235
+ # take any layer that contains cache and not empty tensor
236
+ layer_idx = self.transformer_layers[0] if layer_idx not in self.transformer_layers else layer_idx
237
+ if len(self.key_cache) <= layer_idx or self.key_cache[layer_idx] is None:
238
+ return 0
239
+ return self.key_cache[layer_idx].shape[-2]
240
+
241
+ def get_mask_sizes(self, cache_position: torch.Tensor, layer_idx: int) -> tuple[int, int]:
242
+ """
243
+ Return a tuple (kv_length, kv_offset) corresponding to the length and offset that will be returned for
244
+ the given layer at `layer_idx`.
245
+ The masks are then prepared according to the given lengths (kv_length, kv_offset) and patterns for each layer.
246
+ """
247
+ kv_offset = 0
248
+ query_length = cache_position.shape[0]
249
+ past_seen_tokens = self.get_seq_length(layer_idx)
250
+ kv_length = query_length + past_seen_tokens
251
+ return kv_length, kv_offset
252
+
253
+ @property
254
+ def has_previous_state(self):
255
+ """We have a previous state if the last linear (conv) layer was already updated."""
256
+ if self.last_linear_layer == -1:
257
+ return False
258
+ return self.conv_states[self.last_linear_layer] is not None
259
+
260
+
261
+ class SolarOpen2LinearAttention(nn.Module):
262
+ def __init__(self, config: SolarOpen2Config, layer_idx: int):
263
+ super().__init__()
264
+ self.config = config
265
+ self.mode = "chunk"
266
+
267
+ self.hidden_size = config.hidden_size
268
+ self.conv_size = config.linear_attn_config["short_conv_kernel_size"]
269
+ self.head_dim = config.linear_attn_config["head_dim"]
270
+ self.num_heads = config.linear_attn_config["num_heads"]
271
+ self.num_kv_heads = config.linear_attn_config.get("num_kv_heads", None) or self.num_heads
272
+ self.n_rep = self.num_heads // self.num_kv_heads
273
+ self.head_k_dim = self.head_dim
274
+ self.num_k_heads = self.num_heads
275
+
276
+ self.layer_idx = layer_idx
277
+ self.use_full_proj = getattr(config, 'kda_use_full_proj', False)
278
+ self.gate_lower_bound = getattr(config, 'kda_gate_lower_bound', -5.0)
279
+ self.allow_neg_eigval = getattr(config, 'kda_allow_neg_eigval', False)
280
+
281
+ assert self.mode in [
282
+ 'chunk', 'fused_recurrent'], f"Not suppoerted mode `{self.mode}`."
283
+
284
+ projection_k_size = self.head_k_dim * self.num_k_heads
285
+ projection_size = self.head_dim * self.num_heads
286
+ kv_projection_size = self.head_dim * self.num_kv_heads
287
+
288
+ self.q_proj = nn.Linear(
289
+ self.hidden_size, projection_k_size, bias=False)
290
+ self.k_proj = nn.Linear(
291
+ self.hidden_size, kv_projection_size, bias=False)
292
+ self.v_proj = nn.Linear(self.hidden_size, kv_projection_size, bias=False)
293
+
294
+ self.q_conv1d = ShortConvolution(
295
+ hidden_size=projection_k_size,
296
+ kernel_size=self.conv_size,
297
+ activation='silu',
298
+ )
299
+ self.k_conv1d = ShortConvolution(
300
+ hidden_size=kv_projection_size,
301
+ kernel_size=self.conv_size,
302
+ activation='silu',
303
+ )
304
+ self.v_conv1d = ShortConvolution(
305
+ hidden_size=kv_projection_size,
306
+ kernel_size=self.conv_size,
307
+ activation='silu',
308
+ )
309
+
310
+ self.A_log = torch.nn.Parameter(torch.log(torch.empty(
311
+ self.num_heads, dtype=torch.float32).uniform_(1, 16)).view(1, 1, -1, 1))
312
+
313
+ if self.use_full_proj:
314
+ self.f_proj = nn.Linear(self.hidden_size, projection_size, bias=False)
315
+ else:
316
+ self.f_a_proj = nn.Linear(self.hidden_size, self.head_dim, bias=False)
317
+ self.f_b_proj = nn.Linear(self.head_dim, projection_size, bias=False)
318
+
319
+ self.dt_bias = nn.Parameter(
320
+ torch.empty(projection_size, dtype=torch.float32))
321
+
322
+ self.b_proj = nn.Linear(self.hidden_size, self.num_heads, bias=False)
323
+
324
+ if self.use_full_proj:
325
+ self.g_proj = nn.Linear(self.hidden_size, projection_size, bias=False)
326
+ else:
327
+ self.g_a_proj = nn.Linear(self.hidden_size, self.head_dim, bias=False)
328
+ self.g_b_proj = nn.Linear(self.head_dim, projection_size, bias=False)
329
+
330
+ self.o_norm = FusedRMSNormGated(
331
+ self.head_dim, eps=config.rms_norm_eps, activation='sigmoid')
332
+ self.o_proj = nn.Linear(projection_size, self.hidden_size, bias=False)
333
+
334
+ def forward(
335
+ self,
336
+ hidden_states: torch.Tensor,
337
+ attention_mask: torch.Tensor | None = None,
338
+ past_key_values: SolarOpen2DynamicCache | None = None,
339
+ **kwargs: Unpack[dict],
340
+ ) -> tuple[torch.Tensor, torch.Tensor | None, Cache | None]:
341
+ if attention_mask is not None:
342
+ if attention_mask.dim() != 2:
343
+ attention_mask = kwargs.get("padding_mask")
344
+
345
+ if attention_mask is not None and attention_mask.dim() != 2:
346
+ raise ValueError(
347
+ "attention_mask must be a 0-1 matrix of shape [batch_size, seq_len] "
348
+ "(0 = padding). 3D masks are not supported here.",
349
+ )
350
+ use_cache = past_key_values is not None
351
+ batch_size, q_len, _ = hidden_states.shape
352
+ mode = 'fused_recurrent' if q_len <= 64 else self.mode
353
+ if self.training:
354
+ assert mode == 'chunk', "Only chunk mode is supported in training."
355
+
356
+ cu_seqlens = kwargs.get('cu_seqlens')
357
+ indices = None
358
+ if attention_mask is not None:
359
+ indices, cu_seqlens, _ = get_unpad_data(attention_mask[:, -q_len:])
360
+ hidden_states = index_first_axis(
361
+ rearrange(hidden_states, "b s ... -> (b s) ..."), indices).unsqueeze(0)
362
+
363
+ conv_state_q, conv_state_k, conv_state_v = None, None, None
364
+ recurrent_state = None
365
+ if past_key_values is not None:
366
+ if past_key_values.conv_states[self.layer_idx] is not None:
367
+ conv_state_q, conv_state_k, conv_state_v = past_key_values.conv_states[
368
+ self.layer_idx]
369
+ recurrent_state = past_key_values.recurrent_states[self.layer_idx]
370
+ q, conv_state_q = self.q_conv1d(
371
+ x=self.q_proj(hidden_states),
372
+ cache=conv_state_q,
373
+ output_final_state=use_cache,
374
+ cu_seqlens=cu_seqlens,
375
+ )
376
+ k, conv_state_k = self.k_conv1d(
377
+ x=self.k_proj(hidden_states),
378
+ cache=conv_state_k,
379
+ output_final_state=use_cache,
380
+ cu_seqlens=cu_seqlens,
381
+ )
382
+ v, conv_state_v = self.v_conv1d(
383
+ x=self.v_proj(hidden_states),
384
+ cache=conv_state_v,
385
+ output_final_state=use_cache,
386
+ cu_seqlens=cu_seqlens,
387
+ )
388
+ if self.use_full_proj:
389
+ g = self.f_proj(hidden_states)
390
+ else:
391
+ g = self.f_b_proj(self.f_a_proj(hidden_states))
392
+ q, k, g = (rearrange(x, "... (h d) -> ... h d", d=self.head_k_dim) for x in (q, k, g))
393
+ v = rearrange(v, '... (h d) -> ... h d', d=self.head_dim)
394
+
395
+ # repeat k/v heads if num_kv_heads < num_heads (GQA)
396
+ k = repeat_kv_linear(k, self.n_rep)
397
+ v = repeat_kv_linear(v, self.n_rep)
398
+
399
+ g = fused_kda_gate(g, self.A_log, dt_bias=self.dt_bias, lower_bound=self.gate_lower_bound)
400
+ beta = self.b_proj(hidden_states).sigmoid()
401
+ if self.allow_neg_eigval:
402
+ beta = beta * 2.0
403
+
404
+ if mode == 'chunk':
405
+ o, recurrent_state = chunk_kda(
406
+ q=q,
407
+ k=k,
408
+ v=v,
409
+ g=g,
410
+ beta=beta,
411
+ initial_state=recurrent_state,
412
+ output_final_state=True,
413
+ use_qk_l2norm_in_kernel=True,
414
+ cu_seqlens=cu_seqlens,
415
+ )
416
+ else:
417
+ o, recurrent_state = fused_recurrent_kda(
418
+ q=q,
419
+ k=k,
420
+ v=v,
421
+ g=g,
422
+ beta=beta,
423
+ initial_state=recurrent_state,
424
+ output_final_state=True,
425
+ use_qk_l2norm_in_kernel=True,
426
+ cu_seqlens=cu_seqlens,
427
+ )
428
+ if past_key_values is not None:
429
+ past_key_values.recurrent_states[self.layer_idx] = recurrent_state
430
+ past_key_values.conv_states[self.layer_idx] = (
431
+ conv_state_q, conv_state_k, conv_state_v)
432
+
433
+ if self.use_full_proj:
434
+ g = self.g_proj(hidden_states)
435
+ else:
436
+ g = self.g_b_proj(self.g_a_proj(hidden_states))
437
+ g = rearrange(g, '... (h d) -> ... h d', d=self.head_dim)
438
+ o = self.o_norm(o, g)
439
+
440
+ o = rearrange(o, 'b t h d -> b t (h d)')
441
+ o = self.o_proj(o)
442
+ if attention_mask is not None:
443
+ o = pad_input(o.squeeze(0), indices, batch_size, q_len)
444
+
445
+ return o, _
446
+
447
+
448
+ class SolarOpen2Attention(nn.Module):
449
+ """Multi-headed attention from 'Attention Is All You Need' paper"""
450
+
451
+ def __init__(self, config: SolarOpen2Config, layer_idx: Optional[int] = None):
452
+ super().__init__()
453
+ self.config = config
454
+ self.layer_idx = layer_idx
455
+ self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
456
+ self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
457
+ self.scaling = self.head_dim**-0.5
458
+ self.rope_scaling = config.rope_scaling
459
+ self.attention_dropout = config.attention_dropout
460
+ self.is_causal = True
461
+
462
+ self.q_proj = nn.Linear(
463
+ config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
464
+ )
465
+ self.k_proj = nn.Linear(
466
+ config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
467
+ )
468
+ self.v_proj = nn.Linear(
469
+ config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
470
+ )
471
+ self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim, config.hidden_size, bias=False)
472
+ self.use_qk_norm = config.use_qk_norm
473
+ if self.use_qk_norm:
474
+ self.q_norm = SolarOpen2RMSNorm(self.head_dim, eps=config.rms_norm_eps)
475
+ self.k_norm = SolarOpen2RMSNorm(self.head_dim, eps=config.rms_norm_eps)
476
+
477
+ self.use_gqa_gate = config.use_gqa_gate
478
+ if self.use_gqa_gate:
479
+ self.g_proj = nn.Linear(config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.use_gqa_gate_bias)
480
+
481
+ @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
482
+ def forward(
483
+ self,
484
+ hidden_states: torch.Tensor,
485
+ position_embeddings: tuple[torch.Tensor, torch.Tensor],
486
+ attention_mask: Optional[torch.Tensor],
487
+ past_key_values: Optional[SolarOpen2DynamicCache] = None,
488
+ cache_position: Optional[torch.LongTensor] = None,
489
+ **kwargs: Unpack[FlashAttentionKwargs],
490
+ ) -> tuple[torch.Tensor, Optional[torch.Tensor]]:
491
+ input_shape = hidden_states.shape[:-1]
492
+ hidden_shape = (*input_shape, -1, self.head_dim)
493
+
494
+ query_states = self.q_proj(hidden_states).view(hidden_shape)
495
+ key_states = self.k_proj(hidden_states).view(hidden_shape)
496
+ value_states = self.v_proj(hidden_states).view(hidden_shape)
497
+
498
+ if self.use_qk_norm: # main diff from Llama
499
+ query_states = self.q_norm(query_states)
500
+ key_states = self.k_norm(key_states)
501
+
502
+ query_states = query_states.transpose(1, 2)
503
+ key_states = key_states.transpose(1, 2)
504
+ value_states = value_states.transpose(1, 2)
505
+
506
+ cos, sin = position_embeddings
507
+ if self.config.use_rope:
508
+ query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
509
+
510
+ if past_key_values is not None:
511
+ # sin and cos are specific to RoPE models; position_ids needed for the static cache
512
+ cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
513
+ key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx, cache_kwargs)
514
+
515
+ attention_interface: Callable = eager_attention_forward
516
+ if self.config._attn_implementation != "eager":
517
+ attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
518
+
519
+ attn_output, attn_weights = attention_interface(
520
+ self,
521
+ query_states,
522
+ key_states,
523
+ value_states,
524
+ attention_mask,
525
+ dropout=0.0 if not self.training else self.attention_dropout,
526
+ scaling=self.scaling,
527
+ **kwargs,
528
+ )
529
+
530
+ attn_output = attn_output.reshape(*input_shape, -1).contiguous()
531
+ if self.use_gqa_gate:
532
+ gate = self.g_proj(hidden_states)
533
+ attn_output = attn_output * torch.sigmoid(gate)
534
+
535
+ attn_output = self.o_proj(attn_output)
536
+ return attn_output, attn_weights
537
+
538
+
539
+ class SolarOpen2MLP(nn.Module):
540
+ def __init__(self, config, hidden_size=None, intermediate_size=None):
541
+ super().__init__()
542
+ self.config = config
543
+ self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
544
+ self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
545
+
546
+ self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
547
+ self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
548
+ self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
549
+ self.act_fn = ACT2FN[config.hidden_act]
550
+
551
+ def forward(self, x):
552
+ down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
553
+ return down_proj
554
+
555
+
556
+ class SolarOpen2TopkRouter(nn.Module):
557
+ def __init__(self, config: SolarOpen2Config):
558
+ super().__init__()
559
+ self.config = config
560
+ self.top_k = config.num_experts_per_tok
561
+ self.n_routed_experts = config.n_routed_experts
562
+ self.routed_scaling_factor = config.routed_scaling_factor
563
+ self.n_group = config.n_group
564
+ self.topk_group = config.topk_group
565
+ self.norm_topk_prob = config.norm_topk_prob
566
+
567
+ self.weight = nn.Parameter(torch.empty((self.n_routed_experts, config.hidden_size)))
568
+ self.e_score_correction_bias = nn.Parameter(
569
+ torch.zeros((self.n_routed_experts), dtype=torch.float32))
570
+
571
+ @torch.no_grad()
572
+ def get_topk_indices(self, scores):
573
+ scores_for_choice = scores.view(-1, self.n_routed_experts) + self.e_score_correction_bias.unsqueeze(0)
574
+ group_scores = (
575
+ scores_for_choice.view(-1, self.n_group, self.n_routed_experts // self.n_group)
576
+ .topk(2, dim=-1)[0]
577
+ .sum(dim=-1)
578
+ )
579
+ group_idx = torch.topk(group_scores, k=self.topk_group, dim=-1, sorted=False)[1]
580
+ group_mask = torch.zeros_like(group_scores)
581
+ group_mask.scatter_(1, group_idx, 1)
582
+ score_mask = (
583
+ group_mask.unsqueeze(-1)
584
+ .expand(-1, self.n_group, self.n_routed_experts // self.n_group)
585
+ .reshape(-1, self.n_routed_experts)
586
+ )
587
+ scores_for_choice = scores_for_choice.masked_fill(~score_mask.bool(), 0.0)
588
+ topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
589
+ return topk_indices
590
+
591
+ def forward(self, hidden_states):
592
+ hidden_states = hidden_states.view(-1, self.config.hidden_size)
593
+ router_logits = F.linear(hidden_states.type(torch.float32), self.weight.type(torch.float32))
594
+ scores = router_logits.sigmoid()
595
+ topk_indices = self.get_topk_indices(scores)
596
+ topk_weights = scores.gather(1, topk_indices)
597
+ if self.norm_topk_prob:
598
+ denominator = topk_weights.sum(dim=-1, keepdim=True) + 1e-20
599
+ topk_weights /= denominator
600
+ topk_weights = topk_weights * self.routed_scaling_factor
601
+ return topk_indices, topk_weights
602
+
603
+
604
+ @use_kernel_forward_from_hub("RMSNorm")
605
+ class SolarOpen2RMSNorm(nn.Module):
606
+ def __init__(self, hidden_size, eps=1e-6):
607
+ """
608
+ SolarOpen2RMSNorm is equivalent to T5LayerNorm
609
+ """
610
+ super().__init__()
611
+ self.weight = nn.Parameter(torch.ones(hidden_size))
612
+ self.variance_epsilon = eps
613
+
614
+ def forward(self, hidden_states):
615
+ input_dtype = hidden_states.dtype
616
+ hidden_states = hidden_states.to(torch.float32)
617
+ variance = hidden_states.pow(2).mean(-1, keepdim=True)
618
+ hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
619
+ return self.weight * hidden_states.to(input_dtype)
620
+
621
+ def extra_repr(self):
622
+ return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
623
+
624
+
625
+ class SolarOpen2MoE(nn.Module):
626
+ """
627
+ A mixed expert module containing shared experts.
628
+ """
629
+
630
+ def __init__(self, config):
631
+ super().__init__()
632
+ self.config = config
633
+ self.experts = nn.ModuleList(
634
+ [
635
+ SolarOpen2MLP(config, intermediate_size=config.moe_intermediate_size)
636
+ for _ in range(config.n_routed_experts)
637
+ ]
638
+ )
639
+ self.gate = SolarOpen2TopkRouter(config)
640
+ self.shared_experts = SolarOpen2MLP(
641
+ config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
642
+ )
643
+
644
+ @torch.compiler.disable()
645
+ def moe(self, hidden_states: torch.Tensor, topk_indices: torch.Tensor, topk_weights: torch.Tensor):
646
+ r"""
647
+ MoE forward pass that only executes selected experts.
648
+ Uses @torch.compiler.disable() to allow dynamic shape operations.
649
+ Requires --enforce-eager flag when serving with vLLM.
650
+ """
651
+ final_hidden_states = torch.zeros_like(hidden_states)
652
+
653
+ for expert_idx in range(len(self.experts)):
654
+ expert = self.experts[expert_idx]
655
+
656
+ # Find positions where this expert was selected
657
+ batch_idx, topk_pos = torch.where(topk_indices == expert_idx)
658
+
659
+ if batch_idx.numel() == 0:
660
+ continue
661
+
662
+ # Extract only the tokens routed to this expert
663
+ expert_input = hidden_states[batch_idx]
664
+ expert_output = expert(expert_input)
665
+
666
+ # Apply weights and accumulate results
667
+ weights = topk_weights[batch_idx, topk_pos].unsqueeze(-1)
668
+ final_hidden_states.index_add_(0, batch_idx, (expert_output * weights).to(hidden_states.dtype))
669
+
670
+ return final_hidden_states
671
+
672
+ def forward(self, hidden_states):
673
+ residuals = hidden_states
674
+ orig_shape = hidden_states.shape
675
+ topk_indices, topk_weights = self.gate(hidden_states)
676
+ hidden_states = hidden_states.view(-1, hidden_states.shape[-1])
677
+ hidden_states = self.moe(hidden_states, topk_indices, topk_weights).view(*orig_shape)
678
+ hidden_states = hidden_states + self.shared_experts(residuals)
679
+ return hidden_states
680
+
681
+
682
+ class SolarOpen2HyperConnection(nn.Module):
683
+ """Hyper-Connections module for SolarOpen2 (pure PyTorch, no Triton).
684
+
685
+ Learnable replacement for residual connections that expands the hidden
686
+ state to N copies and uses learned mixing (width) and combining (depth)
687
+ connections.
688
+
689
+ Reference: https://arxiv.org/abs/2409.19606
690
+ """
691
+
692
+ def __init__(self, config: SolarOpen2Config, layer_idx: int):
693
+ super().__init__()
694
+ dim = config.hidden_size
695
+ rate = config.hc_rate
696
+ self.rate = rate
697
+ self.dynamic = config.hc_dynamic
698
+ self.use_tanh = config.hc_use_tanh
699
+ self.use_sigmoid = getattr(config, "hc_use_sigmoid", False)
700
+
701
+ self.static_alpha = nn.Parameter(torch.zeros(rate, rate + 1))
702
+ self.static_beta = nn.Parameter(torch.ones(rate))
703
+
704
+ if self.dynamic:
705
+ self.dynamic_alpha_fn = nn.Parameter(torch.zeros(dim, rate + 1))
706
+ self.dynamic_alpha_scale = nn.Parameter(torch.ones(1) * 0.01)
707
+ self.dynamic_beta_fn = nn.Parameter(torch.zeros(dim))
708
+ self.dynamic_beta_scale = nn.Parameter(torch.ones(1) * 0.01)
709
+
710
+ self.norm = SolarOpen2RMSNorm(dim, eps=config.rms_norm_eps)
711
+
712
+ def width_connection(self, h: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
713
+ """Split h into (h_in, h_res, beta) via learned width mixing.
714
+
715
+ Args:
716
+ h: (..., N, D) hidden states with N copies.
717
+
718
+ Returns:
719
+ h_in: (..., D) input for the sublayer.
720
+ h_res: (..., N, D) residual for depth connection.
721
+ beta: (N,) or (..., N) depth connection weights.
722
+ """
723
+ if self.dynamic:
724
+ norm_h = self.norm(h)
725
+ # norm_h: (..., N, D), dynamic_alpha_fn: (D, rate+1)
726
+ wc_weight = norm_h @ self.dynamic_alpha_fn # (..., N, rate+1)
727
+ if self.use_tanh:
728
+ wc_weight = torch.tanh(wc_weight)
729
+ alpha = wc_weight * self.dynamic_alpha_scale + self.static_alpha
730
+ # dc_weight: (..., N)
731
+ dc_weight = (norm_h * self.dynamic_beta_fn).sum(-1)
732
+ if self.use_tanh:
733
+ dc_weight = torch.tanh(dc_weight)
734
+ beta = dc_weight * self.dynamic_beta_scale + self.static_beta
735
+ else:
736
+ alpha = self.static_alpha
737
+ beta = self.static_beta
738
+
739
+ if self.use_sigmoid:
740
+ alpha = torch.sigmoid(alpha)
741
+ beta = 2.0 * torch.sigmoid(beta)
742
+
743
+ # Mix: alpha^T @ h → (..., N+1, D) — float32 accumulators to match
744
+ # the Triton kernels in TorchTitan which compute in fp32 internally.
745
+ orig_dtype = h.dtype
746
+ alpha_t = alpha.float().transpose(-2, -1) # (..., N+1, N) or (N+1, N)
747
+ mix_h = torch.matmul(alpha_t, h.float()) # (..., N+1, D)
748
+ h_in = mix_h[..., 0, :].to(orig_dtype) # (..., D)
749
+ h_res = mix_h[..., 1:, :].to(orig_dtype) # (..., N, D)
750
+ return h_in, h_res, beta
751
+
752
+ def depth_connection(self, h_res: torch.Tensor, h_out: torch.Tensor, beta: torch.Tensor) -> torch.Tensor:
753
+ """Combine sublayer output with residual via learned depth connection.
754
+
755
+ Args:
756
+ h_res: (..., N, D) residual from width connection.
757
+ h_out: (..., D) sublayer output.
758
+ beta: (N,) or (..., N) depth connection weights.
759
+
760
+ Returns:
761
+ h: (..., N, D) combined hidden states.
762
+ """
763
+ # float32 accumulators to match Triton kernels in TorchTitan.
764
+ return (h_out.float().unsqueeze(-2) * beta.float().unsqueeze(-1)
765
+ + h_res.float()).to(h_out.dtype)
766
+
767
+
768
+ class SolarOpen2DecoderLayer(GradientCheckpointingLayer):
769
+ def __init__(self, config: SolarOpen2Config, layer_idx: int):
770
+ super().__init__()
771
+ self.hidden_size = config.hidden_size
772
+
773
+ # Hyper-Connections
774
+ self.use_hc = getattr(config, "hc_rate", 0) > 0
775
+ if self.use_hc:
776
+ self.attn_hc = SolarOpen2HyperConnection(config, layer_idx=2 * layer_idx)
777
+ self.ffn_hc = SolarOpen2HyperConnection(config, layer_idx=2 * layer_idx + 1)
778
+
779
+ # Determine if this layer uses GQA: gqa_layers takes priority over gqa_interval
780
+ if config.gqa_layers is not None:
781
+ use_gqa = layer_idx in config.gqa_layers
782
+ else:
783
+ use_gqa = (layer_idx + 1) % config.gqa_interval == 0
784
+
785
+ if use_gqa:
786
+ self.is_linear_attn = False
787
+ self.self_attn = SolarOpen2Attention(config=config, layer_idx=layer_idx)
788
+ else:
789
+ self.is_linear_attn = True
790
+ self.self_attn = SolarOpen2LinearAttention(config=config, layer_idx=layer_idx)
791
+
792
+ if layer_idx >= config.first_k_dense_replace:
793
+ self.mlp = SolarOpen2MoE(config)
794
+ else:
795
+ self.mlp = SolarOpen2MLP(config)
796
+
797
+ self.input_layernorm = SolarOpen2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
798
+ self.post_attention_layernorm = SolarOpen2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
799
+
800
+ @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
801
+ def forward(
802
+ self,
803
+ hidden_states: torch.Tensor,
804
+ attention_mask: Optional[torch.Tensor] = None,
805
+ position_ids: Optional[torch.LongTensor] = None,
806
+ past_key_values: Optional[SolarOpen2DynamicCache] = None,
807
+ use_cache: Optional[bool] = False,
808
+ cache_position: Optional[torch.LongTensor] = None,
809
+ position_embeddings: Optional[tuple[torch.Tensor, torch.Tensor]] = None, # necessary, but kept here for BC
810
+ **kwargs: Unpack[TransformersKwargs],
811
+ ) -> torch.Tensor:
812
+ if self.use_hc:
813
+ # ═══ Attention with Hyper-Connections ═══
814
+ h_in, h_res, attn_beta = self.attn_hc.width_connection(hidden_states)
815
+ h_in = self.input_layernorm(h_in)
816
+ h_in, _ = self.self_attn(
817
+ hidden_states=h_in,
818
+ attention_mask=attention_mask,
819
+ position_ids=position_ids,
820
+ past_key_values=past_key_values,
821
+ use_cache=use_cache,
822
+ cache_position=cache_position,
823
+ position_embeddings=position_embeddings,
824
+ **kwargs,
825
+ )
826
+ hidden_states = self.attn_hc.depth_connection(h_res, h_in, attn_beta)
827
+
828
+ # ═══ FFN/MoE with Hyper-Connections ═══
829
+ ffn_h_in, ffn_h_res, ffn_beta = self.ffn_hc.width_connection(hidden_states)
830
+ ffn_h_in = self.post_attention_layernorm(ffn_h_in)
831
+ ffn_h_in = self.mlp(ffn_h_in)
832
+ hidden_states = self.ffn_hc.depth_connection(ffn_h_res, ffn_h_in, ffn_beta)
833
+ return hidden_states
834
+ else:
835
+ residual = hidden_states
836
+ hidden_states = self.input_layernorm(hidden_states)
837
+ # Self Attention
838
+ hidden_states, _ = self.self_attn(
839
+ hidden_states=hidden_states,
840
+ attention_mask=attention_mask,
841
+ position_ids=position_ids,
842
+ past_key_values=past_key_values,
843
+ use_cache=use_cache,
844
+ cache_position=cache_position,
845
+ position_embeddings=position_embeddings,
846
+ **kwargs,
847
+ )
848
+ hidden_states = residual + hidden_states
849
+
850
+ # Fully Connected
851
+ residual = hidden_states
852
+ hidden_states = self.post_attention_layernorm(hidden_states)
853
+ hidden_states = self.mlp(hidden_states)
854
+ hidden_states = residual + hidden_states
855
+ return hidden_states
856
+
857
+
858
+ @auto_docstring
859
+ class SolarOpen2PreTrainedModel(PreTrainedModel):
860
+ config: SolarOpen2Config
861
+ base_model_prefix = "model"
862
+ supports_gradient_checkpointing = True
863
+ _no_split_modules = ["SolarOpen2DecoderLayer"]
864
+ _skip_keys_device_placement = ["past_key_values"]
865
+ _supports_flash_attn = True
866
+ _supports_sdpa = True
867
+ _supports_flex_attn = True
868
+ _can_compile_fullgraph = False
869
+ _supports_attention_backend = True
870
+ _can_record_outputs = {
871
+ "hidden_states": SolarOpen2DecoderLayer,
872
+ "attentions": SolarOpen2Attention,
873
+ }
874
+
875
+ def _init_weights(self, module):
876
+ super()._init_weights(module)
877
+ if isinstance(module, SolarOpen2TopkRouter):
878
+ module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
879
+
880
+
881
+ class SolarOpen2RotaryEmbedding(nn.Module):
882
+ inv_freq: torch.Tensor # fix linting for `register_buffer`
883
+
884
+ def __init__(self, config: SolarOpen2Config, device=None):
885
+ super().__init__()
886
+ # BC: "rope_type" was originally "type"
887
+ if hasattr(config, "rope_scaling") and isinstance(config.rope_scaling, dict):
888
+ self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
889
+ else:
890
+ self.rope_type = "default"
891
+ self.max_seq_len_cached = config.max_position_embeddings
892
+ self.original_max_seq_len = config.max_position_embeddings
893
+
894
+ self.config = config
895
+ self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
896
+
897
+ inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device)
898
+ self.register_buffer("inv_freq", inv_freq, persistent=False)
899
+ self.original_inv_freq = self.inv_freq
900
+
901
+ @torch.no_grad()
902
+ @dynamic_rope_update # power user: used with advanced RoPE types (e.g. dynamic rope)
903
+ def forward(self, x, position_ids):
904
+ inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
905
+ position_ids_expanded = position_ids[:, None, :].float()
906
+
907
+ device_type = x.device.type if isinstance(x.device.type, str) and x.device.type != "mps" else "cpu"
908
+ with torch.autocast(device_type=device_type, enabled=False): # Force float32
909
+ freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
910
+ emb = torch.cat((freqs, freqs), dim=-1)
911
+ cos = emb.cos() * self.attention_scaling
912
+ sin = emb.sin() * self.attention_scaling
913
+
914
+ return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
915
+
916
+
917
+ @auto_docstring
918
+ class SolarOpen2Model(SolarOpen2PreTrainedModel):
919
+ _keys_to_ignore_on_load_unexpected = [r"model\.layers\.92.*", r"model\.layers\.46.*"]
920
+
921
+ def __init__(self, config: SolarOpen2Config):
922
+ super().__init__(config)
923
+ self.padding_idx = config.pad_token_id
924
+ self.vocab_size = config.vocab_size
925
+ self.hc_rate = getattr(config, "hc_rate", 0)
926
+
927
+ self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
928
+ self.layers = nn.ModuleList(
929
+ [SolarOpen2DecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
930
+ )
931
+ self.norm = SolarOpen2RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
932
+ self.rotary_emb = SolarOpen2RotaryEmbedding(config=config)
933
+ self.gradient_checkpointing = False
934
+
935
+ # Initialize weights and apply final processing
936
+ self.post_init()
937
+
938
+ def _update_linear_attn_mask(self, attention_mask, cache_position):
939
+ """
940
+ NOTE: Left-padding is used for linear attention mask.
941
+ No need for zeroing states when
942
+ 1. Cached forward
943
+ 2. Attending to all inputs
944
+ """
945
+ linear_attn_mask = attention_mask
946
+ if cache_position[0] > 0 or (attention_mask is not None and torch.all(attention_mask == 1)):
947
+ linear_attn_mask = None
948
+ return linear_attn_mask
949
+
950
+ @check_model_inputs
951
+ @auto_docstring
952
+ def forward(
953
+ self,
954
+ input_ids: Optional[torch.LongTensor] = None,
955
+ attention_mask: Optional[torch.Tensor] = None,
956
+ position_ids: Optional[torch.LongTensor] = None,
957
+ past_key_values: Optional[SolarOpen2DynamicCache] = None,
958
+ inputs_embeds: Optional[torch.FloatTensor] = None,
959
+ cache_position: Optional[torch.LongTensor] = None,
960
+ use_cache: Optional[bool] = None,
961
+ **kwargs: Unpack[TransformersKwargs],
962
+ ) -> BaseModelOutputWithPast:
963
+ if (input_ids is None) ^ (inputs_embeds is not None):
964
+ raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
965
+
966
+ if inputs_embeds is None:
967
+ inputs_embeds: torch.Tensor = self.embed_tokens(input_ids)
968
+
969
+ if use_cache and not isinstance(past_key_values, SolarOpen2DynamicCache):
970
+ past_key_values = SolarOpen2DynamicCache(config=self.config)
971
+
972
+ if cache_position is None:
973
+ past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
974
+ cache_position: torch.Tensor = torch.arange(
975
+ past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
976
+ )
977
+
978
+ if position_ids is None:
979
+ position_ids = cache_position.unsqueeze(0)
980
+
981
+ causal_mask = create_causal_mask(
982
+ config=self.config,
983
+ input_embeds=inputs_embeds,
984
+ attention_mask=attention_mask,
985
+ cache_position=cache_position,
986
+ past_key_values=past_key_values,
987
+ position_ids=position_ids,
988
+ )
989
+ linear_attn_mask = self._update_linear_attn_mask(attention_mask, cache_position)
990
+
991
+ hidden_states = inputs_embeds
992
+ position_embeddings = self.rotary_emb(hidden_states, position_ids)
993
+
994
+ # Hyper-Connections: expand (B, L, D) → (B, L, N, D) by replicating N times
995
+ if self.hc_rate > 0:
996
+ hidden_states = hidden_states.unsqueeze(-2).expand(
997
+ *hidden_states.shape[:-1], self.hc_rate, hidden_states.shape[-1]
998
+ ).contiguous()
999
+
1000
+ for decoder_layer in self.layers[: self.config.num_hidden_layers]:
1001
+ layer_mask = linear_attn_mask if decoder_layer.is_linear_attn else causal_mask
1002
+ hidden_states = decoder_layer(
1003
+ hidden_states,
1004
+ attention_mask=layer_mask,
1005
+ position_ids=position_ids,
1006
+ past_key_values=past_key_values,
1007
+ cache_position=cache_position,
1008
+ position_embeddings=position_embeddings,
1009
+ **kwargs,
1010
+ )
1011
+
1012
+ # Hyper-Connections: contract (B, L, N, D) → (B, L, D) by summing N copies
1013
+ if self.hc_rate > 0:
1014
+ hidden_states = hidden_states.sum(dim=-2)
1015
+
1016
+ hidden_states = self.norm(hidden_states)
1017
+ return BaseModelOutputWithPast(
1018
+ last_hidden_state=hidden_states,
1019
+ past_key_values=past_key_values,
1020
+ )
1021
+
1022
+
1023
+ @auto_docstring
1024
+ class SolarOpen2ForCausalLM(SolarOpen2PreTrainedModel, GenerationMixin):
1025
+ _tied_weights_keys = ["lm_head.weight"]
1026
+ _tp_plan = {"lm_head": "colwise_rep"}
1027
+ _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
1028
+
1029
+ def __init__(self, config):
1030
+ super().__init__(config)
1031
+ self.model = SolarOpen2Model(config)
1032
+ self.vocab_size = config.vocab_size
1033
+ self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
1034
+
1035
+ # Initialize weights and apply final processing
1036
+ self.post_init()
1037
+
1038
+ @can_return_tuple
1039
+ @auto_docstring
1040
+ def forward(
1041
+ self,
1042
+ input_ids: Optional[torch.LongTensor] = None,
1043
+ attention_mask: Optional[torch.Tensor] = None,
1044
+ position_ids: Optional[torch.LongTensor] = None,
1045
+ past_key_values: Optional[SolarOpen2DynamicCache] = None,
1046
+ inputs_embeds: Optional[torch.FloatTensor] = None,
1047
+ labels: Optional[torch.LongTensor] = None,
1048
+ use_cache: Optional[bool] = None,
1049
+ cache_position: Optional[torch.LongTensor] = None,
1050
+ logits_to_keep: Union[int, torch.Tensor] = 0,
1051
+ **kwargs: Unpack[TransformersKwargs],
1052
+ ) -> CausalLMOutputWithPast:
1053
+
1054
+ outputs: BaseModelOutputWithPast = self.model(
1055
+ input_ids=input_ids,
1056
+ attention_mask=attention_mask,
1057
+ position_ids=position_ids,
1058
+ past_key_values=past_key_values,
1059
+ inputs_embeds=inputs_embeds,
1060
+ use_cache=use_cache,
1061
+ cache_position=cache_position,
1062
+ **kwargs,
1063
+ )
1064
+
1065
+ hidden_states = outputs.last_hidden_state
1066
+ # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
1067
+ slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
1068
+ logits = self.lm_head(hidden_states[:, slice_indices, :])
1069
+
1070
+ loss = None
1071
+ if labels is not None:
1072
+ loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
1073
+
1074
+ return CausalLMOutputWithPast(
1075
+ loss=loss,
1076
+ logits=logits,
1077
+ past_key_values=outputs.past_key_values,
1078
+ hidden_states=outputs.hidden_states,
1079
+ attentions=outputs.attentions,
1080
+ )
1081
+
1082
+
1083
+ __all__ = ["SolarOpen2PreTrainedModel", "SolarOpen2Model", "SolarOpen2ForCausalLM"]
quantization_config.json ADDED
@@ -0,0 +1,2659 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "bits": 4,
3
+ "data_type": "int",
4
+ "group_size": 128,
5
+ "sym": true,
6
+ "iters": 1000,
7
+ "low_gpu_mem_usage": true,
8
+ "nsamples": 1024,
9
+ "autoround_version": "0.13.1",
10
+ "block_name_to_quantize": "model.layers",
11
+ "quant_method": "auto-round",
12
+ "packing_format": "auto_round:auto_gptq",
13
+ "extra_config": {
14
+ "model.layers.0.self_attn.q_proj": {
15
+ "bits": 16,
16
+ "data_type": "fp",
17
+ "act_data_type": "fp"
18
+ },
19
+ "model.layers.0.self_attn.k_proj": {
20
+ "bits": 16,
21
+ "data_type": "fp",
22
+ "act_data_type": "fp"
23
+ },
24
+ "model.layers.0.self_attn.v_proj": {
25
+ "bits": 16,
26
+ "data_type": "fp",
27
+ "act_data_type": "fp"
28
+ },
29
+ "model.layers.0.self_attn.o_proj": {
30
+ "bits": 16,
31
+ "data_type": "fp",
32
+ "act_data_type": "fp"
33
+ },
34
+ "model.layers.0.self_attn.g_proj": {
35
+ "bits": 16,
36
+ "data_type": "fp",
37
+ "act_data_type": "fp"
38
+ },
39
+ "model.layers.1.self_attn.q_proj": {
40
+ "bits": 16,
41
+ "data_type": "fp",
42
+ "act_data_type": "fp"
43
+ },
44
+ "model.layers.1.self_attn.k_proj": {
45
+ "bits": 16,
46
+ "data_type": "fp",
47
+ "act_data_type": "fp"
48
+ },
49
+ "model.layers.1.self_attn.v_proj": {
50
+ "bits": 16,
51
+ "data_type": "fp",
52
+ "act_data_type": "fp"
53
+ },
54
+ "model.layers.1.self_attn.f_a_proj": {
55
+ "bits": 16,
56
+ "data_type": "fp",
57
+ "act_data_type": "fp"
58
+ },
59
+ "model.layers.1.self_attn.f_b_proj": {
60
+ "bits": 16,
61
+ "data_type": "fp",
62
+ "act_data_type": "fp"
63
+ },
64
+ "model.layers.1.self_attn.b_proj": {
65
+ "bits": 16,
66
+ "data_type": "fp",
67
+ "act_data_type": "fp"
68
+ },
69
+ "model.layers.1.self_attn.g_a_proj": {
70
+ "bits": 16,
71
+ "data_type": "fp",
72
+ "act_data_type": "fp"
73
+ },
74
+ "model.layers.1.self_attn.g_b_proj": {
75
+ "bits": 16,
76
+ "data_type": "fp",
77
+ "act_data_type": "fp"
78
+ },
79
+ "model.layers.1.self_attn.o_proj": {
80
+ "bits": 16,
81
+ "data_type": "fp",
82
+ "act_data_type": "fp"
83
+ },
84
+ "model.layers.2.self_attn.q_proj": {
85
+ "bits": 16,
86
+ "data_type": "fp",
87
+ "act_data_type": "fp"
88
+ },
89
+ "model.layers.2.self_attn.k_proj": {
90
+ "bits": 16,
91
+ "data_type": "fp",
92
+ "act_data_type": "fp"
93
+ },
94
+ "model.layers.2.self_attn.v_proj": {
95
+ "bits": 16,
96
+ "data_type": "fp",
97
+ "act_data_type": "fp"
98
+ },
99
+ "model.layers.2.self_attn.f_a_proj": {
100
+ "bits": 16,
101
+ "data_type": "fp",
102
+ "act_data_type": "fp"
103
+ },
104
+ "model.layers.2.self_attn.f_b_proj": {
105
+ "bits": 16,
106
+ "data_type": "fp",
107
+ "act_data_type": "fp"
108
+ },
109
+ "model.layers.2.self_attn.b_proj": {
110
+ "bits": 16,
111
+ "data_type": "fp",
112
+ "act_data_type": "fp"
113
+ },
114
+ "model.layers.2.self_attn.g_a_proj": {
115
+ "bits": 16,
116
+ "data_type": "fp",
117
+ "act_data_type": "fp"
118
+ },
119
+ "model.layers.2.self_attn.g_b_proj": {
120
+ "bits": 16,
121
+ "data_type": "fp",
122
+ "act_data_type": "fp"
123
+ },
124
+ "model.layers.2.self_attn.o_proj": {
125
+ "bits": 16,
126
+ "data_type": "fp",
127
+ "act_data_type": "fp"
128
+ },
129
+ "model.layers.3.self_attn.q_proj": {
130
+ "bits": 16,
131
+ "data_type": "fp",
132
+ "act_data_type": "fp"
133
+ },
134
+ "model.layers.3.self_attn.k_proj": {
135
+ "bits": 16,
136
+ "data_type": "fp",
137
+ "act_data_type": "fp"
138
+ },
139
+ "model.layers.3.self_attn.v_proj": {
140
+ "bits": 16,
141
+ "data_type": "fp",
142
+ "act_data_type": "fp"
143
+ },
144
+ "model.layers.3.self_attn.f_a_proj": {
145
+ "bits": 16,
146
+ "data_type": "fp",
147
+ "act_data_type": "fp"
148
+ },
149
+ "model.layers.3.self_attn.f_b_proj": {
150
+ "bits": 16,
151
+ "data_type": "fp",
152
+ "act_data_type": "fp"
153
+ },
154
+ "model.layers.3.self_attn.b_proj": {
155
+ "bits": 16,
156
+ "data_type": "fp",
157
+ "act_data_type": "fp"
158
+ },
159
+ "model.layers.3.self_attn.g_a_proj": {
160
+ "bits": 16,
161
+ "data_type": "fp",
162
+ "act_data_type": "fp"
163
+ },
164
+ "model.layers.3.self_attn.g_b_proj": {
165
+ "bits": 16,
166
+ "data_type": "fp",
167
+ "act_data_type": "fp"
168
+ },
169
+ "model.layers.3.self_attn.o_proj": {
170
+ "bits": 16,
171
+ "data_type": "fp",
172
+ "act_data_type": "fp"
173
+ },
174
+ "model.layers.4.self_attn.q_proj": {
175
+ "bits": 16,
176
+ "data_type": "fp",
177
+ "act_data_type": "fp"
178
+ },
179
+ "model.layers.4.self_attn.k_proj": {
180
+ "bits": 16,
181
+ "data_type": "fp",
182
+ "act_data_type": "fp"
183
+ },
184
+ "model.layers.4.self_attn.v_proj": {
185
+ "bits": 16,
186
+ "data_type": "fp",
187
+ "act_data_type": "fp"
188
+ },
189
+ "model.layers.4.self_attn.o_proj": {
190
+ "bits": 16,
191
+ "data_type": "fp",
192
+ "act_data_type": "fp"
193
+ },
194
+ "model.layers.4.self_attn.g_proj": {
195
+ "bits": 16,
196
+ "data_type": "fp",
197
+ "act_data_type": "fp"
198
+ },
199
+ "model.layers.5.self_attn.q_proj": {
200
+ "bits": 16,
201
+ "data_type": "fp",
202
+ "act_data_type": "fp"
203
+ },
204
+ "model.layers.5.self_attn.k_proj": {
205
+ "bits": 16,
206
+ "data_type": "fp",
207
+ "act_data_type": "fp"
208
+ },
209
+ "model.layers.5.self_attn.v_proj": {
210
+ "bits": 16,
211
+ "data_type": "fp",
212
+ "act_data_type": "fp"
213
+ },
214
+ "model.layers.5.self_attn.f_a_proj": {
215
+ "bits": 16,
216
+ "data_type": "fp",
217
+ "act_data_type": "fp"
218
+ },
219
+ "model.layers.5.self_attn.f_b_proj": {
220
+ "bits": 16,
221
+ "data_type": "fp",
222
+ "act_data_type": "fp"
223
+ },
224
+ "model.layers.5.self_attn.b_proj": {
225
+ "bits": 16,
226
+ "data_type": "fp",
227
+ "act_data_type": "fp"
228
+ },
229
+ "model.layers.5.self_attn.g_a_proj": {
230
+ "bits": 16,
231
+ "data_type": "fp",
232
+ "act_data_type": "fp"
233
+ },
234
+ "model.layers.5.self_attn.g_b_proj": {
235
+ "bits": 16,
236
+ "data_type": "fp",
237
+ "act_data_type": "fp"
238
+ },
239
+ "model.layers.5.self_attn.o_proj": {
240
+ "bits": 16,
241
+ "data_type": "fp",
242
+ "act_data_type": "fp"
243
+ },
244
+ "model.layers.6.self_attn.q_proj": {
245
+ "bits": 16,
246
+ "data_type": "fp",
247
+ "act_data_type": "fp"
248
+ },
249
+ "model.layers.6.self_attn.k_proj": {
250
+ "bits": 16,
251
+ "data_type": "fp",
252
+ "act_data_type": "fp"
253
+ },
254
+ "model.layers.6.self_attn.v_proj": {
255
+ "bits": 16,
256
+ "data_type": "fp",
257
+ "act_data_type": "fp"
258
+ },
259
+ "model.layers.6.self_attn.f_a_proj": {
260
+ "bits": 16,
261
+ "data_type": "fp",
262
+ "act_data_type": "fp"
263
+ },
264
+ "model.layers.6.self_attn.f_b_proj": {
265
+ "bits": 16,
266
+ "data_type": "fp",
267
+ "act_data_type": "fp"
268
+ },
269
+ "model.layers.6.self_attn.b_proj": {
270
+ "bits": 16,
271
+ "data_type": "fp",
272
+ "act_data_type": "fp"
273
+ },
274
+ "model.layers.6.self_attn.g_a_proj": {
275
+ "bits": 16,
276
+ "data_type": "fp",
277
+ "act_data_type": "fp"
278
+ },
279
+ "model.layers.6.self_attn.g_b_proj": {
280
+ "bits": 16,
281
+ "data_type": "fp",
282
+ "act_data_type": "fp"
283
+ },
284
+ "model.layers.6.self_attn.o_proj": {
285
+ "bits": 16,
286
+ "data_type": "fp",
287
+ "act_data_type": "fp"
288
+ },
289
+ "model.layers.7.self_attn.q_proj": {
290
+ "bits": 16,
291
+ "data_type": "fp",
292
+ "act_data_type": "fp"
293
+ },
294
+ "model.layers.7.self_attn.k_proj": {
295
+ "bits": 16,
296
+ "data_type": "fp",
297
+ "act_data_type": "fp"
298
+ },
299
+ "model.layers.7.self_attn.v_proj": {
300
+ "bits": 16,
301
+ "data_type": "fp",
302
+ "act_data_type": "fp"
303
+ },
304
+ "model.layers.7.self_attn.f_a_proj": {
305
+ "bits": 16,
306
+ "data_type": "fp",
307
+ "act_data_type": "fp"
308
+ },
309
+ "model.layers.7.self_attn.f_b_proj": {
310
+ "bits": 16,
311
+ "data_type": "fp",
312
+ "act_data_type": "fp"
313
+ },
314
+ "model.layers.7.self_attn.b_proj": {
315
+ "bits": 16,
316
+ "data_type": "fp",
317
+ "act_data_type": "fp"
318
+ },
319
+ "model.layers.7.self_attn.g_a_proj": {
320
+ "bits": 16,
321
+ "data_type": "fp",
322
+ "act_data_type": "fp"
323
+ },
324
+ "model.layers.7.self_attn.g_b_proj": {
325
+ "bits": 16,
326
+ "data_type": "fp",
327
+ "act_data_type": "fp"
328
+ },
329
+ "model.layers.7.self_attn.o_proj": {
330
+ "bits": 16,
331
+ "data_type": "fp",
332
+ "act_data_type": "fp"
333
+ },
334
+ "model.layers.8.self_attn.q_proj": {
335
+ "bits": 16,
336
+ "data_type": "fp",
337
+ "act_data_type": "fp"
338
+ },
339
+ "model.layers.8.self_attn.k_proj": {
340
+ "bits": 16,
341
+ "data_type": "fp",
342
+ "act_data_type": "fp"
343
+ },
344
+ "model.layers.8.self_attn.v_proj": {
345
+ "bits": 16,
346
+ "data_type": "fp",
347
+ "act_data_type": "fp"
348
+ },
349
+ "model.layers.8.self_attn.o_proj": {
350
+ "bits": 16,
351
+ "data_type": "fp",
352
+ "act_data_type": "fp"
353
+ },
354
+ "model.layers.8.self_attn.g_proj": {
355
+ "bits": 16,
356
+ "data_type": "fp",
357
+ "act_data_type": "fp"
358
+ },
359
+ "model.layers.9.self_attn.q_proj": {
360
+ "bits": 16,
361
+ "data_type": "fp",
362
+ "act_data_type": "fp"
363
+ },
364
+ "model.layers.9.self_attn.k_proj": {
365
+ "bits": 16,
366
+ "data_type": "fp",
367
+ "act_data_type": "fp"
368
+ },
369
+ "model.layers.9.self_attn.v_proj": {
370
+ "bits": 16,
371
+ "data_type": "fp",
372
+ "act_data_type": "fp"
373
+ },
374
+ "model.layers.9.self_attn.f_a_proj": {
375
+ "bits": 16,
376
+ "data_type": "fp",
377
+ "act_data_type": "fp"
378
+ },
379
+ "model.layers.9.self_attn.f_b_proj": {
380
+ "bits": 16,
381
+ "data_type": "fp",
382
+ "act_data_type": "fp"
383
+ },
384
+ "model.layers.9.self_attn.b_proj": {
385
+ "bits": 16,
386
+ "data_type": "fp",
387
+ "act_data_type": "fp"
388
+ },
389
+ "model.layers.9.self_attn.g_a_proj": {
390
+ "bits": 16,
391
+ "data_type": "fp",
392
+ "act_data_type": "fp"
393
+ },
394
+ "model.layers.9.self_attn.g_b_proj": {
395
+ "bits": 16,
396
+ "data_type": "fp",
397
+ "act_data_type": "fp"
398
+ },
399
+ "model.layers.9.self_attn.o_proj": {
400
+ "bits": 16,
401
+ "data_type": "fp",
402
+ "act_data_type": "fp"
403
+ },
404
+ "model.layers.10.self_attn.q_proj": {
405
+ "bits": 16,
406
+ "data_type": "fp",
407
+ "act_data_type": "fp"
408
+ },
409
+ "model.layers.10.self_attn.k_proj": {
410
+ "bits": 16,
411
+ "data_type": "fp",
412
+ "act_data_type": "fp"
413
+ },
414
+ "model.layers.10.self_attn.v_proj": {
415
+ "bits": 16,
416
+ "data_type": "fp",
417
+ "act_data_type": "fp"
418
+ },
419
+ "model.layers.10.self_attn.f_a_proj": {
420
+ "bits": 16,
421
+ "data_type": "fp",
422
+ "act_data_type": "fp"
423
+ },
424
+ "model.layers.10.self_attn.f_b_proj": {
425
+ "bits": 16,
426
+ "data_type": "fp",
427
+ "act_data_type": "fp"
428
+ },
429
+ "model.layers.10.self_attn.b_proj": {
430
+ "bits": 16,
431
+ "data_type": "fp",
432
+ "act_data_type": "fp"
433
+ },
434
+ "model.layers.10.self_attn.g_a_proj": {
435
+ "bits": 16,
436
+ "data_type": "fp",
437
+ "act_data_type": "fp"
438
+ },
439
+ "model.layers.10.self_attn.g_b_proj": {
440
+ "bits": 16,
441
+ "data_type": "fp",
442
+ "act_data_type": "fp"
443
+ },
444
+ "model.layers.10.self_attn.o_proj": {
445
+ "bits": 16,
446
+ "data_type": "fp",
447
+ "act_data_type": "fp"
448
+ },
449
+ "model.layers.11.self_attn.q_proj": {
450
+ "bits": 16,
451
+ "data_type": "fp",
452
+ "act_data_type": "fp"
453
+ },
454
+ "model.layers.11.self_attn.k_proj": {
455
+ "bits": 16,
456
+ "data_type": "fp",
457
+ "act_data_type": "fp"
458
+ },
459
+ "model.layers.11.self_attn.v_proj": {
460
+ "bits": 16,
461
+ "data_type": "fp",
462
+ "act_data_type": "fp"
463
+ },
464
+ "model.layers.11.self_attn.f_a_proj": {
465
+ "bits": 16,
466
+ "data_type": "fp",
467
+ "act_data_type": "fp"
468
+ },
469
+ "model.layers.11.self_attn.f_b_proj": {
470
+ "bits": 16,
471
+ "data_type": "fp",
472
+ "act_data_type": "fp"
473
+ },
474
+ "model.layers.11.self_attn.b_proj": {
475
+ "bits": 16,
476
+ "data_type": "fp",
477
+ "act_data_type": "fp"
478
+ },
479
+ "model.layers.11.self_attn.g_a_proj": {
480
+ "bits": 16,
481
+ "data_type": "fp",
482
+ "act_data_type": "fp"
483
+ },
484
+ "model.layers.11.self_attn.g_b_proj": {
485
+ "bits": 16,
486
+ "data_type": "fp",
487
+ "act_data_type": "fp"
488
+ },
489
+ "model.layers.11.self_attn.o_proj": {
490
+ "bits": 16,
491
+ "data_type": "fp",
492
+ "act_data_type": "fp"
493
+ },
494
+ "model.layers.12.self_attn.q_proj": {
495
+ "bits": 16,
496
+ "data_type": "fp",
497
+ "act_data_type": "fp"
498
+ },
499
+ "model.layers.12.self_attn.k_proj": {
500
+ "bits": 16,
501
+ "data_type": "fp",
502
+ "act_data_type": "fp"
503
+ },
504
+ "model.layers.12.self_attn.v_proj": {
505
+ "bits": 16,
506
+ "data_type": "fp",
507
+ "act_data_type": "fp"
508
+ },
509
+ "model.layers.12.self_attn.o_proj": {
510
+ "bits": 16,
511
+ "data_type": "fp",
512
+ "act_data_type": "fp"
513
+ },
514
+ "model.layers.12.self_attn.g_proj": {
515
+ "bits": 16,
516
+ "data_type": "fp",
517
+ "act_data_type": "fp"
518
+ },
519
+ "model.layers.13.self_attn.q_proj": {
520
+ "bits": 16,
521
+ "data_type": "fp",
522
+ "act_data_type": "fp"
523
+ },
524
+ "model.layers.13.self_attn.k_proj": {
525
+ "bits": 16,
526
+ "data_type": "fp",
527
+ "act_data_type": "fp"
528
+ },
529
+ "model.layers.13.self_attn.v_proj": {
530
+ "bits": 16,
531
+ "data_type": "fp",
532
+ "act_data_type": "fp"
533
+ },
534
+ "model.layers.13.self_attn.f_a_proj": {
535
+ "bits": 16,
536
+ "data_type": "fp",
537
+ "act_data_type": "fp"
538
+ },
539
+ "model.layers.13.self_attn.f_b_proj": {
540
+ "bits": 16,
541
+ "data_type": "fp",
542
+ "act_data_type": "fp"
543
+ },
544
+ "model.layers.13.self_attn.b_proj": {
545
+ "bits": 16,
546
+ "data_type": "fp",
547
+ "act_data_type": "fp"
548
+ },
549
+ "model.layers.13.self_attn.g_a_proj": {
550
+ "bits": 16,
551
+ "data_type": "fp",
552
+ "act_data_type": "fp"
553
+ },
554
+ "model.layers.13.self_attn.g_b_proj": {
555
+ "bits": 16,
556
+ "data_type": "fp",
557
+ "act_data_type": "fp"
558
+ },
559
+ "model.layers.13.self_attn.o_proj": {
560
+ "bits": 16,
561
+ "data_type": "fp",
562
+ "act_data_type": "fp"
563
+ },
564
+ "model.layers.14.self_attn.q_proj": {
565
+ "bits": 16,
566
+ "data_type": "fp",
567
+ "act_data_type": "fp"
568
+ },
569
+ "model.layers.14.self_attn.k_proj": {
570
+ "bits": 16,
571
+ "data_type": "fp",
572
+ "act_data_type": "fp"
573
+ },
574
+ "model.layers.14.self_attn.v_proj": {
575
+ "bits": 16,
576
+ "data_type": "fp",
577
+ "act_data_type": "fp"
578
+ },
579
+ "model.layers.14.self_attn.f_a_proj": {
580
+ "bits": 16,
581
+ "data_type": "fp",
582
+ "act_data_type": "fp"
583
+ },
584
+ "model.layers.14.self_attn.f_b_proj": {
585
+ "bits": 16,
586
+ "data_type": "fp",
587
+ "act_data_type": "fp"
588
+ },
589
+ "model.layers.14.self_attn.b_proj": {
590
+ "bits": 16,
591
+ "data_type": "fp",
592
+ "act_data_type": "fp"
593
+ },
594
+ "model.layers.14.self_attn.g_a_proj": {
595
+ "bits": 16,
596
+ "data_type": "fp",
597
+ "act_data_type": "fp"
598
+ },
599
+ "model.layers.14.self_attn.g_b_proj": {
600
+ "bits": 16,
601
+ "data_type": "fp",
602
+ "act_data_type": "fp"
603
+ },
604
+ "model.layers.14.self_attn.o_proj": {
605
+ "bits": 16,
606
+ "data_type": "fp",
607
+ "act_data_type": "fp"
608
+ },
609
+ "model.layers.15.self_attn.q_proj": {
610
+ "bits": 16,
611
+ "data_type": "fp",
612
+ "act_data_type": "fp"
613
+ },
614
+ "model.layers.15.self_attn.k_proj": {
615
+ "bits": 16,
616
+ "data_type": "fp",
617
+ "act_data_type": "fp"
618
+ },
619
+ "model.layers.15.self_attn.v_proj": {
620
+ "bits": 16,
621
+ "data_type": "fp",
622
+ "act_data_type": "fp"
623
+ },
624
+ "model.layers.15.self_attn.f_a_proj": {
625
+ "bits": 16,
626
+ "data_type": "fp",
627
+ "act_data_type": "fp"
628
+ },
629
+ "model.layers.15.self_attn.f_b_proj": {
630
+ "bits": 16,
631
+ "data_type": "fp",
632
+ "act_data_type": "fp"
633
+ },
634
+ "model.layers.15.self_attn.b_proj": {
635
+ "bits": 16,
636
+ "data_type": "fp",
637
+ "act_data_type": "fp"
638
+ },
639
+ "model.layers.15.self_attn.g_a_proj": {
640
+ "bits": 16,
641
+ "data_type": "fp",
642
+ "act_data_type": "fp"
643
+ },
644
+ "model.layers.15.self_attn.g_b_proj": {
645
+ "bits": 16,
646
+ "data_type": "fp",
647
+ "act_data_type": "fp"
648
+ },
649
+ "model.layers.15.self_attn.o_proj": {
650
+ "bits": 16,
651
+ "data_type": "fp",
652
+ "act_data_type": "fp"
653
+ },
654
+ "model.layers.16.self_attn.q_proj": {
655
+ "bits": 16,
656
+ "data_type": "fp",
657
+ "act_data_type": "fp"
658
+ },
659
+ "model.layers.16.self_attn.k_proj": {
660
+ "bits": 16,
661
+ "data_type": "fp",
662
+ "act_data_type": "fp"
663
+ },
664
+ "model.layers.16.self_attn.v_proj": {
665
+ "bits": 16,
666
+ "data_type": "fp",
667
+ "act_data_type": "fp"
668
+ },
669
+ "model.layers.16.self_attn.o_proj": {
670
+ "bits": 16,
671
+ "data_type": "fp",
672
+ "act_data_type": "fp"
673
+ },
674
+ "model.layers.16.self_attn.g_proj": {
675
+ "bits": 16,
676
+ "data_type": "fp",
677
+ "act_data_type": "fp"
678
+ },
679
+ "model.layers.17.self_attn.q_proj": {
680
+ "bits": 16,
681
+ "data_type": "fp",
682
+ "act_data_type": "fp"
683
+ },
684
+ "model.layers.17.self_attn.k_proj": {
685
+ "bits": 16,
686
+ "data_type": "fp",
687
+ "act_data_type": "fp"
688
+ },
689
+ "model.layers.17.self_attn.v_proj": {
690
+ "bits": 16,
691
+ "data_type": "fp",
692
+ "act_data_type": "fp"
693
+ },
694
+ "model.layers.17.self_attn.f_a_proj": {
695
+ "bits": 16,
696
+ "data_type": "fp",
697
+ "act_data_type": "fp"
698
+ },
699
+ "model.layers.17.self_attn.f_b_proj": {
700
+ "bits": 16,
701
+ "data_type": "fp",
702
+ "act_data_type": "fp"
703
+ },
704
+ "model.layers.17.self_attn.b_proj": {
705
+ "bits": 16,
706
+ "data_type": "fp",
707
+ "act_data_type": "fp"
708
+ },
709
+ "model.layers.17.self_attn.g_a_proj": {
710
+ "bits": 16,
711
+ "data_type": "fp",
712
+ "act_data_type": "fp"
713
+ },
714
+ "model.layers.17.self_attn.g_b_proj": {
715
+ "bits": 16,
716
+ "data_type": "fp",
717
+ "act_data_type": "fp"
718
+ },
719
+ "model.layers.17.self_attn.o_proj": {
720
+ "bits": 16,
721
+ "data_type": "fp",
722
+ "act_data_type": "fp"
723
+ },
724
+ "model.layers.18.self_attn.q_proj": {
725
+ "bits": 16,
726
+ "data_type": "fp",
727
+ "act_data_type": "fp"
728
+ },
729
+ "model.layers.18.self_attn.k_proj": {
730
+ "bits": 16,
731
+ "data_type": "fp",
732
+ "act_data_type": "fp"
733
+ },
734
+ "model.layers.18.self_attn.v_proj": {
735
+ "bits": 16,
736
+ "data_type": "fp",
737
+ "act_data_type": "fp"
738
+ },
739
+ "model.layers.18.self_attn.f_a_proj": {
740
+ "bits": 16,
741
+ "data_type": "fp",
742
+ "act_data_type": "fp"
743
+ },
744
+ "model.layers.18.self_attn.f_b_proj": {
745
+ "bits": 16,
746
+ "data_type": "fp",
747
+ "act_data_type": "fp"
748
+ },
749
+ "model.layers.18.self_attn.b_proj": {
750
+ "bits": 16,
751
+ "data_type": "fp",
752
+ "act_data_type": "fp"
753
+ },
754
+ "model.layers.18.self_attn.g_a_proj": {
755
+ "bits": 16,
756
+ "data_type": "fp",
757
+ "act_data_type": "fp"
758
+ },
759
+ "model.layers.18.self_attn.g_b_proj": {
760
+ "bits": 16,
761
+ "data_type": "fp",
762
+ "act_data_type": "fp"
763
+ },
764
+ "model.layers.18.self_attn.o_proj": {
765
+ "bits": 16,
766
+ "data_type": "fp",
767
+ "act_data_type": "fp"
768
+ },
769
+ "model.layers.19.self_attn.q_proj": {
770
+ "bits": 16,
771
+ "data_type": "fp",
772
+ "act_data_type": "fp"
773
+ },
774
+ "model.layers.19.self_attn.k_proj": {
775
+ "bits": 16,
776
+ "data_type": "fp",
777
+ "act_data_type": "fp"
778
+ },
779
+ "model.layers.19.self_attn.v_proj": {
780
+ "bits": 16,
781
+ "data_type": "fp",
782
+ "act_data_type": "fp"
783
+ },
784
+ "model.layers.19.self_attn.f_a_proj": {
785
+ "bits": 16,
786
+ "data_type": "fp",
787
+ "act_data_type": "fp"
788
+ },
789
+ "model.layers.19.self_attn.f_b_proj": {
790
+ "bits": 16,
791
+ "data_type": "fp",
792
+ "act_data_type": "fp"
793
+ },
794
+ "model.layers.19.self_attn.b_proj": {
795
+ "bits": 16,
796
+ "data_type": "fp",
797
+ "act_data_type": "fp"
798
+ },
799
+ "model.layers.19.self_attn.g_a_proj": {
800
+ "bits": 16,
801
+ "data_type": "fp",
802
+ "act_data_type": "fp"
803
+ },
804
+ "model.layers.19.self_attn.g_b_proj": {
805
+ "bits": 16,
806
+ "data_type": "fp",
807
+ "act_data_type": "fp"
808
+ },
809
+ "model.layers.19.self_attn.o_proj": {
810
+ "bits": 16,
811
+ "data_type": "fp",
812
+ "act_data_type": "fp"
813
+ },
814
+ "model.layers.20.self_attn.q_proj": {
815
+ "bits": 16,
816
+ "data_type": "fp",
817
+ "act_data_type": "fp"
818
+ },
819
+ "model.layers.20.self_attn.k_proj": {
820
+ "bits": 16,
821
+ "data_type": "fp",
822
+ "act_data_type": "fp"
823
+ },
824
+ "model.layers.20.self_attn.v_proj": {
825
+ "bits": 16,
826
+ "data_type": "fp",
827
+ "act_data_type": "fp"
828
+ },
829
+ "model.layers.20.self_attn.o_proj": {
830
+ "bits": 16,
831
+ "data_type": "fp",
832
+ "act_data_type": "fp"
833
+ },
834
+ "model.layers.20.self_attn.g_proj": {
835
+ "bits": 16,
836
+ "data_type": "fp",
837
+ "act_data_type": "fp"
838
+ },
839
+ "model.layers.21.self_attn.q_proj": {
840
+ "bits": 16,
841
+ "data_type": "fp",
842
+ "act_data_type": "fp"
843
+ },
844
+ "model.layers.21.self_attn.k_proj": {
845
+ "bits": 16,
846
+ "data_type": "fp",
847
+ "act_data_type": "fp"
848
+ },
849
+ "model.layers.21.self_attn.v_proj": {
850
+ "bits": 16,
851
+ "data_type": "fp",
852
+ "act_data_type": "fp"
853
+ },
854
+ "model.layers.21.self_attn.f_a_proj": {
855
+ "bits": 16,
856
+ "data_type": "fp",
857
+ "act_data_type": "fp"
858
+ },
859
+ "model.layers.21.self_attn.f_b_proj": {
860
+ "bits": 16,
861
+ "data_type": "fp",
862
+ "act_data_type": "fp"
863
+ },
864
+ "model.layers.21.self_attn.b_proj": {
865
+ "bits": 16,
866
+ "data_type": "fp",
867
+ "act_data_type": "fp"
868
+ },
869
+ "model.layers.21.self_attn.g_a_proj": {
870
+ "bits": 16,
871
+ "data_type": "fp",
872
+ "act_data_type": "fp"
873
+ },
874
+ "model.layers.21.self_attn.g_b_proj": {
875
+ "bits": 16,
876
+ "data_type": "fp",
877
+ "act_data_type": "fp"
878
+ },
879
+ "model.layers.21.self_attn.o_proj": {
880
+ "bits": 16,
881
+ "data_type": "fp",
882
+ "act_data_type": "fp"
883
+ },
884
+ "model.layers.22.self_attn.q_proj": {
885
+ "bits": 16,
886
+ "data_type": "fp",
887
+ "act_data_type": "fp"
888
+ },
889
+ "model.layers.22.self_attn.k_proj": {
890
+ "bits": 16,
891
+ "data_type": "fp",
892
+ "act_data_type": "fp"
893
+ },
894
+ "model.layers.22.self_attn.v_proj": {
895
+ "bits": 16,
896
+ "data_type": "fp",
897
+ "act_data_type": "fp"
898
+ },
899
+ "model.layers.22.self_attn.f_a_proj": {
900
+ "bits": 16,
901
+ "data_type": "fp",
902
+ "act_data_type": "fp"
903
+ },
904
+ "model.layers.22.self_attn.f_b_proj": {
905
+ "bits": 16,
906
+ "data_type": "fp",
907
+ "act_data_type": "fp"
908
+ },
909
+ "model.layers.22.self_attn.b_proj": {
910
+ "bits": 16,
911
+ "data_type": "fp",
912
+ "act_data_type": "fp"
913
+ },
914
+ "model.layers.22.self_attn.g_a_proj": {
915
+ "bits": 16,
916
+ "data_type": "fp",
917
+ "act_data_type": "fp"
918
+ },
919
+ "model.layers.22.self_attn.g_b_proj": {
920
+ "bits": 16,
921
+ "data_type": "fp",
922
+ "act_data_type": "fp"
923
+ },
924
+ "model.layers.22.self_attn.o_proj": {
925
+ "bits": 16,
926
+ "data_type": "fp",
927
+ "act_data_type": "fp"
928
+ },
929
+ "model.layers.23.self_attn.q_proj": {
930
+ "bits": 16,
931
+ "data_type": "fp",
932
+ "act_data_type": "fp"
933
+ },
934
+ "model.layers.23.self_attn.k_proj": {
935
+ "bits": 16,
936
+ "data_type": "fp",
937
+ "act_data_type": "fp"
938
+ },
939
+ "model.layers.23.self_attn.v_proj": {
940
+ "bits": 16,
941
+ "data_type": "fp",
942
+ "act_data_type": "fp"
943
+ },
944
+ "model.layers.23.self_attn.f_a_proj": {
945
+ "bits": 16,
946
+ "data_type": "fp",
947
+ "act_data_type": "fp"
948
+ },
949
+ "model.layers.23.self_attn.f_b_proj": {
950
+ "bits": 16,
951
+ "data_type": "fp",
952
+ "act_data_type": "fp"
953
+ },
954
+ "model.layers.23.self_attn.b_proj": {
955
+ "bits": 16,
956
+ "data_type": "fp",
957
+ "act_data_type": "fp"
958
+ },
959
+ "model.layers.23.self_attn.g_a_proj": {
960
+ "bits": 16,
961
+ "data_type": "fp",
962
+ "act_data_type": "fp"
963
+ },
964
+ "model.layers.23.self_attn.g_b_proj": {
965
+ "bits": 16,
966
+ "data_type": "fp",
967
+ "act_data_type": "fp"
968
+ },
969
+ "model.layers.23.self_attn.o_proj": {
970
+ "bits": 16,
971
+ "data_type": "fp",
972
+ "act_data_type": "fp"
973
+ },
974
+ "model.layers.24.self_attn.q_proj": {
975
+ "bits": 16,
976
+ "data_type": "fp",
977
+ "act_data_type": "fp"
978
+ },
979
+ "model.layers.24.self_attn.k_proj": {
980
+ "bits": 16,
981
+ "data_type": "fp",
982
+ "act_data_type": "fp"
983
+ },
984
+ "model.layers.24.self_attn.v_proj": {
985
+ "bits": 16,
986
+ "data_type": "fp",
987
+ "act_data_type": "fp"
988
+ },
989
+ "model.layers.24.self_attn.o_proj": {
990
+ "bits": 16,
991
+ "data_type": "fp",
992
+ "act_data_type": "fp"
993
+ },
994
+ "model.layers.24.self_attn.g_proj": {
995
+ "bits": 16,
996
+ "data_type": "fp",
997
+ "act_data_type": "fp"
998
+ },
999
+ "model.layers.25.self_attn.q_proj": {
1000
+ "bits": 16,
1001
+ "data_type": "fp",
1002
+ "act_data_type": "fp"
1003
+ },
1004
+ "model.layers.25.self_attn.k_proj": {
1005
+ "bits": 16,
1006
+ "data_type": "fp",
1007
+ "act_data_type": "fp"
1008
+ },
1009
+ "model.layers.25.self_attn.v_proj": {
1010
+ "bits": 16,
1011
+ "data_type": "fp",
1012
+ "act_data_type": "fp"
1013
+ },
1014
+ "model.layers.25.self_attn.f_a_proj": {
1015
+ "bits": 16,
1016
+ "data_type": "fp",
1017
+ "act_data_type": "fp"
1018
+ },
1019
+ "model.layers.25.self_attn.f_b_proj": {
1020
+ "bits": 16,
1021
+ "data_type": "fp",
1022
+ "act_data_type": "fp"
1023
+ },
1024
+ "model.layers.25.self_attn.b_proj": {
1025
+ "bits": 16,
1026
+ "data_type": "fp",
1027
+ "act_data_type": "fp"
1028
+ },
1029
+ "model.layers.25.self_attn.g_a_proj": {
1030
+ "bits": 16,
1031
+ "data_type": "fp",
1032
+ "act_data_type": "fp"
1033
+ },
1034
+ "model.layers.25.self_attn.g_b_proj": {
1035
+ "bits": 16,
1036
+ "data_type": "fp",
1037
+ "act_data_type": "fp"
1038
+ },
1039
+ "model.layers.25.self_attn.o_proj": {
1040
+ "bits": 16,
1041
+ "data_type": "fp",
1042
+ "act_data_type": "fp"
1043
+ },
1044
+ "model.layers.26.self_attn.q_proj": {
1045
+ "bits": 16,
1046
+ "data_type": "fp",
1047
+ "act_data_type": "fp"
1048
+ },
1049
+ "model.layers.26.self_attn.k_proj": {
1050
+ "bits": 16,
1051
+ "data_type": "fp",
1052
+ "act_data_type": "fp"
1053
+ },
1054
+ "model.layers.26.self_attn.v_proj": {
1055
+ "bits": 16,
1056
+ "data_type": "fp",
1057
+ "act_data_type": "fp"
1058
+ },
1059
+ "model.layers.26.self_attn.f_a_proj": {
1060
+ "bits": 16,
1061
+ "data_type": "fp",
1062
+ "act_data_type": "fp"
1063
+ },
1064
+ "model.layers.26.self_attn.f_b_proj": {
1065
+ "bits": 16,
1066
+ "data_type": "fp",
1067
+ "act_data_type": "fp"
1068
+ },
1069
+ "model.layers.26.self_attn.b_proj": {
1070
+ "bits": 16,
1071
+ "data_type": "fp",
1072
+ "act_data_type": "fp"
1073
+ },
1074
+ "model.layers.26.self_attn.g_a_proj": {
1075
+ "bits": 16,
1076
+ "data_type": "fp",
1077
+ "act_data_type": "fp"
1078
+ },
1079
+ "model.layers.26.self_attn.g_b_proj": {
1080
+ "bits": 16,
1081
+ "data_type": "fp",
1082
+ "act_data_type": "fp"
1083
+ },
1084
+ "model.layers.26.self_attn.o_proj": {
1085
+ "bits": 16,
1086
+ "data_type": "fp",
1087
+ "act_data_type": "fp"
1088
+ },
1089
+ "model.layers.27.self_attn.q_proj": {
1090
+ "bits": 16,
1091
+ "data_type": "fp",
1092
+ "act_data_type": "fp"
1093
+ },
1094
+ "model.layers.27.self_attn.k_proj": {
1095
+ "bits": 16,
1096
+ "data_type": "fp",
1097
+ "act_data_type": "fp"
1098
+ },
1099
+ "model.layers.27.self_attn.v_proj": {
1100
+ "bits": 16,
1101
+ "data_type": "fp",
1102
+ "act_data_type": "fp"
1103
+ },
1104
+ "model.layers.27.self_attn.f_a_proj": {
1105
+ "bits": 16,
1106
+ "data_type": "fp",
1107
+ "act_data_type": "fp"
1108
+ },
1109
+ "model.layers.27.self_attn.f_b_proj": {
1110
+ "bits": 16,
1111
+ "data_type": "fp",
1112
+ "act_data_type": "fp"
1113
+ },
1114
+ "model.layers.27.self_attn.b_proj": {
1115
+ "bits": 16,
1116
+ "data_type": "fp",
1117
+ "act_data_type": "fp"
1118
+ },
1119
+ "model.layers.27.self_attn.g_a_proj": {
1120
+ "bits": 16,
1121
+ "data_type": "fp",
1122
+ "act_data_type": "fp"
1123
+ },
1124
+ "model.layers.27.self_attn.g_b_proj": {
1125
+ "bits": 16,
1126
+ "data_type": "fp",
1127
+ "act_data_type": "fp"
1128
+ },
1129
+ "model.layers.27.self_attn.o_proj": {
1130
+ "bits": 16,
1131
+ "data_type": "fp",
1132
+ "act_data_type": "fp"
1133
+ },
1134
+ "model.layers.28.self_attn.q_proj": {
1135
+ "bits": 16,
1136
+ "data_type": "fp",
1137
+ "act_data_type": "fp"
1138
+ },
1139
+ "model.layers.28.self_attn.k_proj": {
1140
+ "bits": 16,
1141
+ "data_type": "fp",
1142
+ "act_data_type": "fp"
1143
+ },
1144
+ "model.layers.28.self_attn.v_proj": {
1145
+ "bits": 16,
1146
+ "data_type": "fp",
1147
+ "act_data_type": "fp"
1148
+ },
1149
+ "model.layers.28.self_attn.o_proj": {
1150
+ "bits": 16,
1151
+ "data_type": "fp",
1152
+ "act_data_type": "fp"
1153
+ },
1154
+ "model.layers.28.self_attn.g_proj": {
1155
+ "bits": 16,
1156
+ "data_type": "fp",
1157
+ "act_data_type": "fp"
1158
+ },
1159
+ "model.layers.29.self_attn.q_proj": {
1160
+ "bits": 16,
1161
+ "data_type": "fp",
1162
+ "act_data_type": "fp"
1163
+ },
1164
+ "model.layers.29.self_attn.k_proj": {
1165
+ "bits": 16,
1166
+ "data_type": "fp",
1167
+ "act_data_type": "fp"
1168
+ },
1169
+ "model.layers.29.self_attn.v_proj": {
1170
+ "bits": 16,
1171
+ "data_type": "fp",
1172
+ "act_data_type": "fp"
1173
+ },
1174
+ "model.layers.29.self_attn.f_a_proj": {
1175
+ "bits": 16,
1176
+ "data_type": "fp",
1177
+ "act_data_type": "fp"
1178
+ },
1179
+ "model.layers.29.self_attn.f_b_proj": {
1180
+ "bits": 16,
1181
+ "data_type": "fp",
1182
+ "act_data_type": "fp"
1183
+ },
1184
+ "model.layers.29.self_attn.b_proj": {
1185
+ "bits": 16,
1186
+ "data_type": "fp",
1187
+ "act_data_type": "fp"
1188
+ },
1189
+ "model.layers.29.self_attn.g_a_proj": {
1190
+ "bits": 16,
1191
+ "data_type": "fp",
1192
+ "act_data_type": "fp"
1193
+ },
1194
+ "model.layers.29.self_attn.g_b_proj": {
1195
+ "bits": 16,
1196
+ "data_type": "fp",
1197
+ "act_data_type": "fp"
1198
+ },
1199
+ "model.layers.29.self_attn.o_proj": {
1200
+ "bits": 16,
1201
+ "data_type": "fp",
1202
+ "act_data_type": "fp"
1203
+ },
1204
+ "model.layers.30.self_attn.q_proj": {
1205
+ "bits": 16,
1206
+ "data_type": "fp",
1207
+ "act_data_type": "fp"
1208
+ },
1209
+ "model.layers.30.self_attn.k_proj": {
1210
+ "bits": 16,
1211
+ "data_type": "fp",
1212
+ "act_data_type": "fp"
1213
+ },
1214
+ "model.layers.30.self_attn.v_proj": {
1215
+ "bits": 16,
1216
+ "data_type": "fp",
1217
+ "act_data_type": "fp"
1218
+ },
1219
+ "model.layers.30.self_attn.f_a_proj": {
1220
+ "bits": 16,
1221
+ "data_type": "fp",
1222
+ "act_data_type": "fp"
1223
+ },
1224
+ "model.layers.30.self_attn.f_b_proj": {
1225
+ "bits": 16,
1226
+ "data_type": "fp",
1227
+ "act_data_type": "fp"
1228
+ },
1229
+ "model.layers.30.self_attn.b_proj": {
1230
+ "bits": 16,
1231
+ "data_type": "fp",
1232
+ "act_data_type": "fp"
1233
+ },
1234
+ "model.layers.30.self_attn.g_a_proj": {
1235
+ "bits": 16,
1236
+ "data_type": "fp",
1237
+ "act_data_type": "fp"
1238
+ },
1239
+ "model.layers.30.self_attn.g_b_proj": {
1240
+ "bits": 16,
1241
+ "data_type": "fp",
1242
+ "act_data_type": "fp"
1243
+ },
1244
+ "model.layers.30.self_attn.o_proj": {
1245
+ "bits": 16,
1246
+ "data_type": "fp",
1247
+ "act_data_type": "fp"
1248
+ },
1249
+ "model.layers.31.self_attn.q_proj": {
1250
+ "bits": 16,
1251
+ "data_type": "fp",
1252
+ "act_data_type": "fp"
1253
+ },
1254
+ "model.layers.31.self_attn.k_proj": {
1255
+ "bits": 16,
1256
+ "data_type": "fp",
1257
+ "act_data_type": "fp"
1258
+ },
1259
+ "model.layers.31.self_attn.v_proj": {
1260
+ "bits": 16,
1261
+ "data_type": "fp",
1262
+ "act_data_type": "fp"
1263
+ },
1264
+ "model.layers.31.self_attn.f_a_proj": {
1265
+ "bits": 16,
1266
+ "data_type": "fp",
1267
+ "act_data_type": "fp"
1268
+ },
1269
+ "model.layers.31.self_attn.f_b_proj": {
1270
+ "bits": 16,
1271
+ "data_type": "fp",
1272
+ "act_data_type": "fp"
1273
+ },
1274
+ "model.layers.31.self_attn.b_proj": {
1275
+ "bits": 16,
1276
+ "data_type": "fp",
1277
+ "act_data_type": "fp"
1278
+ },
1279
+ "model.layers.31.self_attn.g_a_proj": {
1280
+ "bits": 16,
1281
+ "data_type": "fp",
1282
+ "act_data_type": "fp"
1283
+ },
1284
+ "model.layers.31.self_attn.g_b_proj": {
1285
+ "bits": 16,
1286
+ "data_type": "fp",
1287
+ "act_data_type": "fp"
1288
+ },
1289
+ "model.layers.31.self_attn.o_proj": {
1290
+ "bits": 16,
1291
+ "data_type": "fp",
1292
+ "act_data_type": "fp"
1293
+ },
1294
+ "model.layers.32.self_attn.q_proj": {
1295
+ "bits": 16,
1296
+ "data_type": "fp",
1297
+ "act_data_type": "fp"
1298
+ },
1299
+ "model.layers.32.self_attn.k_proj": {
1300
+ "bits": 16,
1301
+ "data_type": "fp",
1302
+ "act_data_type": "fp"
1303
+ },
1304
+ "model.layers.32.self_attn.v_proj": {
1305
+ "bits": 16,
1306
+ "data_type": "fp",
1307
+ "act_data_type": "fp"
1308
+ },
1309
+ "model.layers.32.self_attn.o_proj": {
1310
+ "bits": 16,
1311
+ "data_type": "fp",
1312
+ "act_data_type": "fp"
1313
+ },
1314
+ "model.layers.32.self_attn.g_proj": {
1315
+ "bits": 16,
1316
+ "data_type": "fp",
1317
+ "act_data_type": "fp"
1318
+ },
1319
+ "model.layers.33.self_attn.q_proj": {
1320
+ "bits": 16,
1321
+ "data_type": "fp",
1322
+ "act_data_type": "fp"
1323
+ },
1324
+ "model.layers.33.self_attn.k_proj": {
1325
+ "bits": 16,
1326
+ "data_type": "fp",
1327
+ "act_data_type": "fp"
1328
+ },
1329
+ "model.layers.33.self_attn.v_proj": {
1330
+ "bits": 16,
1331
+ "data_type": "fp",
1332
+ "act_data_type": "fp"
1333
+ },
1334
+ "model.layers.33.self_attn.f_a_proj": {
1335
+ "bits": 16,
1336
+ "data_type": "fp",
1337
+ "act_data_type": "fp"
1338
+ },
1339
+ "model.layers.33.self_attn.f_b_proj": {
1340
+ "bits": 16,
1341
+ "data_type": "fp",
1342
+ "act_data_type": "fp"
1343
+ },
1344
+ "model.layers.33.self_attn.b_proj": {
1345
+ "bits": 16,
1346
+ "data_type": "fp",
1347
+ "act_data_type": "fp"
1348
+ },
1349
+ "model.layers.33.self_attn.g_a_proj": {
1350
+ "bits": 16,
1351
+ "data_type": "fp",
1352
+ "act_data_type": "fp"
1353
+ },
1354
+ "model.layers.33.self_attn.g_b_proj": {
1355
+ "bits": 16,
1356
+ "data_type": "fp",
1357
+ "act_data_type": "fp"
1358
+ },
1359
+ "model.layers.33.self_attn.o_proj": {
1360
+ "bits": 16,
1361
+ "data_type": "fp",
1362
+ "act_data_type": "fp"
1363
+ },
1364
+ "model.layers.34.self_attn.q_proj": {
1365
+ "bits": 16,
1366
+ "data_type": "fp",
1367
+ "act_data_type": "fp"
1368
+ },
1369
+ "model.layers.34.self_attn.k_proj": {
1370
+ "bits": 16,
1371
+ "data_type": "fp",
1372
+ "act_data_type": "fp"
1373
+ },
1374
+ "model.layers.34.self_attn.v_proj": {
1375
+ "bits": 16,
1376
+ "data_type": "fp",
1377
+ "act_data_type": "fp"
1378
+ },
1379
+ "model.layers.34.self_attn.f_a_proj": {
1380
+ "bits": 16,
1381
+ "data_type": "fp",
1382
+ "act_data_type": "fp"
1383
+ },
1384
+ "model.layers.34.self_attn.f_b_proj": {
1385
+ "bits": 16,
1386
+ "data_type": "fp",
1387
+ "act_data_type": "fp"
1388
+ },
1389
+ "model.layers.34.self_attn.b_proj": {
1390
+ "bits": 16,
1391
+ "data_type": "fp",
1392
+ "act_data_type": "fp"
1393
+ },
1394
+ "model.layers.34.self_attn.g_a_proj": {
1395
+ "bits": 16,
1396
+ "data_type": "fp",
1397
+ "act_data_type": "fp"
1398
+ },
1399
+ "model.layers.34.self_attn.g_b_proj": {
1400
+ "bits": 16,
1401
+ "data_type": "fp",
1402
+ "act_data_type": "fp"
1403
+ },
1404
+ "model.layers.34.self_attn.o_proj": {
1405
+ "bits": 16,
1406
+ "data_type": "fp",
1407
+ "act_data_type": "fp"
1408
+ },
1409
+ "model.layers.35.self_attn.q_proj": {
1410
+ "bits": 16,
1411
+ "data_type": "fp",
1412
+ "act_data_type": "fp"
1413
+ },
1414
+ "model.layers.35.self_attn.k_proj": {
1415
+ "bits": 16,
1416
+ "data_type": "fp",
1417
+ "act_data_type": "fp"
1418
+ },
1419
+ "model.layers.35.self_attn.v_proj": {
1420
+ "bits": 16,
1421
+ "data_type": "fp",
1422
+ "act_data_type": "fp"
1423
+ },
1424
+ "model.layers.35.self_attn.f_a_proj": {
1425
+ "bits": 16,
1426
+ "data_type": "fp",
1427
+ "act_data_type": "fp"
1428
+ },
1429
+ "model.layers.35.self_attn.f_b_proj": {
1430
+ "bits": 16,
1431
+ "data_type": "fp",
1432
+ "act_data_type": "fp"
1433
+ },
1434
+ "model.layers.35.self_attn.b_proj": {
1435
+ "bits": 16,
1436
+ "data_type": "fp",
1437
+ "act_data_type": "fp"
1438
+ },
1439
+ "model.layers.35.self_attn.g_a_proj": {
1440
+ "bits": 16,
1441
+ "data_type": "fp",
1442
+ "act_data_type": "fp"
1443
+ },
1444
+ "model.layers.35.self_attn.g_b_proj": {
1445
+ "bits": 16,
1446
+ "data_type": "fp",
1447
+ "act_data_type": "fp"
1448
+ },
1449
+ "model.layers.35.self_attn.o_proj": {
1450
+ "bits": 16,
1451
+ "data_type": "fp",
1452
+ "act_data_type": "fp"
1453
+ },
1454
+ "model.layers.36.self_attn.q_proj": {
1455
+ "bits": 16,
1456
+ "data_type": "fp",
1457
+ "act_data_type": "fp"
1458
+ },
1459
+ "model.layers.36.self_attn.k_proj": {
1460
+ "bits": 16,
1461
+ "data_type": "fp",
1462
+ "act_data_type": "fp"
1463
+ },
1464
+ "model.layers.36.self_attn.v_proj": {
1465
+ "bits": 16,
1466
+ "data_type": "fp",
1467
+ "act_data_type": "fp"
1468
+ },
1469
+ "model.layers.36.self_attn.o_proj": {
1470
+ "bits": 16,
1471
+ "data_type": "fp",
1472
+ "act_data_type": "fp"
1473
+ },
1474
+ "model.layers.36.self_attn.g_proj": {
1475
+ "bits": 16,
1476
+ "data_type": "fp",
1477
+ "act_data_type": "fp"
1478
+ },
1479
+ "model.layers.37.self_attn.q_proj": {
1480
+ "bits": 16,
1481
+ "data_type": "fp",
1482
+ "act_data_type": "fp"
1483
+ },
1484
+ "model.layers.37.self_attn.k_proj": {
1485
+ "bits": 16,
1486
+ "data_type": "fp",
1487
+ "act_data_type": "fp"
1488
+ },
1489
+ "model.layers.37.self_attn.v_proj": {
1490
+ "bits": 16,
1491
+ "data_type": "fp",
1492
+ "act_data_type": "fp"
1493
+ },
1494
+ "model.layers.37.self_attn.f_a_proj": {
1495
+ "bits": 16,
1496
+ "data_type": "fp",
1497
+ "act_data_type": "fp"
1498
+ },
1499
+ "model.layers.37.self_attn.f_b_proj": {
1500
+ "bits": 16,
1501
+ "data_type": "fp",
1502
+ "act_data_type": "fp"
1503
+ },
1504
+ "model.layers.37.self_attn.b_proj": {
1505
+ "bits": 16,
1506
+ "data_type": "fp",
1507
+ "act_data_type": "fp"
1508
+ },
1509
+ "model.layers.37.self_attn.g_a_proj": {
1510
+ "bits": 16,
1511
+ "data_type": "fp",
1512
+ "act_data_type": "fp"
1513
+ },
1514
+ "model.layers.37.self_attn.g_b_proj": {
1515
+ "bits": 16,
1516
+ "data_type": "fp",
1517
+ "act_data_type": "fp"
1518
+ },
1519
+ "model.layers.37.self_attn.o_proj": {
1520
+ "bits": 16,
1521
+ "data_type": "fp",
1522
+ "act_data_type": "fp"
1523
+ },
1524
+ "model.layers.38.self_attn.q_proj": {
1525
+ "bits": 16,
1526
+ "data_type": "fp",
1527
+ "act_data_type": "fp"
1528
+ },
1529
+ "model.layers.38.self_attn.k_proj": {
1530
+ "bits": 16,
1531
+ "data_type": "fp",
1532
+ "act_data_type": "fp"
1533
+ },
1534
+ "model.layers.38.self_attn.v_proj": {
1535
+ "bits": 16,
1536
+ "data_type": "fp",
1537
+ "act_data_type": "fp"
1538
+ },
1539
+ "model.layers.38.self_attn.f_a_proj": {
1540
+ "bits": 16,
1541
+ "data_type": "fp",
1542
+ "act_data_type": "fp"
1543
+ },
1544
+ "model.layers.38.self_attn.f_b_proj": {
1545
+ "bits": 16,
1546
+ "data_type": "fp",
1547
+ "act_data_type": "fp"
1548
+ },
1549
+ "model.layers.38.self_attn.b_proj": {
1550
+ "bits": 16,
1551
+ "data_type": "fp",
1552
+ "act_data_type": "fp"
1553
+ },
1554
+ "model.layers.38.self_attn.g_a_proj": {
1555
+ "bits": 16,
1556
+ "data_type": "fp",
1557
+ "act_data_type": "fp"
1558
+ },
1559
+ "model.layers.38.self_attn.g_b_proj": {
1560
+ "bits": 16,
1561
+ "data_type": "fp",
1562
+ "act_data_type": "fp"
1563
+ },
1564
+ "model.layers.38.self_attn.o_proj": {
1565
+ "bits": 16,
1566
+ "data_type": "fp",
1567
+ "act_data_type": "fp"
1568
+ },
1569
+ "model.layers.39.self_attn.q_proj": {
1570
+ "bits": 16,
1571
+ "data_type": "fp",
1572
+ "act_data_type": "fp"
1573
+ },
1574
+ "model.layers.39.self_attn.k_proj": {
1575
+ "bits": 16,
1576
+ "data_type": "fp",
1577
+ "act_data_type": "fp"
1578
+ },
1579
+ "model.layers.39.self_attn.v_proj": {
1580
+ "bits": 16,
1581
+ "data_type": "fp",
1582
+ "act_data_type": "fp"
1583
+ },
1584
+ "model.layers.39.self_attn.f_a_proj": {
1585
+ "bits": 16,
1586
+ "data_type": "fp",
1587
+ "act_data_type": "fp"
1588
+ },
1589
+ "model.layers.39.self_attn.f_b_proj": {
1590
+ "bits": 16,
1591
+ "data_type": "fp",
1592
+ "act_data_type": "fp"
1593
+ },
1594
+ "model.layers.39.self_attn.b_proj": {
1595
+ "bits": 16,
1596
+ "data_type": "fp",
1597
+ "act_data_type": "fp"
1598
+ },
1599
+ "model.layers.39.self_attn.g_a_proj": {
1600
+ "bits": 16,
1601
+ "data_type": "fp",
1602
+ "act_data_type": "fp"
1603
+ },
1604
+ "model.layers.39.self_attn.g_b_proj": {
1605
+ "bits": 16,
1606
+ "data_type": "fp",
1607
+ "act_data_type": "fp"
1608
+ },
1609
+ "model.layers.39.self_attn.o_proj": {
1610
+ "bits": 16,
1611
+ "data_type": "fp",
1612
+ "act_data_type": "fp"
1613
+ },
1614
+ "model.layers.40.self_attn.q_proj": {
1615
+ "bits": 16,
1616
+ "data_type": "fp",
1617
+ "act_data_type": "fp"
1618
+ },
1619
+ "model.layers.40.self_attn.k_proj": {
1620
+ "bits": 16,
1621
+ "data_type": "fp",
1622
+ "act_data_type": "fp"
1623
+ },
1624
+ "model.layers.40.self_attn.v_proj": {
1625
+ "bits": 16,
1626
+ "data_type": "fp",
1627
+ "act_data_type": "fp"
1628
+ },
1629
+ "model.layers.40.self_attn.o_proj": {
1630
+ "bits": 16,
1631
+ "data_type": "fp",
1632
+ "act_data_type": "fp"
1633
+ },
1634
+ "model.layers.40.self_attn.g_proj": {
1635
+ "bits": 16,
1636
+ "data_type": "fp",
1637
+ "act_data_type": "fp"
1638
+ },
1639
+ "model.layers.41.self_attn.q_proj": {
1640
+ "bits": 16,
1641
+ "data_type": "fp",
1642
+ "act_data_type": "fp"
1643
+ },
1644
+ "model.layers.41.self_attn.k_proj": {
1645
+ "bits": 16,
1646
+ "data_type": "fp",
1647
+ "act_data_type": "fp"
1648
+ },
1649
+ "model.layers.41.self_attn.v_proj": {
1650
+ "bits": 16,
1651
+ "data_type": "fp",
1652
+ "act_data_type": "fp"
1653
+ },
1654
+ "model.layers.41.self_attn.f_a_proj": {
1655
+ "bits": 16,
1656
+ "data_type": "fp",
1657
+ "act_data_type": "fp"
1658
+ },
1659
+ "model.layers.41.self_attn.f_b_proj": {
1660
+ "bits": 16,
1661
+ "data_type": "fp",
1662
+ "act_data_type": "fp"
1663
+ },
1664
+ "model.layers.41.self_attn.b_proj": {
1665
+ "bits": 16,
1666
+ "data_type": "fp",
1667
+ "act_data_type": "fp"
1668
+ },
1669
+ "model.layers.41.self_attn.g_a_proj": {
1670
+ "bits": 16,
1671
+ "data_type": "fp",
1672
+ "act_data_type": "fp"
1673
+ },
1674
+ "model.layers.41.self_attn.g_b_proj": {
1675
+ "bits": 16,
1676
+ "data_type": "fp",
1677
+ "act_data_type": "fp"
1678
+ },
1679
+ "model.layers.41.self_attn.o_proj": {
1680
+ "bits": 16,
1681
+ "data_type": "fp",
1682
+ "act_data_type": "fp"
1683
+ },
1684
+ "model.layers.42.self_attn.q_proj": {
1685
+ "bits": 16,
1686
+ "data_type": "fp",
1687
+ "act_data_type": "fp"
1688
+ },
1689
+ "model.layers.42.self_attn.k_proj": {
1690
+ "bits": 16,
1691
+ "data_type": "fp",
1692
+ "act_data_type": "fp"
1693
+ },
1694
+ "model.layers.42.self_attn.v_proj": {
1695
+ "bits": 16,
1696
+ "data_type": "fp",
1697
+ "act_data_type": "fp"
1698
+ },
1699
+ "model.layers.42.self_attn.f_a_proj": {
1700
+ "bits": 16,
1701
+ "data_type": "fp",
1702
+ "act_data_type": "fp"
1703
+ },
1704
+ "model.layers.42.self_attn.f_b_proj": {
1705
+ "bits": 16,
1706
+ "data_type": "fp",
1707
+ "act_data_type": "fp"
1708
+ },
1709
+ "model.layers.42.self_attn.b_proj": {
1710
+ "bits": 16,
1711
+ "data_type": "fp",
1712
+ "act_data_type": "fp"
1713
+ },
1714
+ "model.layers.42.self_attn.g_a_proj": {
1715
+ "bits": 16,
1716
+ "data_type": "fp",
1717
+ "act_data_type": "fp"
1718
+ },
1719
+ "model.layers.42.self_attn.g_b_proj": {
1720
+ "bits": 16,
1721
+ "data_type": "fp",
1722
+ "act_data_type": "fp"
1723
+ },
1724
+ "model.layers.42.self_attn.o_proj": {
1725
+ "bits": 16,
1726
+ "data_type": "fp",
1727
+ "act_data_type": "fp"
1728
+ },
1729
+ "model.layers.43.self_attn.q_proj": {
1730
+ "bits": 16,
1731
+ "data_type": "fp",
1732
+ "act_data_type": "fp"
1733
+ },
1734
+ "model.layers.43.self_attn.k_proj": {
1735
+ "bits": 16,
1736
+ "data_type": "fp",
1737
+ "act_data_type": "fp"
1738
+ },
1739
+ "model.layers.43.self_attn.v_proj": {
1740
+ "bits": 16,
1741
+ "data_type": "fp",
1742
+ "act_data_type": "fp"
1743
+ },
1744
+ "model.layers.43.self_attn.f_a_proj": {
1745
+ "bits": 16,
1746
+ "data_type": "fp",
1747
+ "act_data_type": "fp"
1748
+ },
1749
+ "model.layers.43.self_attn.f_b_proj": {
1750
+ "bits": 16,
1751
+ "data_type": "fp",
1752
+ "act_data_type": "fp"
1753
+ },
1754
+ "model.layers.43.self_attn.b_proj": {
1755
+ "bits": 16,
1756
+ "data_type": "fp",
1757
+ "act_data_type": "fp"
1758
+ },
1759
+ "model.layers.43.self_attn.g_a_proj": {
1760
+ "bits": 16,
1761
+ "data_type": "fp",
1762
+ "act_data_type": "fp"
1763
+ },
1764
+ "model.layers.43.self_attn.g_b_proj": {
1765
+ "bits": 16,
1766
+ "data_type": "fp",
1767
+ "act_data_type": "fp"
1768
+ },
1769
+ "model.layers.43.self_attn.o_proj": {
1770
+ "bits": 16,
1771
+ "data_type": "fp",
1772
+ "act_data_type": "fp"
1773
+ },
1774
+ "model.layers.44.self_attn.q_proj": {
1775
+ "bits": 16,
1776
+ "data_type": "fp",
1777
+ "act_data_type": "fp"
1778
+ },
1779
+ "model.layers.44.self_attn.k_proj": {
1780
+ "bits": 16,
1781
+ "data_type": "fp",
1782
+ "act_data_type": "fp"
1783
+ },
1784
+ "model.layers.44.self_attn.v_proj": {
1785
+ "bits": 16,
1786
+ "data_type": "fp",
1787
+ "act_data_type": "fp"
1788
+ },
1789
+ "model.layers.44.self_attn.o_proj": {
1790
+ "bits": 16,
1791
+ "data_type": "fp",
1792
+ "act_data_type": "fp"
1793
+ },
1794
+ "model.layers.44.self_attn.g_proj": {
1795
+ "bits": 16,
1796
+ "data_type": "fp",
1797
+ "act_data_type": "fp"
1798
+ },
1799
+ "model.layers.45.self_attn.q_proj": {
1800
+ "bits": 16,
1801
+ "data_type": "fp",
1802
+ "act_data_type": "fp"
1803
+ },
1804
+ "model.layers.45.self_attn.k_proj": {
1805
+ "bits": 16,
1806
+ "data_type": "fp",
1807
+ "act_data_type": "fp"
1808
+ },
1809
+ "model.layers.45.self_attn.v_proj": {
1810
+ "bits": 16,
1811
+ "data_type": "fp",
1812
+ "act_data_type": "fp"
1813
+ },
1814
+ "model.layers.45.self_attn.f_a_proj": {
1815
+ "bits": 16,
1816
+ "data_type": "fp",
1817
+ "act_data_type": "fp"
1818
+ },
1819
+ "model.layers.45.self_attn.f_b_proj": {
1820
+ "bits": 16,
1821
+ "data_type": "fp",
1822
+ "act_data_type": "fp"
1823
+ },
1824
+ "model.layers.45.self_attn.b_proj": {
1825
+ "bits": 16,
1826
+ "data_type": "fp",
1827
+ "act_data_type": "fp"
1828
+ },
1829
+ "model.layers.45.self_attn.g_a_proj": {
1830
+ "bits": 16,
1831
+ "data_type": "fp",
1832
+ "act_data_type": "fp"
1833
+ },
1834
+ "model.layers.45.self_attn.g_b_proj": {
1835
+ "bits": 16,
1836
+ "data_type": "fp",
1837
+ "act_data_type": "fp"
1838
+ },
1839
+ "model.layers.45.self_attn.o_proj": {
1840
+ "bits": 16,
1841
+ "data_type": "fp",
1842
+ "act_data_type": "fp"
1843
+ },
1844
+ "model.layers.46.self_attn.q_proj": {
1845
+ "bits": 16,
1846
+ "data_type": "fp",
1847
+ "act_data_type": "fp"
1848
+ },
1849
+ "model.layers.46.self_attn.k_proj": {
1850
+ "bits": 16,
1851
+ "data_type": "fp",
1852
+ "act_data_type": "fp"
1853
+ },
1854
+ "model.layers.46.self_attn.v_proj": {
1855
+ "bits": 16,
1856
+ "data_type": "fp",
1857
+ "act_data_type": "fp"
1858
+ },
1859
+ "model.layers.46.self_attn.f_a_proj": {
1860
+ "bits": 16,
1861
+ "data_type": "fp",
1862
+ "act_data_type": "fp"
1863
+ },
1864
+ "model.layers.46.self_attn.f_b_proj": {
1865
+ "bits": 16,
1866
+ "data_type": "fp",
1867
+ "act_data_type": "fp"
1868
+ },
1869
+ "model.layers.46.self_attn.b_proj": {
1870
+ "bits": 16,
1871
+ "data_type": "fp",
1872
+ "act_data_type": "fp"
1873
+ },
1874
+ "model.layers.46.self_attn.g_a_proj": {
1875
+ "bits": 16,
1876
+ "data_type": "fp",
1877
+ "act_data_type": "fp"
1878
+ },
1879
+ "model.layers.46.self_attn.g_b_proj": {
1880
+ "bits": 16,
1881
+ "data_type": "fp",
1882
+ "act_data_type": "fp"
1883
+ },
1884
+ "model.layers.46.self_attn.o_proj": {
1885
+ "bits": 16,
1886
+ "data_type": "fp",
1887
+ "act_data_type": "fp"
1888
+ },
1889
+ "model.layers.47.self_attn.q_proj": {
1890
+ "bits": 16,
1891
+ "data_type": "fp",
1892
+ "act_data_type": "fp"
1893
+ },
1894
+ "model.layers.47.self_attn.k_proj": {
1895
+ "bits": 16,
1896
+ "data_type": "fp",
1897
+ "act_data_type": "fp"
1898
+ },
1899
+ "model.layers.47.self_attn.v_proj": {
1900
+ "bits": 16,
1901
+ "data_type": "fp",
1902
+ "act_data_type": "fp"
1903
+ },
1904
+ "model.layers.47.self_attn.f_a_proj": {
1905
+ "bits": 16,
1906
+ "data_type": "fp",
1907
+ "act_data_type": "fp"
1908
+ },
1909
+ "model.layers.47.self_attn.f_b_proj": {
1910
+ "bits": 16,
1911
+ "data_type": "fp",
1912
+ "act_data_type": "fp"
1913
+ },
1914
+ "model.layers.47.self_attn.b_proj": {
1915
+ "bits": 16,
1916
+ "data_type": "fp",
1917
+ "act_data_type": "fp"
1918
+ },
1919
+ "model.layers.47.self_attn.g_a_proj": {
1920
+ "bits": 16,
1921
+ "data_type": "fp",
1922
+ "act_data_type": "fp"
1923
+ },
1924
+ "model.layers.47.self_attn.g_b_proj": {
1925
+ "bits": 16,
1926
+ "data_type": "fp",
1927
+ "act_data_type": "fp"
1928
+ },
1929
+ "model.layers.47.self_attn.o_proj": {
1930
+ "bits": 16,
1931
+ "data_type": "fp",
1932
+ "act_data_type": "fp"
1933
+ },
1934
+ "model.layers.0.mlp.shared_experts.gate_proj": {
1935
+ "bits": 16,
1936
+ "data_type": "fp",
1937
+ "act_data_type": "fp"
1938
+ },
1939
+ "model.layers.0.mlp.shared_experts.up_proj": {
1940
+ "bits": 16,
1941
+ "data_type": "fp",
1942
+ "act_data_type": "fp"
1943
+ },
1944
+ "model.layers.0.mlp.shared_experts.down_proj": {
1945
+ "bits": 16,
1946
+ "data_type": "fp",
1947
+ "act_data_type": "fp"
1948
+ },
1949
+ "model.layers.1.mlp.shared_experts.gate_proj": {
1950
+ "bits": 16,
1951
+ "data_type": "fp",
1952
+ "act_data_type": "fp"
1953
+ },
1954
+ "model.layers.1.mlp.shared_experts.up_proj": {
1955
+ "bits": 16,
1956
+ "data_type": "fp",
1957
+ "act_data_type": "fp"
1958
+ },
1959
+ "model.layers.1.mlp.shared_experts.down_proj": {
1960
+ "bits": 16,
1961
+ "data_type": "fp",
1962
+ "act_data_type": "fp"
1963
+ },
1964
+ "model.layers.2.mlp.shared_experts.gate_proj": {
1965
+ "bits": 16,
1966
+ "data_type": "fp",
1967
+ "act_data_type": "fp"
1968
+ },
1969
+ "model.layers.2.mlp.shared_experts.up_proj": {
1970
+ "bits": 16,
1971
+ "data_type": "fp",
1972
+ "act_data_type": "fp"
1973
+ },
1974
+ "model.layers.2.mlp.shared_experts.down_proj": {
1975
+ "bits": 16,
1976
+ "data_type": "fp",
1977
+ "act_data_type": "fp"
1978
+ },
1979
+ "model.layers.3.mlp.shared_experts.gate_proj": {
1980
+ "bits": 16,
1981
+ "data_type": "fp",
1982
+ "act_data_type": "fp"
1983
+ },
1984
+ "model.layers.3.mlp.shared_experts.up_proj": {
1985
+ "bits": 16,
1986
+ "data_type": "fp",
1987
+ "act_data_type": "fp"
1988
+ },
1989
+ "model.layers.3.mlp.shared_experts.down_proj": {
1990
+ "bits": 16,
1991
+ "data_type": "fp",
1992
+ "act_data_type": "fp"
1993
+ },
1994
+ "model.layers.4.mlp.shared_experts.gate_proj": {
1995
+ "bits": 16,
1996
+ "data_type": "fp",
1997
+ "act_data_type": "fp"
1998
+ },
1999
+ "model.layers.4.mlp.shared_experts.up_proj": {
2000
+ "bits": 16,
2001
+ "data_type": "fp",
2002
+ "act_data_type": "fp"
2003
+ },
2004
+ "model.layers.4.mlp.shared_experts.down_proj": {
2005
+ "bits": 16,
2006
+ "data_type": "fp",
2007
+ "act_data_type": "fp"
2008
+ },
2009
+ "model.layers.5.mlp.shared_experts.gate_proj": {
2010
+ "bits": 16,
2011
+ "data_type": "fp",
2012
+ "act_data_type": "fp"
2013
+ },
2014
+ "model.layers.5.mlp.shared_experts.up_proj": {
2015
+ "bits": 16,
2016
+ "data_type": "fp",
2017
+ "act_data_type": "fp"
2018
+ },
2019
+ "model.layers.5.mlp.shared_experts.down_proj": {
2020
+ "bits": 16,
2021
+ "data_type": "fp",
2022
+ "act_data_type": "fp"
2023
+ },
2024
+ "model.layers.6.mlp.shared_experts.gate_proj": {
2025
+ "bits": 16,
2026
+ "data_type": "fp",
2027
+ "act_data_type": "fp"
2028
+ },
2029
+ "model.layers.6.mlp.shared_experts.up_proj": {
2030
+ "bits": 16,
2031
+ "data_type": "fp",
2032
+ "act_data_type": "fp"
2033
+ },
2034
+ "model.layers.6.mlp.shared_experts.down_proj": {
2035
+ "bits": 16,
2036
+ "data_type": "fp",
2037
+ "act_data_type": "fp"
2038
+ },
2039
+ "model.layers.7.mlp.shared_experts.gate_proj": {
2040
+ "bits": 16,
2041
+ "data_type": "fp",
2042
+ "act_data_type": "fp"
2043
+ },
2044
+ "model.layers.7.mlp.shared_experts.up_proj": {
2045
+ "bits": 16,
2046
+ "data_type": "fp",
2047
+ "act_data_type": "fp"
2048
+ },
2049
+ "model.layers.7.mlp.shared_experts.down_proj": {
2050
+ "bits": 16,
2051
+ "data_type": "fp",
2052
+ "act_data_type": "fp"
2053
+ },
2054
+ "model.layers.8.mlp.shared_experts.gate_proj": {
2055
+ "bits": 16,
2056
+ "data_type": "fp",
2057
+ "act_data_type": "fp"
2058
+ },
2059
+ "model.layers.8.mlp.shared_experts.up_proj": {
2060
+ "bits": 16,
2061
+ "data_type": "fp",
2062
+ "act_data_type": "fp"
2063
+ },
2064
+ "model.layers.8.mlp.shared_experts.down_proj": {
2065
+ "bits": 16,
2066
+ "data_type": "fp",
2067
+ "act_data_type": "fp"
2068
+ },
2069
+ "model.layers.9.mlp.shared_experts.gate_proj": {
2070
+ "bits": 16,
2071
+ "data_type": "fp",
2072
+ "act_data_type": "fp"
2073
+ },
2074
+ "model.layers.9.mlp.shared_experts.up_proj": {
2075
+ "bits": 16,
2076
+ "data_type": "fp",
2077
+ "act_data_type": "fp"
2078
+ },
2079
+ "model.layers.9.mlp.shared_experts.down_proj": {
2080
+ "bits": 16,
2081
+ "data_type": "fp",
2082
+ "act_data_type": "fp"
2083
+ },
2084
+ "model.layers.10.mlp.shared_experts.gate_proj": {
2085
+ "bits": 16,
2086
+ "data_type": "fp",
2087
+ "act_data_type": "fp"
2088
+ },
2089
+ "model.layers.10.mlp.shared_experts.up_proj": {
2090
+ "bits": 16,
2091
+ "data_type": "fp",
2092
+ "act_data_type": "fp"
2093
+ },
2094
+ "model.layers.10.mlp.shared_experts.down_proj": {
2095
+ "bits": 16,
2096
+ "data_type": "fp",
2097
+ "act_data_type": "fp"
2098
+ },
2099
+ "model.layers.11.mlp.shared_experts.gate_proj": {
2100
+ "bits": 16,
2101
+ "data_type": "fp",
2102
+ "act_data_type": "fp"
2103
+ },
2104
+ "model.layers.11.mlp.shared_experts.up_proj": {
2105
+ "bits": 16,
2106
+ "data_type": "fp",
2107
+ "act_data_type": "fp"
2108
+ },
2109
+ "model.layers.11.mlp.shared_experts.down_proj": {
2110
+ "bits": 16,
2111
+ "data_type": "fp",
2112
+ "act_data_type": "fp"
2113
+ },
2114
+ "model.layers.12.mlp.shared_experts.gate_proj": {
2115
+ "bits": 16,
2116
+ "data_type": "fp",
2117
+ "act_data_type": "fp"
2118
+ },
2119
+ "model.layers.12.mlp.shared_experts.up_proj": {
2120
+ "bits": 16,
2121
+ "data_type": "fp",
2122
+ "act_data_type": "fp"
2123
+ },
2124
+ "model.layers.12.mlp.shared_experts.down_proj": {
2125
+ "bits": 16,
2126
+ "data_type": "fp",
2127
+ "act_data_type": "fp"
2128
+ },
2129
+ "model.layers.13.mlp.shared_experts.gate_proj": {
2130
+ "bits": 16,
2131
+ "data_type": "fp",
2132
+ "act_data_type": "fp"
2133
+ },
2134
+ "model.layers.13.mlp.shared_experts.up_proj": {
2135
+ "bits": 16,
2136
+ "data_type": "fp",
2137
+ "act_data_type": "fp"
2138
+ },
2139
+ "model.layers.13.mlp.shared_experts.down_proj": {
2140
+ "bits": 16,
2141
+ "data_type": "fp",
2142
+ "act_data_type": "fp"
2143
+ },
2144
+ "model.layers.14.mlp.shared_experts.gate_proj": {
2145
+ "bits": 16,
2146
+ "data_type": "fp",
2147
+ "act_data_type": "fp"
2148
+ },
2149
+ "model.layers.14.mlp.shared_experts.up_proj": {
2150
+ "bits": 16,
2151
+ "data_type": "fp",
2152
+ "act_data_type": "fp"
2153
+ },
2154
+ "model.layers.14.mlp.shared_experts.down_proj": {
2155
+ "bits": 16,
2156
+ "data_type": "fp",
2157
+ "act_data_type": "fp"
2158
+ },
2159
+ "model.layers.15.mlp.shared_experts.gate_proj": {
2160
+ "bits": 16,
2161
+ "data_type": "fp",
2162
+ "act_data_type": "fp"
2163
+ },
2164
+ "model.layers.15.mlp.shared_experts.up_proj": {
2165
+ "bits": 16,
2166
+ "data_type": "fp",
2167
+ "act_data_type": "fp"
2168
+ },
2169
+ "model.layers.15.mlp.shared_experts.down_proj": {
2170
+ "bits": 16,
2171
+ "data_type": "fp",
2172
+ "act_data_type": "fp"
2173
+ },
2174
+ "model.layers.16.mlp.shared_experts.gate_proj": {
2175
+ "bits": 16,
2176
+ "data_type": "fp",
2177
+ "act_data_type": "fp"
2178
+ },
2179
+ "model.layers.16.mlp.shared_experts.up_proj": {
2180
+ "bits": 16,
2181
+ "data_type": "fp",
2182
+ "act_data_type": "fp"
2183
+ },
2184
+ "model.layers.16.mlp.shared_experts.down_proj": {
2185
+ "bits": 16,
2186
+ "data_type": "fp",
2187
+ "act_data_type": "fp"
2188
+ },
2189
+ "model.layers.17.mlp.shared_experts.gate_proj": {
2190
+ "bits": 16,
2191
+ "data_type": "fp",
2192
+ "act_data_type": "fp"
2193
+ },
2194
+ "model.layers.17.mlp.shared_experts.up_proj": {
2195
+ "bits": 16,
2196
+ "data_type": "fp",
2197
+ "act_data_type": "fp"
2198
+ },
2199
+ "model.layers.17.mlp.shared_experts.down_proj": {
2200
+ "bits": 16,
2201
+ "data_type": "fp",
2202
+ "act_data_type": "fp"
2203
+ },
2204
+ "model.layers.18.mlp.shared_experts.gate_proj": {
2205
+ "bits": 16,
2206
+ "data_type": "fp",
2207
+ "act_data_type": "fp"
2208
+ },
2209
+ "model.layers.18.mlp.shared_experts.up_proj": {
2210
+ "bits": 16,
2211
+ "data_type": "fp",
2212
+ "act_data_type": "fp"
2213
+ },
2214
+ "model.layers.18.mlp.shared_experts.down_proj": {
2215
+ "bits": 16,
2216
+ "data_type": "fp",
2217
+ "act_data_type": "fp"
2218
+ },
2219
+ "model.layers.19.mlp.shared_experts.gate_proj": {
2220
+ "bits": 16,
2221
+ "data_type": "fp",
2222
+ "act_data_type": "fp"
2223
+ },
2224
+ "model.layers.19.mlp.shared_experts.up_proj": {
2225
+ "bits": 16,
2226
+ "data_type": "fp",
2227
+ "act_data_type": "fp"
2228
+ },
2229
+ "model.layers.19.mlp.shared_experts.down_proj": {
2230
+ "bits": 16,
2231
+ "data_type": "fp",
2232
+ "act_data_type": "fp"
2233
+ },
2234
+ "model.layers.20.mlp.shared_experts.gate_proj": {
2235
+ "bits": 16,
2236
+ "data_type": "fp",
2237
+ "act_data_type": "fp"
2238
+ },
2239
+ "model.layers.20.mlp.shared_experts.up_proj": {
2240
+ "bits": 16,
2241
+ "data_type": "fp",
2242
+ "act_data_type": "fp"
2243
+ },
2244
+ "model.layers.20.mlp.shared_experts.down_proj": {
2245
+ "bits": 16,
2246
+ "data_type": "fp",
2247
+ "act_data_type": "fp"
2248
+ },
2249
+ "model.layers.21.mlp.shared_experts.gate_proj": {
2250
+ "bits": 16,
2251
+ "data_type": "fp",
2252
+ "act_data_type": "fp"
2253
+ },
2254
+ "model.layers.21.mlp.shared_experts.up_proj": {
2255
+ "bits": 16,
2256
+ "data_type": "fp",
2257
+ "act_data_type": "fp"
2258
+ },
2259
+ "model.layers.21.mlp.shared_experts.down_proj": {
2260
+ "bits": 16,
2261
+ "data_type": "fp",
2262
+ "act_data_type": "fp"
2263
+ },
2264
+ "model.layers.22.mlp.shared_experts.gate_proj": {
2265
+ "bits": 16,
2266
+ "data_type": "fp",
2267
+ "act_data_type": "fp"
2268
+ },
2269
+ "model.layers.22.mlp.shared_experts.up_proj": {
2270
+ "bits": 16,
2271
+ "data_type": "fp",
2272
+ "act_data_type": "fp"
2273
+ },
2274
+ "model.layers.22.mlp.shared_experts.down_proj": {
2275
+ "bits": 16,
2276
+ "data_type": "fp",
2277
+ "act_data_type": "fp"
2278
+ },
2279
+ "model.layers.23.mlp.shared_experts.gate_proj": {
2280
+ "bits": 16,
2281
+ "data_type": "fp",
2282
+ "act_data_type": "fp"
2283
+ },
2284
+ "model.layers.23.mlp.shared_experts.up_proj": {
2285
+ "bits": 16,
2286
+ "data_type": "fp",
2287
+ "act_data_type": "fp"
2288
+ },
2289
+ "model.layers.23.mlp.shared_experts.down_proj": {
2290
+ "bits": 16,
2291
+ "data_type": "fp",
2292
+ "act_data_type": "fp"
2293
+ },
2294
+ "model.layers.24.mlp.shared_experts.gate_proj": {
2295
+ "bits": 16,
2296
+ "data_type": "fp",
2297
+ "act_data_type": "fp"
2298
+ },
2299
+ "model.layers.24.mlp.shared_experts.up_proj": {
2300
+ "bits": 16,
2301
+ "data_type": "fp",
2302
+ "act_data_type": "fp"
2303
+ },
2304
+ "model.layers.24.mlp.shared_experts.down_proj": {
2305
+ "bits": 16,
2306
+ "data_type": "fp",
2307
+ "act_data_type": "fp"
2308
+ },
2309
+ "model.layers.25.mlp.shared_experts.gate_proj": {
2310
+ "bits": 16,
2311
+ "data_type": "fp",
2312
+ "act_data_type": "fp"
2313
+ },
2314
+ "model.layers.25.mlp.shared_experts.up_proj": {
2315
+ "bits": 16,
2316
+ "data_type": "fp",
2317
+ "act_data_type": "fp"
2318
+ },
2319
+ "model.layers.25.mlp.shared_experts.down_proj": {
2320
+ "bits": 16,
2321
+ "data_type": "fp",
2322
+ "act_data_type": "fp"
2323
+ },
2324
+ "model.layers.26.mlp.shared_experts.gate_proj": {
2325
+ "bits": 16,
2326
+ "data_type": "fp",
2327
+ "act_data_type": "fp"
2328
+ },
2329
+ "model.layers.26.mlp.shared_experts.up_proj": {
2330
+ "bits": 16,
2331
+ "data_type": "fp",
2332
+ "act_data_type": "fp"
2333
+ },
2334
+ "model.layers.26.mlp.shared_experts.down_proj": {
2335
+ "bits": 16,
2336
+ "data_type": "fp",
2337
+ "act_data_type": "fp"
2338
+ },
2339
+ "model.layers.27.mlp.shared_experts.gate_proj": {
2340
+ "bits": 16,
2341
+ "data_type": "fp",
2342
+ "act_data_type": "fp"
2343
+ },
2344
+ "model.layers.27.mlp.shared_experts.up_proj": {
2345
+ "bits": 16,
2346
+ "data_type": "fp",
2347
+ "act_data_type": "fp"
2348
+ },
2349
+ "model.layers.27.mlp.shared_experts.down_proj": {
2350
+ "bits": 16,
2351
+ "data_type": "fp",
2352
+ "act_data_type": "fp"
2353
+ },
2354
+ "model.layers.28.mlp.shared_experts.gate_proj": {
2355
+ "bits": 16,
2356
+ "data_type": "fp",
2357
+ "act_data_type": "fp"
2358
+ },
2359
+ "model.layers.28.mlp.shared_experts.up_proj": {
2360
+ "bits": 16,
2361
+ "data_type": "fp",
2362
+ "act_data_type": "fp"
2363
+ },
2364
+ "model.layers.28.mlp.shared_experts.down_proj": {
2365
+ "bits": 16,
2366
+ "data_type": "fp",
2367
+ "act_data_type": "fp"
2368
+ },
2369
+ "model.layers.29.mlp.shared_experts.gate_proj": {
2370
+ "bits": 16,
2371
+ "data_type": "fp",
2372
+ "act_data_type": "fp"
2373
+ },
2374
+ "model.layers.29.mlp.shared_experts.up_proj": {
2375
+ "bits": 16,
2376
+ "data_type": "fp",
2377
+ "act_data_type": "fp"
2378
+ },
2379
+ "model.layers.29.mlp.shared_experts.down_proj": {
2380
+ "bits": 16,
2381
+ "data_type": "fp",
2382
+ "act_data_type": "fp"
2383
+ },
2384
+ "model.layers.30.mlp.shared_experts.gate_proj": {
2385
+ "bits": 16,
2386
+ "data_type": "fp",
2387
+ "act_data_type": "fp"
2388
+ },
2389
+ "model.layers.30.mlp.shared_experts.up_proj": {
2390
+ "bits": 16,
2391
+ "data_type": "fp",
2392
+ "act_data_type": "fp"
2393
+ },
2394
+ "model.layers.30.mlp.shared_experts.down_proj": {
2395
+ "bits": 16,
2396
+ "data_type": "fp",
2397
+ "act_data_type": "fp"
2398
+ },
2399
+ "model.layers.31.mlp.shared_experts.gate_proj": {
2400
+ "bits": 16,
2401
+ "data_type": "fp",
2402
+ "act_data_type": "fp"
2403
+ },
2404
+ "model.layers.31.mlp.shared_experts.up_proj": {
2405
+ "bits": 16,
2406
+ "data_type": "fp",
2407
+ "act_data_type": "fp"
2408
+ },
2409
+ "model.layers.31.mlp.shared_experts.down_proj": {
2410
+ "bits": 16,
2411
+ "data_type": "fp",
2412
+ "act_data_type": "fp"
2413
+ },
2414
+ "model.layers.32.mlp.shared_experts.gate_proj": {
2415
+ "bits": 16,
2416
+ "data_type": "fp",
2417
+ "act_data_type": "fp"
2418
+ },
2419
+ "model.layers.32.mlp.shared_experts.up_proj": {
2420
+ "bits": 16,
2421
+ "data_type": "fp",
2422
+ "act_data_type": "fp"
2423
+ },
2424
+ "model.layers.32.mlp.shared_experts.down_proj": {
2425
+ "bits": 16,
2426
+ "data_type": "fp",
2427
+ "act_data_type": "fp"
2428
+ },
2429
+ "model.layers.33.mlp.shared_experts.gate_proj": {
2430
+ "bits": 16,
2431
+ "data_type": "fp",
2432
+ "act_data_type": "fp"
2433
+ },
2434
+ "model.layers.33.mlp.shared_experts.up_proj": {
2435
+ "bits": 16,
2436
+ "data_type": "fp",
2437
+ "act_data_type": "fp"
2438
+ },
2439
+ "model.layers.33.mlp.shared_experts.down_proj": {
2440
+ "bits": 16,
2441
+ "data_type": "fp",
2442
+ "act_data_type": "fp"
2443
+ },
2444
+ "model.layers.34.mlp.shared_experts.gate_proj": {
2445
+ "bits": 16,
2446
+ "data_type": "fp",
2447
+ "act_data_type": "fp"
2448
+ },
2449
+ "model.layers.34.mlp.shared_experts.up_proj": {
2450
+ "bits": 16,
2451
+ "data_type": "fp",
2452
+ "act_data_type": "fp"
2453
+ },
2454
+ "model.layers.34.mlp.shared_experts.down_proj": {
2455
+ "bits": 16,
2456
+ "data_type": "fp",
2457
+ "act_data_type": "fp"
2458
+ },
2459
+ "model.layers.35.mlp.shared_experts.gate_proj": {
2460
+ "bits": 16,
2461
+ "data_type": "fp",
2462
+ "act_data_type": "fp"
2463
+ },
2464
+ "model.layers.35.mlp.shared_experts.up_proj": {
2465
+ "bits": 16,
2466
+ "data_type": "fp",
2467
+ "act_data_type": "fp"
2468
+ },
2469
+ "model.layers.35.mlp.shared_experts.down_proj": {
2470
+ "bits": 16,
2471
+ "data_type": "fp",
2472
+ "act_data_type": "fp"
2473
+ },
2474
+ "model.layers.36.mlp.shared_experts.gate_proj": {
2475
+ "bits": 16,
2476
+ "data_type": "fp",
2477
+ "act_data_type": "fp"
2478
+ },
2479
+ "model.layers.36.mlp.shared_experts.up_proj": {
2480
+ "bits": 16,
2481
+ "data_type": "fp",
2482
+ "act_data_type": "fp"
2483
+ },
2484
+ "model.layers.36.mlp.shared_experts.down_proj": {
2485
+ "bits": 16,
2486
+ "data_type": "fp",
2487
+ "act_data_type": "fp"
2488
+ },
2489
+ "model.layers.37.mlp.shared_experts.gate_proj": {
2490
+ "bits": 16,
2491
+ "data_type": "fp",
2492
+ "act_data_type": "fp"
2493
+ },
2494
+ "model.layers.37.mlp.shared_experts.up_proj": {
2495
+ "bits": 16,
2496
+ "data_type": "fp",
2497
+ "act_data_type": "fp"
2498
+ },
2499
+ "model.layers.37.mlp.shared_experts.down_proj": {
2500
+ "bits": 16,
2501
+ "data_type": "fp",
2502
+ "act_data_type": "fp"
2503
+ },
2504
+ "model.layers.38.mlp.shared_experts.gate_proj": {
2505
+ "bits": 16,
2506
+ "data_type": "fp",
2507
+ "act_data_type": "fp"
2508
+ },
2509
+ "model.layers.38.mlp.shared_experts.up_proj": {
2510
+ "bits": 16,
2511
+ "data_type": "fp",
2512
+ "act_data_type": "fp"
2513
+ },
2514
+ "model.layers.38.mlp.shared_experts.down_proj": {
2515
+ "bits": 16,
2516
+ "data_type": "fp",
2517
+ "act_data_type": "fp"
2518
+ },
2519
+ "model.layers.39.mlp.shared_experts.gate_proj": {
2520
+ "bits": 16,
2521
+ "data_type": "fp",
2522
+ "act_data_type": "fp"
2523
+ },
2524
+ "model.layers.39.mlp.shared_experts.up_proj": {
2525
+ "bits": 16,
2526
+ "data_type": "fp",
2527
+ "act_data_type": "fp"
2528
+ },
2529
+ "model.layers.39.mlp.shared_experts.down_proj": {
2530
+ "bits": 16,
2531
+ "data_type": "fp",
2532
+ "act_data_type": "fp"
2533
+ },
2534
+ "model.layers.40.mlp.shared_experts.gate_proj": {
2535
+ "bits": 16,
2536
+ "data_type": "fp",
2537
+ "act_data_type": "fp"
2538
+ },
2539
+ "model.layers.40.mlp.shared_experts.up_proj": {
2540
+ "bits": 16,
2541
+ "data_type": "fp",
2542
+ "act_data_type": "fp"
2543
+ },
2544
+ "model.layers.40.mlp.shared_experts.down_proj": {
2545
+ "bits": 16,
2546
+ "data_type": "fp",
2547
+ "act_data_type": "fp"
2548
+ },
2549
+ "model.layers.41.mlp.shared_experts.gate_proj": {
2550
+ "bits": 16,
2551
+ "data_type": "fp",
2552
+ "act_data_type": "fp"
2553
+ },
2554
+ "model.layers.41.mlp.shared_experts.up_proj": {
2555
+ "bits": 16,
2556
+ "data_type": "fp",
2557
+ "act_data_type": "fp"
2558
+ },
2559
+ "model.layers.41.mlp.shared_experts.down_proj": {
2560
+ "bits": 16,
2561
+ "data_type": "fp",
2562
+ "act_data_type": "fp"
2563
+ },
2564
+ "model.layers.42.mlp.shared_experts.gate_proj": {
2565
+ "bits": 16,
2566
+ "data_type": "fp",
2567
+ "act_data_type": "fp"
2568
+ },
2569
+ "model.layers.42.mlp.shared_experts.up_proj": {
2570
+ "bits": 16,
2571
+ "data_type": "fp",
2572
+ "act_data_type": "fp"
2573
+ },
2574
+ "model.layers.42.mlp.shared_experts.down_proj": {
2575
+ "bits": 16,
2576
+ "data_type": "fp",
2577
+ "act_data_type": "fp"
2578
+ },
2579
+ "model.layers.43.mlp.shared_experts.gate_proj": {
2580
+ "bits": 16,
2581
+ "data_type": "fp",
2582
+ "act_data_type": "fp"
2583
+ },
2584
+ "model.layers.43.mlp.shared_experts.up_proj": {
2585
+ "bits": 16,
2586
+ "data_type": "fp",
2587
+ "act_data_type": "fp"
2588
+ },
2589
+ "model.layers.43.mlp.shared_experts.down_proj": {
2590
+ "bits": 16,
2591
+ "data_type": "fp",
2592
+ "act_data_type": "fp"
2593
+ },
2594
+ "model.layers.44.mlp.shared_experts.gate_proj": {
2595
+ "bits": 16,
2596
+ "data_type": "fp",
2597
+ "act_data_type": "fp"
2598
+ },
2599
+ "model.layers.44.mlp.shared_experts.up_proj": {
2600
+ "bits": 16,
2601
+ "data_type": "fp",
2602
+ "act_data_type": "fp"
2603
+ },
2604
+ "model.layers.44.mlp.shared_experts.down_proj": {
2605
+ "bits": 16,
2606
+ "data_type": "fp",
2607
+ "act_data_type": "fp"
2608
+ },
2609
+ "model.layers.45.mlp.shared_experts.gate_proj": {
2610
+ "bits": 16,
2611
+ "data_type": "fp",
2612
+ "act_data_type": "fp"
2613
+ },
2614
+ "model.layers.45.mlp.shared_experts.up_proj": {
2615
+ "bits": 16,
2616
+ "data_type": "fp",
2617
+ "act_data_type": "fp"
2618
+ },
2619
+ "model.layers.45.mlp.shared_experts.down_proj": {
2620
+ "bits": 16,
2621
+ "data_type": "fp",
2622
+ "act_data_type": "fp"
2623
+ },
2624
+ "model.layers.46.mlp.shared_experts.gate_proj": {
2625
+ "bits": 16,
2626
+ "data_type": "fp",
2627
+ "act_data_type": "fp"
2628
+ },
2629
+ "model.layers.46.mlp.shared_experts.up_proj": {
2630
+ "bits": 16,
2631
+ "data_type": "fp",
2632
+ "act_data_type": "fp"
2633
+ },
2634
+ "model.layers.46.mlp.shared_experts.down_proj": {
2635
+ "bits": 16,
2636
+ "data_type": "fp",
2637
+ "act_data_type": "fp"
2638
+ },
2639
+ "model.layers.47.mlp.shared_experts.gate_proj": {
2640
+ "bits": 16,
2641
+ "data_type": "fp",
2642
+ "act_data_type": "fp"
2643
+ },
2644
+ "model.layers.47.mlp.shared_experts.up_proj": {
2645
+ "bits": 16,
2646
+ "data_type": "fp",
2647
+ "act_data_type": "fp"
2648
+ },
2649
+ "model.layers.47.mlp.shared_experts.down_proj": {
2650
+ "bits": 16,
2651
+ "data_type": "fp",
2652
+ "act_data_type": "fp"
2653
+ },
2654
+ ".*model\\.layers\\.[0-47]\\.mlp\\.gate.*": {
2655
+ "bits": 16,
2656
+ "data_type": "float"
2657
+ }
2658
+ }
2659
+ }
special_tokens_map.json ADDED
@@ -0,0 +1,4008 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "additional_special_tokens": [
3
+ "<unk>",
4
+ "<|startoftext|>",
5
+ "<|endoftext|>",
6
+ "<|fim_prefix|>",
7
+ "<|fim_middle|>",
8
+ "<|fim_suffix|>",
9
+ "<|repo_name|>",
10
+ "<|file_sep|>",
11
+ "<|pr|>",
12
+ "<|pr_status|>",
13
+ "<|pr_base|>",
14
+ "<|pr_file|>",
15
+ "<|pr_base_code|>",
16
+ "<|pr_diff|>",
17
+ "<|pr_diff_hunk|>",
18
+ "<|pr_comment|>",
19
+ "<|pr_event_id|>",
20
+ "<|pr_review|>",
21
+ "<|pr_review_state|>",
22
+ "<|pr_review_comment|>",
23
+ "<|pr_in_reply_to_review_id|>",
24
+ "<|pr_in_reply_to_comment_id|>",
25
+ "<|pr_diff_hunk_comment_line|>",
26
+ "<|pr_is_merged|>",
27
+ "<|pr_linked_issue|>",
28
+ "<|pr_linked_issue_body|>",
29
+ "<|special_26|>",
30
+ "<|special_27|>",
31
+ "<|special_28|>",
32
+ "<|special_29|>",
33
+ "<|special_30|>",
34
+ "<|special_31|>",
35
+ "<|special_32|>",
36
+ "<|special_33|>",
37
+ "<|special_34|>",
38
+ "<|special_35|>",
39
+ "<|special_36|>",
40
+ "<|special_37|>",
41
+ "<|special_38|>",
42
+ "<|special_39|>",
43
+ "<|special_40|>",
44
+ "<|special_41|>",
45
+ "<|special_42|>",
46
+ "<|special_43|>",
47
+ "<|special_44|>",
48
+ "<|special_45|>",
49
+ "<|special_46|>",
50
+ "<|special_47|>",
51
+ "<|special_48|>",
52
+ "<|special_49|>",
53
+ "<|special_50|>",
54
+ "<|special_51|>",
55
+ "<|special_52|>",
56
+ "<|special_53|>",
57
+ "<|special_54|>",
58
+ "<|special_55|>",
59
+ "<|special_56|>",
60
+ "<|special_57|>",
61
+ "<|special_58|>",
62
+ "<|special_59|>",
63
+ "<|special_60|>",
64
+ "<|special_61|>",
65
+ "<|special_62|>",
66
+ "<|special_63|>",
67
+ "<|special_64|>",
68
+ "<|special_65|>",
69
+ "<|special_66|>",
70
+ "<|special_67|>",
71
+ "<|special_68|>",
72
+ "<|special_69|>",
73
+ "<|special_70|>",
74
+ "<|special_71|>",
75
+ "<|special_72|>",
76
+ "<|special_73|>",
77
+ "<|special_74|>",
78
+ "<|special_75|>",
79
+ "<|special_76|>",
80
+ "<|special_77|>",
81
+ "<|special_78|>",
82
+ "<|special_79|>",
83
+ "<|special_80|>",
84
+ "<|special_81|>",
85
+ "<|special_82|>",
86
+ "<|special_83|>",
87
+ "<|special_84|>",
88
+ "<|special_85|>",
89
+ "<|special_86|>",
90
+ "<|special_87|>",
91
+ "<|special_88|>",
92
+ "<|special_89|>",
93
+ "<|special_90|>",
94
+ "<|special_91|>",
95
+ "<|special_92|>",
96
+ "<|special_93|>",
97
+ "<|special_94|>",
98
+ "<|special_95|>",
99
+ "<|special_96|>",
100
+ "<|special_97|>",
101
+ "<|special_98|>",
102
+ "<|special_99|>",
103
+ "<|special_100|>",
104
+ "<|special_101|>",
105
+ "<|special_102|>",
106
+ "<|special_103|>",
107
+ "<|special_104|>",
108
+ "<|special_105|>",
109
+ "<|special_106|>",
110
+ "<|special_107|>",
111
+ "<|special_108|>",
112
+ "<|special_109|>",
113
+ "<|special_110|>",
114
+ "<|special_111|>",
115
+ "<|special_112|>",
116
+ "<|special_113|>",
117
+ "<|special_114|>",
118
+ "<|special_115|>",
119
+ "<|special_116|>",
120
+ "<|special_117|>",
121
+ "<|special_118|>",
122
+ "<|special_119|>",
123
+ "<|special_120|>",
124
+ "<|special_121|>",
125
+ "<|special_122|>",
126
+ "<|special_123|>",
127
+ "<|special_124|>",
128
+ "<|special_125|>",
129
+ "<|special_126|>",
130
+ "<|special_127|>",
131
+ "<|im:start|>",
132
+ "<|im:end|>",
133
+ "<|im:content|>",
134
+ "<|tool:start|>",
135
+ "<|tool:end|>",
136
+ "<|tool_response:start|>",
137
+ "<|tool_response:end|>",
138
+ "<|special_142|>",
139
+ "<|special_143|>",
140
+ "<|special_144|>",
141
+ "<|special_145|>",
142
+ "<|special_146|>",
143
+ "<|special_147|>",
144
+ "<|special_148|>",
145
+ "<|special_149|>",
146
+ "<|special_150|>",
147
+ "<|special_151|>",
148
+ "<|special_152|>",
149
+ "<|special_153|>",
150
+ "<|special_154|>",
151
+ "<|special_155|>",
152
+ "<|special_156|>",
153
+ "<|special_157|>",
154
+ "<|special_158|>",
155
+ "<|special_159|>",
156
+ "<|special_160|>",
157
+ "<|special_161|>",
158
+ "<|special_162|>",
159
+ "<|special_163|>",
160
+ "<|special_164|>",
161
+ "<|special_165|>",
162
+ "<|special_166|>",
163
+ "<|special_167|>",
164
+ "<|special_168|>",
165
+ "<|special_169|>",
166
+ "<|special_170|>",
167
+ "<|special_171|>",
168
+ "<|special_172|>",
169
+ "<|special_173|>",
170
+ "<|special_174|>",
171
+ "<|special_175|>",
172
+ "<|special_176|>",
173
+ "<|special_177|>",
174
+ "<|special_178|>",
175
+ "<|special_179|>",
176
+ "<|special_180|>",
177
+ "<|special_181|>",
178
+ "<|special_182|>",
179
+ "<|special_183|>",
180
+ "<|special_184|>",
181
+ "<|special_185|>",
182
+ "<|special_186|>",
183
+ "<|special_187|>",
184
+ "<|special_188|>",
185
+ "<|special_189|>",
186
+ "<|special_190|>",
187
+ "<|special_191|>",
188
+ "<|special_192|>",
189
+ "<|special_193|>",
190
+ "<|special_194|>",
191
+ "<|special_195|>",
192
+ "<|special_196|>",
193
+ "<|special_197|>",
194
+ "<|special_198|>",
195
+ "<|special_199|>",
196
+ "<|special_200|>",
197
+ "<|special_201|>",
198
+ "<|special_202|>",
199
+ "<|special_203|>",
200
+ "<|special_204|>",
201
+ "<|special_205|>",
202
+ "<|special_206|>",
203
+ "<|special_207|>",
204
+ "<|special_208|>",
205
+ "<|special_209|>",
206
+ "<|special_210|>",
207
+ "<|special_211|>",
208
+ "<|special_212|>",
209
+ "<|special_213|>",
210
+ "<|special_214|>",
211
+ "<|special_215|>",
212
+ "<|special_216|>",
213
+ "<|special_217|>",
214
+ "<|special_218|>",
215
+ "<|special_219|>",
216
+ "<|special_220|>",
217
+ "<|special_221|>",
218
+ "<|special_222|>",
219
+ "<|special_223|>",
220
+ "<|special_224|>",
221
+ "<|special_225|>",
222
+ "<|special_226|>",
223
+ "<|special_227|>",
224
+ "<|special_228|>",
225
+ "<|special_229|>",
226
+ "<|special_230|>",
227
+ "<|special_231|>",
228
+ "<|special_232|>",
229
+ "<|special_233|>",
230
+ "<|special_234|>",
231
+ "<|special_235|>",
232
+ "<|special_236|>",
233
+ "<|special_237|>",
234
+ "<|special_238|>",
235
+ "<|special_239|>",
236
+ "<|special_240|>",
237
+ "<|special_241|>",
238
+ "<|special_242|>",
239
+ "<|special_243|>",
240
+ "<|special_244|>",
241
+ "<|special_245|>",
242
+ "<|special_246|>",
243
+ "<|special_247|>",
244
+ "<|special_248|>",
245
+ "<|special_249|>",
246
+ "<|special_250|>",
247
+ "<|special_251|>",
248
+ "<|special_252|>",
249
+ "<|special_253|>",
250
+ "<|special_254|>",
251
+ "<|special_255|>",
252
+ "<|special_256|>",
253
+ "<|special_257|>",
254
+ "<|special_258|>",
255
+ "<|special_259|>",
256
+ "<|special_260|>",
257
+ "<|special_261|>",
258
+ "<|special_262|>",
259
+ "<|special_263|>",
260
+ "<|special_264|>",
261
+ "<|special_265|>",
262
+ "<|special_266|>",
263
+ "<|special_267|>",
264
+ "<|special_268|>",
265
+ "<|special_269|>",
266
+ "<|special_270|>",
267
+ "<|special_271|>",
268
+ "<|special_272|>",
269
+ "<|special_273|>",
270
+ "<|special_274|>",
271
+ "<|special_275|>",
272
+ "<|special_276|>",
273
+ "<|special_277|>",
274
+ "<|special_278|>",
275
+ "<|special_279|>",
276
+ "<|special_280|>",
277
+ "<|special_281|>",
278
+ "<|special_282|>",
279
+ "<|special_283|>",
280
+ "<|special_284|>",
281
+ "<|special_285|>",
282
+ "<|special_286|>",
283
+ "<|special_287|>",
284
+ "<|special_288|>",
285
+ "<|special_289|>",
286
+ "<|special_290|>",
287
+ "<|special_291|>",
288
+ "<|special_292|>",
289
+ "<|special_293|>",
290
+ "<|special_294|>",
291
+ "<|special_295|>",
292
+ "<|special_296|>",
293
+ "<|special_297|>",
294
+ "<|special_298|>",
295
+ "<|special_299|>",
296
+ "<|special_300|>",
297
+ "<|special_301|>",
298
+ "<|special_302|>",
299
+ "<|special_303|>",
300
+ "<|special_304|>",
301
+ "<|special_305|>",
302
+ "<|special_306|>",
303
+ "<|special_307|>",
304
+ "<|special_308|>",
305
+ "<|special_309|>",
306
+ "<|special_310|>",
307
+ "<|special_311|>",
308
+ "<|special_312|>",
309
+ "<|special_313|>",
310
+ "<|special_314|>",
311
+ "<|special_315|>",
312
+ "<|special_316|>",
313
+ "<|special_317|>",
314
+ "<|special_318|>",
315
+ "<|special_319|>",
316
+ "<|special_320|>",
317
+ "<|special_321|>",
318
+ "<|special_322|>",
319
+ "<|special_323|>",
320
+ "<|special_324|>",
321
+ "<|special_325|>",
322
+ "<|special_326|>",
323
+ "<|special_327|>",
324
+ "<|special_328|>",
325
+ "<|special_329|>",
326
+ "<|special_330|>",
327
+ "<|special_331|>",
328
+ "<|special_332|>",
329
+ "<|special_333|>",
330
+ "<|special_334|>",
331
+ "<|special_335|>",
332
+ "<|special_336|>",
333
+ "<|special_337|>",
334
+ "<|special_338|>",
335
+ "<|special_339|>",
336
+ "<|special_340|>",
337
+ "<|special_341|>",
338
+ "<|special_342|>",
339
+ "<|special_343|>",
340
+ "<|special_344|>",
341
+ "<|special_345|>",
342
+ "<|special_346|>",
343
+ "<|special_347|>",
344
+ "<|special_348|>",
345
+ "<|special_349|>",
346
+ "<|special_350|>",
347
+ "<|special_351|>",
348
+ "<|special_352|>",
349
+ "<|special_353|>",
350
+ "<|special_354|>",
351
+ "<|special_355|>",
352
+ "<|special_356|>",
353
+ "<|special_357|>",
354
+ "<|special_358|>",
355
+ "<|special_359|>",
356
+ "<|special_360|>",
357
+ "<|special_361|>",
358
+ "<|special_362|>",
359
+ "<|special_363|>",
360
+ "<|special_364|>",
361
+ "<|special_365|>",
362
+ "<|special_366|>",
363
+ "<|special_367|>",
364
+ "<|special_368|>",
365
+ "<|special_369|>",
366
+ "<|special_370|>",
367
+ "<|special_371|>",
368
+ "<|special_372|>",
369
+ "<|special_373|>",
370
+ "<|special_374|>",
371
+ "<|special_375|>",
372
+ "<|special_376|>",
373
+ "<|special_377|>",
374
+ "<|special_378|>",
375
+ "<|special_379|>",
376
+ "<|special_380|>",
377
+ "<|special_381|>",
378
+ "<|special_382|>",
379
+ "<|special_383|>",
380
+ "<|special_384|>",
381
+ "<|special_385|>",
382
+ "<|special_386|>",
383
+ "<|special_387|>",
384
+ "<|special_388|>",
385
+ "<|special_389|>",
386
+ "<|special_390|>",
387
+ "<|special_391|>",
388
+ "<|special_392|>",
389
+ "<|special_393|>",
390
+ "<|special_394|>",
391
+ "<|special_395|>",
392
+ "<|special_396|>",
393
+ "<|special_397|>",
394
+ "<|special_398|>",
395
+ "<|special_399|>",
396
+ "<|special_400|>",
397
+ "<|special_401|>",
398
+ "<|special_402|>",
399
+ "<|special_403|>",
400
+ "<|special_404|>",
401
+ "<|special_405|>",
402
+ "<|special_406|>",
403
+ "<|special_407|>",
404
+ "<|special_408|>",
405
+ "<|special_409|>",
406
+ "<|special_410|>",
407
+ "<|special_411|>",
408
+ "<|special_412|>",
409
+ "<|special_413|>",
410
+ "<|special_414|>",
411
+ "<|special_415|>",
412
+ "<|special_416|>",
413
+ "<|special_417|>",
414
+ "<|special_418|>",
415
+ "<|special_419|>",
416
+ "<|special_420|>",
417
+ "<|special_421|>",
418
+ "<|special_422|>",
419
+ "<|special_423|>",
420
+ "<|special_424|>",
421
+ "<|special_425|>",
422
+ "<|special_426|>",
423
+ "<|special_427|>",
424
+ "<|special_428|>",
425
+ "<|special_429|>",
426
+ "<|special_430|>",
427
+ "<|special_431|>",
428
+ "<|special_432|>",
429
+ "<|special_433|>",
430
+ "<|special_434|>",
431
+ "<|special_435|>",
432
+ "<|special_436|>",
433
+ "<|special_437|>",
434
+ "<|special_438|>",
435
+ "<|special_439|>",
436
+ "<|special_440|>",
437
+ "<|special_441|>",
438
+ "<|special_442|>",
439
+ "<|special_443|>",
440
+ "<|special_444|>",
441
+ "<|special_445|>",
442
+ "<|special_446|>",
443
+ "<|special_447|>",
444
+ "<|special_448|>",
445
+ "<|special_449|>",
446
+ "<|special_450|>",
447
+ "<|special_451|>",
448
+ "<|special_452|>",
449
+ "<|special_453|>",
450
+ "<|special_454|>",
451
+ "<|special_455|>",
452
+ "<|special_456|>",
453
+ "<|special_457|>",
454
+ "<|special_458|>",
455
+ "<|special_459|>",
456
+ "<|special_460|>",
457
+ "<|special_461|>",
458
+ "<|special_462|>",
459
+ "<|special_463|>",
460
+ "<|special_464|>",
461
+ "<|special_465|>",
462
+ "<|special_466|>",
463
+ "<|special_467|>",
464
+ "<|special_468|>",
465
+ "<|special_469|>",
466
+ "<|special_470|>",
467
+ "<|special_471|>",
468
+ "<|special_472|>",
469
+ "<|special_473|>",
470
+ "<|special_474|>",
471
+ "<|special_475|>",
472
+ "<|special_476|>",
473
+ "<|special_477|>",
474
+ "<|special_478|>",
475
+ "<|special_479|>",
476
+ "<|special_480|>",
477
+ "<|special_481|>",
478
+ "<|special_482|>",
479
+ "<|special_483|>",
480
+ "<|special_484|>",
481
+ "<|special_485|>",
482
+ "<|special_486|>",
483
+ "<|special_487|>",
484
+ "<|special_488|>",
485
+ "<|special_489|>",
486
+ "<|special_490|>",
487
+ "<|special_491|>",
488
+ "<|special_492|>",
489
+ "<|special_493|>",
490
+ "<|special_494|>",
491
+ "<|special_495|>",
492
+ "<|special_496|>",
493
+ "<|special_497|>",
494
+ "<|special_498|>",
495
+ "<|special_499|>",
496
+ "<|special_500|>",
497
+ "<|special_501|>",
498
+ "<|special_502|>",
499
+ "<|special_503|>",
500
+ "<|special_504|>",
501
+ "<|special_505|>",
502
+ "<|special_506|>",
503
+ "<|special_507|>",
504
+ "<|special_508|>",
505
+ "<|special_509|>",
506
+ "<|special_510|>",
507
+ "<|special_511|>",
508
+ "<|special_625|>",
509
+ "<|special_626|>",
510
+ "<|special_627|>",
511
+ "<|special_628|>",
512
+ "<|special_629|>",
513
+ "<|special_630|>",
514
+ "<|special_631|>",
515
+ "<|special_632|>",
516
+ "<|special_633|>",
517
+ "<|special_634|>",
518
+ "<|special_635|>",
519
+ "<|special_636|>",
520
+ "<|special_637|>",
521
+ "<|special_638|>",
522
+ "<|special_639|>",
523
+ "<|special_640|>",
524
+ "<|special_641|>",
525
+ "<|special_642|>",
526
+ "<|special_643|>",
527
+ "<|special_644|>",
528
+ "<|special_645|>",
529
+ "<|special_646|>",
530
+ "<|special_647|>",
531
+ "<|special_648|>",
532
+ "<|special_649|>",
533
+ "<|special_650|>",
534
+ "<|special_651|>",
535
+ "<|special_652|>",
536
+ "<|special_653|>",
537
+ "<|special_654|>",
538
+ "<|special_655|>",
539
+ "<|special_656|>",
540
+ "<|special_657|>",
541
+ "<|special_658|>",
542
+ "<|special_659|>",
543
+ "<|special_660|>",
544
+ "<|special_661|>",
545
+ "<|special_662|>",
546
+ "<|special_663|>",
547
+ "<|special_664|>",
548
+ "<|special_665|>",
549
+ "<|special_666|>",
550
+ "<|special_667|>",
551
+ "<|special_668|>",
552
+ "<|special_669|>",
553
+ "<|special_670|>",
554
+ "<|special_671|>",
555
+ "<|special_672|>",
556
+ "<|special_673|>",
557
+ "<|special_674|>",
558
+ "<|special_675|>",
559
+ "<|special_676|>",
560
+ "<|special_677|>",
561
+ "<|special_678|>",
562
+ "<|special_679|>",
563
+ "<|special_680|>",
564
+ "<|special_681|>",
565
+ "<|special_682|>",
566
+ "<|special_683|>",
567
+ "<|special_684|>",
568
+ "<|special_685|>",
569
+ "<|special_686|>",
570
+ "<|special_687|>",
571
+ "<|special_688|>",
572
+ "<|special_689|>",
573
+ "<|special_690|>",
574
+ "<|special_691|>",
575
+ "<|special_692|>",
576
+ "<|special_693|>",
577
+ "<|special_694|>",
578
+ "<|special_695|>",
579
+ "<|special_696|>",
580
+ "<|special_697|>",
581
+ "<|special_698|>",
582
+ "<|special_699|>",
583
+ "<|special_700|>",
584
+ "<|special_701|>",
585
+ "<|special_702|>",
586
+ "<|special_703|>",
587
+ "<|special_704|>",
588
+ "<|special_705|>",
589
+ "<|special_706|>",
590
+ "<|special_707|>",
591
+ "<|special_708|>",
592
+ "<|special_709|>",
593
+ "<|special_710|>",
594
+ "<|special_711|>",
595
+ "<|special_712|>",
596
+ "<|special_713|>",
597
+ "<|special_714|>",
598
+ "<|special_715|>",
599
+ "<|special_716|>",
600
+ "<|special_717|>",
601
+ "<|special_718|>",
602
+ "<|special_719|>",
603
+ "<|special_720|>",
604
+ "<|special_721|>",
605
+ "<|special_722|>",
606
+ "<|special_723|>",
607
+ "<|special_724|>",
608
+ "<|special_725|>",
609
+ "<|special_726|>",
610
+ "<|special_727|>",
611
+ "<|special_728|>",
612
+ "<|special_729|>",
613
+ "<|special_730|>",
614
+ "<|special_731|>",
615
+ "<|special_732|>",
616
+ "<|special_733|>",
617
+ "<|special_734|>",
618
+ "<|special_735|>",
619
+ "<|special_736|>",
620
+ "<|special_737|>",
621
+ "<|special_738|>",
622
+ "<|special_739|>",
623
+ "<|special_740|>",
624
+ "<|special_741|>",
625
+ "<|special_742|>",
626
+ "<|special_743|>",
627
+ "<|special_744|>",
628
+ "<|special_745|>",
629
+ "<|special_746|>",
630
+ "<|special_747|>",
631
+ "<|special_748|>",
632
+ "<|special_749|>",
633
+ "<|special_750|>",
634
+ "<|special_751|>",
635
+ "<|special_752|>",
636
+ "<|special_753|>",
637
+ "<|special_754|>",
638
+ "<|special_755|>",
639
+ "<|special_756|>",
640
+ "<|special_757|>",
641
+ "<|special_758|>",
642
+ "<|special_759|>",
643
+ "<|special_760|>",
644
+ "<|special_761|>",
645
+ "<|special_762|>",
646
+ "<|special_763|>",
647
+ "<|special_764|>",
648
+ "<|special_765|>",
649
+ "<|special_766|>",
650
+ "<|special_767|>",
651
+ "<|special_768|>",
652
+ "<|special_769|>",
653
+ "<|special_770|>",
654
+ "<|special_771|>",
655
+ "<|special_772|>",
656
+ "<|special_773|>",
657
+ "<|special_774|>",
658
+ "<|special_775|>",
659
+ "<|special_776|>",
660
+ "<|special_777|>",
661
+ "<|special_778|>",
662
+ "<|special_779|>",
663
+ "<|special_780|>",
664
+ "<|special_781|>",
665
+ "<|special_782|>",
666
+ "<|special_783|>",
667
+ "<|special_784|>",
668
+ "<|special_785|>",
669
+ "<|special_786|>",
670
+ "<|special_787|>",
671
+ "<|special_788|>",
672
+ "<|special_789|>",
673
+ "<|special_790|>",
674
+ "<|special_791|>",
675
+ "<|special_792|>",
676
+ "<|special_793|>",
677
+ "<|special_794|>",
678
+ "<|special_795|>",
679
+ "<|special_796|>",
680
+ "<|special_797|>",
681
+ "<|special_798|>",
682
+ "<|special_799|>",
683
+ "<|special_800|>",
684
+ "<|special_801|>",
685
+ "<|special_802|>",
686
+ "<|special_803|>",
687
+ "<|special_804|>",
688
+ "<|special_805|>",
689
+ "<|special_806|>",
690
+ "<|special_807|>",
691
+ "<|special_808|>",
692
+ "<|special_809|>",
693
+ "<|special_810|>",
694
+ "<|special_811|>",
695
+ "<|special_812|>",
696
+ "<|special_813|>",
697
+ "<|special_814|>",
698
+ "<|special_815|>",
699
+ "<|special_816|>",
700
+ "<|special_817|>",
701
+ "<|special_818|>",
702
+ "<|special_819|>",
703
+ "<|special_820|>",
704
+ "<|special_821|>",
705
+ "<|special_822|>",
706
+ "<|special_823|>",
707
+ "<|special_824|>",
708
+ "<|special_825|>",
709
+ "<|special_826|>",
710
+ "<|special_827|>",
711
+ "<|special_828|>",
712
+ "<|special_829|>",
713
+ "<|special_830|>",
714
+ "<|special_831|>",
715
+ "<|special_832|>",
716
+ "<|special_833|>",
717
+ "<|special_834|>",
718
+ "<|special_835|>",
719
+ "<|special_836|>",
720
+ "<|special_837|>",
721
+ "<|special_838|>",
722
+ "<|special_839|>",
723
+ "<|special_840|>",
724
+ "<|special_841|>",
725
+ "<|special_842|>",
726
+ "<|special_843|>",
727
+ "<|special_844|>",
728
+ "<|special_845|>",
729
+ "<|special_846|>",
730
+ "<|special_847|>",
731
+ "<|special_848|>",
732
+ "<|special_849|>",
733
+ "<|special_850|>",
734
+ "<|special_851|>",
735
+ "<|special_852|>",
736
+ "<|special_853|>",
737
+ "<|special_854|>",
738
+ "<|special_855|>",
739
+ "<|special_856|>",
740
+ "<|special_857|>",
741
+ "<|special_858|>",
742
+ "<|special_859|>",
743
+ "<|special_860|>",
744
+ "<|special_861|>",
745
+ "<|special_862|>",
746
+ "<|special_863|>",
747
+ "<|special_864|>",
748
+ "<|special_865|>",
749
+ "<|special_866|>",
750
+ "<|special_867|>",
751
+ "<|special_868|>",
752
+ "<|special_869|>",
753
+ "<|special_870|>",
754
+ "<|special_871|>",
755
+ "<|special_872|>",
756
+ "<|special_873|>",
757
+ "<|special_874|>",
758
+ "<|special_875|>",
759
+ "<|special_876|>",
760
+ "<|special_877|>",
761
+ "<|special_878|>",
762
+ "<|special_879|>",
763
+ "<|special_880|>",
764
+ "<|special_881|>",
765
+ "<|special_882|>",
766
+ "<|special_883|>",
767
+ "<|special_884|>",
768
+ "<|special_885|>",
769
+ "<|special_886|>",
770
+ "<|special_887|>",
771
+ "<|special_888|>",
772
+ "<|special_889|>",
773
+ "<|special_890|>",
774
+ "<|special_891|>",
775
+ "<|special_892|>",
776
+ "<|special_893|>",
777
+ "<|special_894|>",
778
+ "<|special_895|>",
779
+ "<|special_896|>",
780
+ "<|special_897|>",
781
+ "<|special_898|>",
782
+ "<|special_899|>",
783
+ "<|special_900|>",
784
+ "<|special_901|>",
785
+ "<|special_902|>",
786
+ "<|special_903|>",
787
+ "<|special_904|>",
788
+ "<|special_905|>",
789
+ "<|special_906|>",
790
+ "<|special_907|>",
791
+ "<|special_908|>",
792
+ "<|special_909|>",
793
+ "<|special_910|>",
794
+ "<|special_911|>",
795
+ "<|special_912|>",
796
+ "<|special_913|>",
797
+ "<|special_914|>",
798
+ "<|special_915|>",
799
+ "<|special_916|>",
800
+ "<|special_917|>",
801
+ "<|special_918|>",
802
+ "<|special_919|>",
803
+ "<|special_920|>",
804
+ "<|special_921|>",
805
+ "<|special_922|>",
806
+ "<|special_923|>",
807
+ "<|special_924|>",
808
+ "<|special_925|>",
809
+ "<|special_926|>",
810
+ "<|special_927|>",
811
+ "<|special_928|>",
812
+ "<|special_929|>",
813
+ "<|special_930|>",
814
+ "<|special_931|>",
815
+ "<|special_932|>",
816
+ "<|special_933|>",
817
+ "<|special_934|>",
818
+ "<|special_935|>",
819
+ "<|special_936|>",
820
+ "<|special_937|>",
821
+ "<|special_938|>",
822
+ "<|special_939|>",
823
+ "<|special_940|>",
824
+ "<|special_941|>",
825
+ "<|special_942|>",
826
+ "<|special_943|>",
827
+ "<|special_944|>",
828
+ "<|special_945|>",
829
+ "<|special_946|>",
830
+ "<|special_947|>",
831
+ "<|special_948|>",
832
+ "<|special_949|>",
833
+ "<|special_950|>",
834
+ "<|special_951|>",
835
+ "<|special_952|>",
836
+ "<|special_953|>",
837
+ "<|special_954|>",
838
+ "<|special_955|>",
839
+ "<|special_956|>",
840
+ "<|special_957|>",
841
+ "<|special_958|>",
842
+ "<|special_959|>",
843
+ "<|special_960|>",
844
+ "<|special_961|>",
845
+ "<|special_962|>",
846
+ "<|special_963|>",
847
+ "<|special_964|>",
848
+ "<|special_965|>",
849
+ "<|special_966|>",
850
+ "<|special_967|>",
851
+ "<|special_968|>",
852
+ "<|special_969|>",
853
+ "<|special_970|>",
854
+ "<|special_971|>",
855
+ "<|special_972|>",
856
+ "<|special_973|>",
857
+ "<|special_974|>",
858
+ "<|special_975|>",
859
+ "<|special_976|>",
860
+ "<|special_977|>",
861
+ "<|special_978|>",
862
+ "<|special_979|>",
863
+ "<|special_980|>",
864
+ "<|special_981|>",
865
+ "<|special_982|>",
866
+ "<|special_983|>",
867
+ "<|special_984|>",
868
+ "<|special_985|>",
869
+ "<|special_986|>",
870
+ "<|special_987|>",
871
+ "<|special_988|>",
872
+ "<|special_989|>",
873
+ "<|special_990|>",
874
+ "<|special_991|>",
875
+ "<|special_992|>",
876
+ "<|special_993|>",
877
+ "<|special_994|>",
878
+ "<|special_995|>",
879
+ "<|special_996|>",
880
+ "<|special_997|>",
881
+ "<|special_998|>",
882
+ "<|special_999|>",
883
+ "<|special_1000|>",
884
+ "<|special_1001|>",
885
+ "<|special_1002|>",
886
+ "<|special_1003|>",
887
+ "<|special_1004|>",
888
+ "<|special_1005|>",
889
+ "<|special_1006|>",
890
+ "<|special_1007|>",
891
+ "<|special_1008|>",
892
+ "<|special_1009|>",
893
+ "<|special_1010|>",
894
+ "<|special_1011|>",
895
+ "<|special_1012|>",
896
+ "<|special_1013|>",
897
+ "<|special_1014|>",
898
+ "<|special_1015|>",
899
+ "<|special_1016|>",
900
+ "<|special_1017|>",
901
+ "<|special_1018|>",
902
+ "<|special_1019|>",
903
+ "<|special_1020|>",
904
+ "<|special_1021|>",
905
+ "<|special_1022|>",
906
+ "<|special_1023|>",
907
+ "<|special_1024|>",
908
+ "<|special_1025|>",
909
+ "<|special_1026|>",
910
+ "<|special_1027|>",
911
+ "<|special_1028|>",
912
+ "<|special_1029|>",
913
+ "<|special_1030|>",
914
+ "<|special_1031|>",
915
+ "<|special_1032|>",
916
+ "<|special_1033|>",
917
+ "<|special_1034|>",
918
+ "<|special_1035|>",
919
+ "<|special_1036|>",
920
+ "<|special_1037|>",
921
+ "<|special_1038|>",
922
+ "<|special_1039|>",
923
+ "<|special_1040|>",
924
+ "<|special_1041|>",
925
+ "<|special_1042|>",
926
+ "<|special_1043|>",
927
+ "<|special_1044|>",
928
+ "<|special_1045|>",
929
+ "<|special_1046|>",
930
+ "<|special_1047|>",
931
+ "<|special_1048|>",
932
+ "<|special_1049|>",
933
+ "<|special_1050|>",
934
+ "<|special_1051|>",
935
+ "<|special_1052|>",
936
+ "<|special_1053|>",
937
+ "<|special_1054|>",
938
+ "<|special_1055|>",
939
+ "<|special_1056|>",
940
+ "<|special_1057|>",
941
+ "<|special_1058|>",
942
+ "<|special_1059|>",
943
+ "<|special_1060|>",
944
+ "<|special_1061|>",
945
+ "<|special_1062|>",
946
+ "<|special_1063|>",
947
+ "<|special_1064|>",
948
+ "<|special_1065|>",
949
+ "<|special_1066|>",
950
+ "<|special_1067|>",
951
+ "<|special_1068|>",
952
+ "<|special_1069|>",
953
+ "<|special_1070|>",
954
+ "<|special_1071|>",
955
+ "<|special_1072|>",
956
+ "<|special_1073|>",
957
+ "<|special_1074|>",
958
+ "<|special_1075|>",
959
+ "<|special_1076|>",
960
+ "<|special_1077|>",
961
+ "<|special_1078|>",
962
+ "<|special_1079|>",
963
+ "<|special_1080|>",
964
+ "<|special_1081|>",
965
+ "<|special_1082|>",
966
+ "<|special_1083|>",
967
+ "<|special_1084|>",
968
+ "<|special_1085|>",
969
+ "<|special_1086|>",
970
+ "<|special_1087|>",
971
+ "<|special_1088|>",
972
+ "<|special_1089|>",
973
+ "<|special_1090|>",
974
+ "<|special_1091|>",
975
+ "<|special_1092|>",
976
+ "<|special_1093|>",
977
+ "<|special_1094|>",
978
+ "<|special_1095|>",
979
+ "<|special_1096|>",
980
+ "<|special_1097|>",
981
+ "<|special_1098|>",
982
+ "<|special_1099|>",
983
+ "<|special_1100|>",
984
+ "<|special_1101|>",
985
+ "<|special_1102|>",
986
+ "<|special_1103|>",
987
+ "<|special_1104|>",
988
+ "<|special_1105|>",
989
+ "<|special_1106|>",
990
+ "<|special_1107|>",
991
+ "<|special_1108|>",
992
+ "<|special_1109|>",
993
+ "<|special_1110|>",
994
+ "<|special_1111|>",
995
+ "<|special_1112|>",
996
+ "<|special_1113|>",
997
+ "<|special_1114|>",
998
+ "<|special_1115|>",
999
+ "<|special_1116|>",
1000
+ "<|special_1117|>",
1001
+ "<|special_1118|>",
1002
+ "<|special_1119|>",
1003
+ "<|special_1120|>",
1004
+ "<|special_1121|>",
1005
+ "<|special_1122|>",
1006
+ "<|special_1123|>",
1007
+ "<|special_1124|>",
1008
+ "<|special_1125|>",
1009
+ "<|special_1126|>",
1010
+ "<|special_1127|>",
1011
+ "<|special_1128|>",
1012
+ "<|special_1129|>",
1013
+ "<|special_1130|>",
1014
+ "<|special_1131|>",
1015
+ "<|special_1132|>",
1016
+ "<|special_1133|>",
1017
+ "<|special_1134|>",
1018
+ "<|special_1135|>",
1019
+ "<|special_1136|>",
1020
+ "<|special_1137|>",
1021
+ "<|special_1138|>",
1022
+ "<|special_1139|>",
1023
+ "<|special_1140|>",
1024
+ "<|special_1141|>",
1025
+ "<|special_1142|>",
1026
+ "<|special_1143|>",
1027
+ "<|special_1144|>",
1028
+ "<|special_1145|>",
1029
+ "<|special_1146|>",
1030
+ "<|special_1147|>",
1031
+ "<|special_1148|>",
1032
+ "<|special_1149|>",
1033
+ "<|special_1150|>",
1034
+ "<|special_1151|>",
1035
+ "<|special_1152|>",
1036
+ "<|special_1153|>",
1037
+ "<|special_1154|>",
1038
+ "<|special_1155|>",
1039
+ "<|special_1156|>",
1040
+ "<|special_1157|>",
1041
+ "<|special_1158|>",
1042
+ "<|special_1159|>",
1043
+ "<|special_1160|>",
1044
+ "<|special_1161|>",
1045
+ "<|special_1162|>",
1046
+ "<|special_1163|>",
1047
+ "<|special_1164|>",
1048
+ "<|special_1165|>",
1049
+ "<|special_1166|>",
1050
+ "<|special_1167|>",
1051
+ "<|special_1168|>",
1052
+ "<|special_1169|>",
1053
+ "<|special_1170|>",
1054
+ "<|special_1171|>",
1055
+ "<|special_1172|>",
1056
+ "<|special_1173|>",
1057
+ "<|special_1174|>",
1058
+ "<|special_1175|>",
1059
+ "<|special_1176|>",
1060
+ "<|special_1177|>",
1061
+ "<|special_1178|>",
1062
+ "<|special_1179|>",
1063
+ "<|special_1180|>",
1064
+ "<|special_1181|>",
1065
+ "<|special_1182|>",
1066
+ "<|special_1183|>",
1067
+ "<|special_1184|>",
1068
+ "<|special_1185|>",
1069
+ "<|special_1186|>",
1070
+ "<|special_1187|>",
1071
+ "<|special_1188|>",
1072
+ "<|special_1189|>",
1073
+ "<|special_1190|>",
1074
+ "<|special_1191|>",
1075
+ "<|special_1192|>",
1076
+ "<|special_1193|>",
1077
+ "<|special_1194|>",
1078
+ "<|special_1195|>",
1079
+ "<|special_1196|>",
1080
+ "<|special_1197|>",
1081
+ "<|special_1198|>",
1082
+ "<|special_1199|>",
1083
+ "<|special_1200|>",
1084
+ "<|special_1201|>",
1085
+ "<|special_1202|>",
1086
+ "<|special_1203|>",
1087
+ "<|special_1204|>",
1088
+ "<|special_1205|>",
1089
+ "<|special_1206|>",
1090
+ "<|special_1207|>",
1091
+ "<|special_1208|>",
1092
+ "<|special_1209|>",
1093
+ "<|special_1210|>",
1094
+ "<|special_1211|>",
1095
+ "<|special_1212|>",
1096
+ "<|special_1213|>",
1097
+ "<|special_1214|>",
1098
+ "<|special_1215|>",
1099
+ "<|special_1216|>",
1100
+ "<|special_1217|>",
1101
+ "<|special_1218|>",
1102
+ "<|special_1219|>",
1103
+ "<|special_1220|>",
1104
+ "<|special_1221|>",
1105
+ "<|special_1222|>",
1106
+ "<|special_1223|>",
1107
+ "<|special_1224|>",
1108
+ "<|special_1225|>",
1109
+ "<|special_1226|>",
1110
+ "<|special_1227|>",
1111
+ "<|special_1228|>",
1112
+ "<|special_1229|>",
1113
+ "<|special_1230|>",
1114
+ "<|special_1231|>",
1115
+ "<|special_1232|>",
1116
+ "<|special_1233|>",
1117
+ "<|special_1234|>",
1118
+ "<|special_1235|>",
1119
+ "<|special_1236|>",
1120
+ "<|special_1237|>",
1121
+ "<|special_1238|>",
1122
+ "<|special_1239|>",
1123
+ "<|special_1240|>",
1124
+ "<|special_1241|>",
1125
+ "<|special_1242|>",
1126
+ "<|special_1243|>",
1127
+ "<|special_1244|>",
1128
+ "<|special_1245|>",
1129
+ "<|special_1246|>",
1130
+ "<|special_1247|>",
1131
+ "<|special_1248|>",
1132
+ "<|special_1249|>",
1133
+ "<|special_1250|>",
1134
+ "<|special_1251|>",
1135
+ "<|special_1252|>",
1136
+ "<|special_1253|>",
1137
+ "<|special_1254|>",
1138
+ "<|special_1255|>",
1139
+ "<|special_1256|>",
1140
+ "<|special_1257|>",
1141
+ "<|special_1258|>",
1142
+ "<|special_1259|>",
1143
+ "<|special_1260|>",
1144
+ "<|special_1261|>",
1145
+ "<|special_1262|>",
1146
+ "<|special_1263|>",
1147
+ "<|special_1264|>",
1148
+ "<|special_1265|>",
1149
+ "<|special_1266|>",
1150
+ "<|special_1267|>",
1151
+ "<|special_1268|>",
1152
+ "<|special_1269|>",
1153
+ "<|special_1270|>",
1154
+ "<|special_1271|>",
1155
+ "<|special_1272|>",
1156
+ "<|special_1273|>",
1157
+ "<|special_1274|>",
1158
+ "<|special_1275|>",
1159
+ "<|special_1276|>",
1160
+ "<|special_1277|>",
1161
+ "<|special_1278|>",
1162
+ "<|special_1279|>",
1163
+ "<|special_1280|>",
1164
+ "<|special_1281|>",
1165
+ "<|special_1282|>",
1166
+ "<|special_1283|>",
1167
+ "<|special_1284|>",
1168
+ "<|special_1285|>",
1169
+ "<|special_1286|>",
1170
+ "<|special_1287|>",
1171
+ "<|special_1288|>",
1172
+ "<|special_1289|>",
1173
+ "<|special_1290|>",
1174
+ "<|special_1291|>",
1175
+ "<|special_1292|>",
1176
+ "<|special_1293|>",
1177
+ "<|special_1294|>",
1178
+ "<|special_1295|>",
1179
+ "<|special_1296|>",
1180
+ "<|special_1297|>",
1181
+ "<|special_1298|>",
1182
+ "<|special_1299|>",
1183
+ "<|special_1300|>",
1184
+ "<|special_1301|>",
1185
+ "<|special_1302|>",
1186
+ "<|special_1303|>",
1187
+ "<|special_1304|>",
1188
+ "<|special_1305|>",
1189
+ "<|special_1306|>",
1190
+ "<|special_1307|>",
1191
+ "<|special_1308|>",
1192
+ "<|special_1309|>",
1193
+ "<|special_1310|>",
1194
+ "<|special_1311|>",
1195
+ "<|special_1312|>",
1196
+ "<|special_1313|>",
1197
+ "<|special_1314|>",
1198
+ "<|special_1315|>",
1199
+ "<|special_1316|>",
1200
+ "<|special_1317|>",
1201
+ "<|special_1318|>",
1202
+ "<|special_1319|>",
1203
+ "<|special_1320|>",
1204
+ "<|special_1321|>",
1205
+ "<|special_1322|>",
1206
+ "<|special_1323|>",
1207
+ "<|special_1324|>",
1208
+ "<|special_1325|>",
1209
+ "<|special_1326|>",
1210
+ "<|special_1327|>",
1211
+ "<|special_1328|>",
1212
+ "<|special_1329|>",
1213
+ "<|special_1330|>",
1214
+ "<|special_1331|>",
1215
+ "<|special_1332|>",
1216
+ "<|special_1333|>",
1217
+ "<|special_1334|>",
1218
+ "<|special_1335|>",
1219
+ "<|special_1336|>",
1220
+ "<|special_1337|>",
1221
+ "<|special_1338|>",
1222
+ "<|special_1339|>",
1223
+ "<|special_1340|>",
1224
+ "<|special_1341|>",
1225
+ "<|special_1342|>",
1226
+ "<|special_1343|>",
1227
+ "<|special_1344|>",
1228
+ "<|special_1345|>",
1229
+ "<|special_1346|>",
1230
+ "<|special_1347|>",
1231
+ "<|special_1348|>",
1232
+ "<|special_1349|>",
1233
+ "<|special_1350|>",
1234
+ "<|special_1351|>",
1235
+ "<|special_1352|>",
1236
+ "<|special_1353|>",
1237
+ "<|special_1354|>",
1238
+ "<|special_1355|>",
1239
+ "<|special_1356|>",
1240
+ "<|special_1357|>",
1241
+ "<|special_1358|>",
1242
+ "<|special_1359|>",
1243
+ "<|special_1360|>",
1244
+ "<|special_1361|>",
1245
+ "<|special_1362|>",
1246
+ "<|special_1363|>",
1247
+ "<|special_1364|>",
1248
+ "<|special_1365|>",
1249
+ "<|special_1366|>",
1250
+ "<|special_1367|>",
1251
+ "<|special_1368|>",
1252
+ "<|special_1369|>",
1253
+ "<|special_1370|>",
1254
+ "<|special_1371|>",
1255
+ "<|special_1372|>",
1256
+ "<|special_1373|>",
1257
+ "<|special_1374|>",
1258
+ "<|special_1375|>",
1259
+ "<|special_1376|>",
1260
+ "<|special_1377|>",
1261
+ "<|special_1378|>",
1262
+ "<|special_1379|>",
1263
+ "<|special_1380|>",
1264
+ "<|special_1381|>",
1265
+ "<|special_1382|>",
1266
+ "<|special_1383|>",
1267
+ "<|special_1384|>",
1268
+ "<|special_1385|>",
1269
+ "<|special_1386|>",
1270
+ "<|special_1387|>",
1271
+ "<|special_1388|>",
1272
+ "<|special_1389|>",
1273
+ "<|special_1390|>",
1274
+ "<|special_1391|>",
1275
+ "<|special_1392|>",
1276
+ "<|special_1393|>",
1277
+ "<|special_1394|>",
1278
+ "<|special_1395|>",
1279
+ "<|special_1396|>",
1280
+ "<|special_1397|>",
1281
+ "<|special_1398|>",
1282
+ "<|special_1399|>",
1283
+ "<|special_1400|>",
1284
+ "<|special_1401|>",
1285
+ "<|special_1402|>",
1286
+ "<|special_1403|>",
1287
+ "<|special_1404|>",
1288
+ "<|special_1405|>",
1289
+ "<|special_1406|>",
1290
+ "<|special_1407|>",
1291
+ "<|special_1408|>",
1292
+ "<|special_1409|>",
1293
+ "<|special_1410|>",
1294
+ "<|special_1411|>",
1295
+ "<|special_1412|>",
1296
+ "<|special_1413|>",
1297
+ "<|special_1414|>",
1298
+ "<|special_1415|>",
1299
+ "<|special_1416|>",
1300
+ "<|special_1417|>",
1301
+ "<|special_1418|>",
1302
+ "<|special_1419|>",
1303
+ "<|special_1420|>",
1304
+ "<|special_1421|>",
1305
+ "<|special_1422|>",
1306
+ "<|special_1423|>",
1307
+ "<|special_1424|>",
1308
+ "<|special_1425|>",
1309
+ "<|special_1426|>",
1310
+ "<|special_1427|>",
1311
+ "<|special_1428|>",
1312
+ "<|special_1429|>",
1313
+ "<|special_1430|>",
1314
+ "<|special_1431|>",
1315
+ "<|special_1432|>",
1316
+ "<|special_1433|>",
1317
+ "<|special_1434|>",
1318
+ "<|special_1435|>",
1319
+ "<|special_1436|>",
1320
+ "<|special_1437|>",
1321
+ "<|special_1438|>",
1322
+ "<|special_1439|>",
1323
+ "<|special_1440|>",
1324
+ "<|special_1441|>",
1325
+ "<|special_1442|>",
1326
+ "<|special_1443|>",
1327
+ "<|special_1444|>",
1328
+ "<|special_1445|>",
1329
+ "<|special_1446|>",
1330
+ "<|special_1447|>",
1331
+ "<|special_1448|>",
1332
+ "<|special_1449|>",
1333
+ "<|special_1450|>",
1334
+ "<|special_1451|>",
1335
+ "<|special_1452|>",
1336
+ "<|special_1453|>",
1337
+ "<|special_1454|>",
1338
+ "<|special_1455|>",
1339
+ "<|special_1456|>",
1340
+ "<|special_1457|>",
1341
+ "<|special_1458|>",
1342
+ "<|special_1459|>",
1343
+ "<|special_1460|>",
1344
+ "<|special_1461|>",
1345
+ "<|special_1462|>",
1346
+ "<|special_1463|>",
1347
+ "<|special_1464|>",
1348
+ "<|special_1465|>",
1349
+ "<|special_1466|>",
1350
+ "<|special_1467|>",
1351
+ "<|special_1468|>",
1352
+ "<|special_1469|>",
1353
+ "<|special_1470|>",
1354
+ "<|special_1471|>",
1355
+ "<|special_1472|>",
1356
+ "<|special_1473|>",
1357
+ "<|special_1474|>",
1358
+ "<|special_1475|>",
1359
+ "<|special_1476|>",
1360
+ "<|special_1477|>",
1361
+ "<|special_1478|>",
1362
+ "<|special_1479|>",
1363
+ "<|special_1480|>",
1364
+ "<|special_1481|>",
1365
+ "<|special_1482|>",
1366
+ "<|special_1483|>",
1367
+ "<|special_1484|>",
1368
+ "<|special_1485|>",
1369
+ "<|special_1486|>",
1370
+ "<|special_1487|>",
1371
+ "<|special_1488|>",
1372
+ "<|special_1489|>",
1373
+ "<|special_1490|>",
1374
+ "<|special_1491|>",
1375
+ "<|special_1492|>",
1376
+ "<|special_1493|>",
1377
+ "<|special_1494|>",
1378
+ "<|special_1495|>",
1379
+ "<|special_1496|>",
1380
+ "<|special_1497|>",
1381
+ "<|special_1498|>",
1382
+ "<|special_1499|>",
1383
+ "<|special_1500|>",
1384
+ "<|special_1501|>",
1385
+ "<|special_1502|>",
1386
+ "<|special_1503|>",
1387
+ "<|special_1504|>",
1388
+ "<|special_1505|>",
1389
+ "<|special_1506|>",
1390
+ "<|special_1507|>",
1391
+ "<|special_1508|>",
1392
+ "<|special_1509|>",
1393
+ "<|special_1510|>",
1394
+ "<|special_1511|>",
1395
+ "<|special_1512|>",
1396
+ "<|special_1513|>",
1397
+ "<|special_1514|>",
1398
+ "<|special_1515|>",
1399
+ "<|special_1516|>",
1400
+ "<|special_1517|>",
1401
+ "<|special_1518|>",
1402
+ "<|special_1519|>",
1403
+ "<|special_1520|>",
1404
+ "<|special_1521|>",
1405
+ "<|special_1522|>",
1406
+ "<|special_1523|>",
1407
+ "<|special_1524|>",
1408
+ "<|special_1525|>",
1409
+ "<|special_1526|>",
1410
+ "<|special_1527|>",
1411
+ "<|special_1528|>",
1412
+ "<|special_1529|>",
1413
+ "<|special_1530|>",
1414
+ "<|special_1531|>",
1415
+ "<|special_1532|>",
1416
+ "<|special_1533|>",
1417
+ "<|special_1534|>",
1418
+ "<|special_1535|>",
1419
+ "<|special_1536|>",
1420
+ "<|special_1537|>",
1421
+ "<|special_1538|>",
1422
+ "<|special_1539|>",
1423
+ "<|special_1540|>",
1424
+ "<|special_1541|>",
1425
+ "<|special_1542|>",
1426
+ "<|special_1543|>",
1427
+ "<|special_1544|>",
1428
+ "<|special_1545|>",
1429
+ "<|special_1546|>",
1430
+ "<|special_1547|>",
1431
+ "<|special_1548|>",
1432
+ "<|special_1549|>",
1433
+ "<|special_1550|>",
1434
+ "<|special_1551|>",
1435
+ "<|special_1552|>",
1436
+ "<|special_1553|>",
1437
+ "<|special_1554|>",
1438
+ "<|special_1555|>",
1439
+ "<|special_1556|>",
1440
+ "<|special_1557|>",
1441
+ "<|special_1558|>",
1442
+ "<|special_1559|>",
1443
+ "<|special_1560|>",
1444
+ "<|special_1561|>",
1445
+ "<|special_1562|>",
1446
+ "<|special_1563|>",
1447
+ "<|special_1564|>",
1448
+ "<|special_1565|>",
1449
+ "<|special_1566|>",
1450
+ "<|special_1567|>",
1451
+ "<|special_1568|>",
1452
+ "<|special_1569|>",
1453
+ "<|special_1570|>",
1454
+ "<|special_1571|>",
1455
+ "<|special_1572|>",
1456
+ "<|special_1573|>",
1457
+ "<|special_1574|>",
1458
+ "<|special_1575|>",
1459
+ "<|special_1576|>",
1460
+ "<|special_1577|>",
1461
+ "<|special_1578|>",
1462
+ "<|special_1579|>",
1463
+ "<|special_1580|>",
1464
+ "<|special_1581|>",
1465
+ "<|special_1582|>",
1466
+ "<|special_1583|>",
1467
+ "<|special_1584|>",
1468
+ "<|special_1585|>",
1469
+ "<|special_1586|>",
1470
+ "<|special_1587|>",
1471
+ "<|special_1588|>",
1472
+ "<|special_1589|>",
1473
+ "<|special_1590|>",
1474
+ "<|special_1591|>",
1475
+ "<|special_1592|>",
1476
+ "<|special_1593|>",
1477
+ "<|special_1594|>",
1478
+ "<|special_1595|>",
1479
+ "<|special_1596|>",
1480
+ "<|special_1597|>",
1481
+ "<|special_1598|>",
1482
+ "<|special_1599|>",
1483
+ "<|special_1600|>",
1484
+ "<|special_1601|>",
1485
+ "<|special_1602|>",
1486
+ "<|special_1603|>",
1487
+ "<|special_1604|>",
1488
+ "<|special_1605|>",
1489
+ "<|special_1606|>",
1490
+ "<|special_1607|>",
1491
+ "<|special_1608|>",
1492
+ "<|special_1609|>",
1493
+ "<|special_1610|>",
1494
+ "<|special_1611|>",
1495
+ "<|special_1612|>",
1496
+ "<|special_1613|>",
1497
+ "<|special_1614|>",
1498
+ "<|special_1615|>",
1499
+ "<|special_1616|>",
1500
+ "<|special_1617|>",
1501
+ "<|special_1618|>",
1502
+ "<|special_1619|>",
1503
+ "<|special_1620|>",
1504
+ "<|special_1621|>",
1505
+ "<|special_1622|>",
1506
+ "<|special_1623|>",
1507
+ "<|special_1624|>",
1508
+ "<|special_1625|>",
1509
+ "<|special_1626|>",
1510
+ "<|special_1627|>",
1511
+ "<|special_1628|>",
1512
+ "<|special_1629|>",
1513
+ "<|special_1630|>",
1514
+ "<|special_1631|>",
1515
+ "<|special_1632|>",
1516
+ "<|special_1633|>",
1517
+ "<|special_1634|>",
1518
+ "<|special_1635|>",
1519
+ "<|special_1636|>",
1520
+ "<|special_1637|>",
1521
+ "<|special_1638|>",
1522
+ "<|special_1639|>",
1523
+ "<|special_1640|>",
1524
+ "<|special_1641|>",
1525
+ "<|special_1642|>",
1526
+ "<|special_1643|>",
1527
+ "<|special_1644|>",
1528
+ "<|special_1645|>",
1529
+ "<|special_1646|>",
1530
+ "<|special_1647|>",
1531
+ "<|special_1648|>",
1532
+ "<|special_1649|>",
1533
+ "<|special_1650|>",
1534
+ "<|special_1651|>",
1535
+ "<|special_1652|>",
1536
+ "<|special_1653|>",
1537
+ "<|special_1654|>",
1538
+ "<|special_1655|>",
1539
+ "<|special_1656|>",
1540
+ "<|special_1657|>",
1541
+ "<|special_1658|>",
1542
+ "<|special_1659|>",
1543
+ "<|special_1660|>",
1544
+ "<|special_1661|>",
1545
+ "<|special_1662|>",
1546
+ "<|special_1663|>",
1547
+ "<|special_1664|>",
1548
+ "<|special_1665|>",
1549
+ "<|special_1666|>",
1550
+ "<|special_1667|>",
1551
+ "<|special_1668|>",
1552
+ "<|special_1669|>",
1553
+ "<|special_1670|>",
1554
+ "<|special_1671|>",
1555
+ "<|special_1672|>",
1556
+ "<|special_1673|>",
1557
+ "<|special_1674|>",
1558
+ "<|special_1675|>",
1559
+ "<|special_1676|>",
1560
+ "<|special_1677|>",
1561
+ "<|special_1678|>",
1562
+ "<|special_1679|>",
1563
+ "<|special_1680|>",
1564
+ "<|special_1681|>",
1565
+ "<|special_1682|>",
1566
+ "<|special_1683|>",
1567
+ "<|special_1684|>",
1568
+ "<|special_1685|>",
1569
+ "<|special_1686|>",
1570
+ "<|special_1687|>",
1571
+ "<|special_1688|>",
1572
+ "<|special_1689|>",
1573
+ "<|special_1690|>",
1574
+ "<|special_1691|>",
1575
+ "<|special_1692|>",
1576
+ "<|special_1693|>",
1577
+ "<|special_1694|>",
1578
+ "<|special_1695|>",
1579
+ "<|special_1696|>",
1580
+ "<|special_1697|>",
1581
+ "<|special_1698|>",
1582
+ "<|special_1699|>",
1583
+ "<|special_1700|>",
1584
+ "<|special_1701|>",
1585
+ "<|special_1702|>",
1586
+ "<|special_1703|>",
1587
+ "<|special_1704|>",
1588
+ "<|special_1705|>",
1589
+ "<|special_1706|>",
1590
+ "<|special_1707|>",
1591
+ "<|special_1708|>",
1592
+ "<|special_1709|>",
1593
+ "<|special_1710|>",
1594
+ "<|special_1711|>",
1595
+ "<|special_1712|>",
1596
+ "<|special_1713|>",
1597
+ "<|special_1714|>",
1598
+ "<|special_1715|>",
1599
+ "<|special_1716|>",
1600
+ "<|special_1717|>",
1601
+ "<|special_1718|>",
1602
+ "<|special_1719|>",
1603
+ "<|special_1720|>",
1604
+ "<|special_1721|>",
1605
+ "<|special_1722|>",
1606
+ "<|special_1723|>",
1607
+ "<|special_1724|>",
1608
+ "<|special_1725|>",
1609
+ "<|special_1726|>",
1610
+ "<|special_1727|>",
1611
+ "<|special_1728|>",
1612
+ "<|special_1729|>",
1613
+ "<|special_1730|>",
1614
+ "<|special_1731|>",
1615
+ "<|special_1732|>",
1616
+ "<|special_1733|>",
1617
+ "<|special_1734|>",
1618
+ "<|special_1735|>",
1619
+ "<|special_1736|>",
1620
+ "<|special_1737|>",
1621
+ "<|special_1738|>",
1622
+ "<|special_1739|>",
1623
+ "<|special_1740|>",
1624
+ "<|special_1741|>",
1625
+ "<|special_1742|>",
1626
+ "<|special_1743|>",
1627
+ "<|special_1744|>",
1628
+ "<|special_1745|>",
1629
+ "<|special_1746|>",
1630
+ "<|special_1747|>",
1631
+ "<|special_1748|>",
1632
+ "<|special_1749|>",
1633
+ "<|special_1750|>",
1634
+ "<|special_1751|>",
1635
+ "<|special_1752|>",
1636
+ "<|special_1753|>",
1637
+ "<|special_1754|>",
1638
+ "<|special_1755|>",
1639
+ "<|special_1756|>",
1640
+ "<|special_1757|>",
1641
+ "<|special_1758|>",
1642
+ "<|special_1759|>",
1643
+ "<|special_1760|>",
1644
+ "<|special_1761|>",
1645
+ "<|special_1762|>",
1646
+ "<|special_1763|>",
1647
+ "<|special_1764|>",
1648
+ "<|special_1765|>",
1649
+ "<|special_1766|>",
1650
+ "<|special_1767|>",
1651
+ "<|special_1768|>",
1652
+ "<|special_1769|>",
1653
+ "<|special_1770|>",
1654
+ "<|special_1771|>",
1655
+ "<|special_1772|>",
1656
+ "<|special_1773|>",
1657
+ "<|special_1774|>",
1658
+ "<|special_1775|>",
1659
+ "<|special_1776|>",
1660
+ "<|special_1777|>",
1661
+ "<|special_1778|>",
1662
+ "<|special_1779|>",
1663
+ "<|special_1780|>",
1664
+ "<|special_1781|>",
1665
+ "<|special_1782|>",
1666
+ "<|special_1783|>",
1667
+ "<|special_1784|>",
1668
+ "<|special_1785|>",
1669
+ "<|special_1786|>",
1670
+ "<|special_1787|>",
1671
+ "<|special_1788|>",
1672
+ "<|special_1789|>",
1673
+ "<|special_1790|>",
1674
+ "<|special_1791|>",
1675
+ "<|special_1792|>",
1676
+ "<|special_1793|>",
1677
+ "<|special_1794|>",
1678
+ "<|special_1795|>",
1679
+ "<|special_1796|>",
1680
+ "<|special_1797|>",
1681
+ "<|special_1798|>",
1682
+ "<|special_1799|>",
1683
+ "<|special_1800|>",
1684
+ "<|special_1801|>",
1685
+ "<|special_1802|>",
1686
+ "<|special_1803|>",
1687
+ "<|special_1804|>",
1688
+ "<|special_1805|>",
1689
+ "<|special_1806|>",
1690
+ "<|special_1807|>",
1691
+ "<|special_1808|>",
1692
+ "<|special_1809|>",
1693
+ "<|special_1810|>",
1694
+ "<|special_1811|>",
1695
+ "<|special_1812|>",
1696
+ "<|special_1813|>",
1697
+ "<|special_1814|>",
1698
+ "<|special_1815|>",
1699
+ "<|special_1816|>",
1700
+ "<|special_1817|>",
1701
+ "<|special_1818|>",
1702
+ "<|special_1819|>",
1703
+ "<|special_1820|>",
1704
+ "<|special_1821|>",
1705
+ "<|special_1822|>",
1706
+ "<|special_1823|>",
1707
+ "<|special_1824|>",
1708
+ "<|special_1825|>",
1709
+ "<|special_1826|>",
1710
+ "<|special_1827|>",
1711
+ "<|special_1828|>",
1712
+ "<|special_1829|>",
1713
+ "<|special_1830|>",
1714
+ "<|special_1831|>",
1715
+ "<|special_1832|>",
1716
+ "<|special_1833|>",
1717
+ "<|special_1834|>",
1718
+ "<|special_1835|>",
1719
+ "<|special_1836|>",
1720
+ "<|special_1837|>",
1721
+ "<|special_1838|>",
1722
+ "<|special_1839|>",
1723
+ "<|special_1840|>",
1724
+ "<|special_1841|>",
1725
+ "<|special_1842|>",
1726
+ "<|special_1843|>",
1727
+ "<|special_1844|>",
1728
+ "<|special_1845|>",
1729
+ "<|special_1846|>",
1730
+ "<|special_1847|>",
1731
+ "<|special_1848|>",
1732
+ "<|special_1849|>",
1733
+ "<|special_1850|>",
1734
+ "<|special_1851|>",
1735
+ "<|special_1852|>",
1736
+ "<|special_1853|>",
1737
+ "<|special_1854|>",
1738
+ "<|special_1855|>",
1739
+ "<|special_1856|>",
1740
+ "<|special_1857|>",
1741
+ "<|special_1858|>",
1742
+ "<|special_1859|>",
1743
+ "<|special_1860|>",
1744
+ "<|special_1861|>",
1745
+ "<|special_1862|>",
1746
+ "<|special_1863|>",
1747
+ "<|special_1864|>",
1748
+ "<|special_1865|>",
1749
+ "<|special_1866|>",
1750
+ "<|special_1867|>",
1751
+ "<|special_1868|>",
1752
+ "<|special_1869|>",
1753
+ "<|special_1870|>",
1754
+ "<|special_1871|>",
1755
+ "<|special_1872|>",
1756
+ "<|special_1873|>",
1757
+ "<|special_1874|>",
1758
+ "<|special_1875|>",
1759
+ "<|special_1876|>",
1760
+ "<|special_1877|>",
1761
+ "<|special_1878|>",
1762
+ "<|special_1879|>",
1763
+ "<|special_1880|>",
1764
+ "<|special_1881|>",
1765
+ "<|special_1882|>",
1766
+ "<|special_1883|>",
1767
+ "<|special_1884|>",
1768
+ "<|special_1885|>",
1769
+ "<|special_1886|>",
1770
+ "<|special_1887|>",
1771
+ "<|special_1888|>",
1772
+ "<|special_1889|>",
1773
+ "<|special_1890|>",
1774
+ "<|special_1891|>",
1775
+ "<|special_1892|>",
1776
+ "<|special_1893|>",
1777
+ "<|special_1894|>",
1778
+ "<|special_1895|>",
1779
+ "<|special_1896|>",
1780
+ "<|special_1897|>",
1781
+ "<|special_1898|>",
1782
+ "<|special_1899|>",
1783
+ "<|special_1900|>",
1784
+ "<|special_1901|>",
1785
+ "<|special_1902|>",
1786
+ "<|special_1903|>",
1787
+ "<|special_1904|>",
1788
+ "<|special_1905|>",
1789
+ "<|special_1906|>",
1790
+ "<|special_1907|>",
1791
+ "<|special_1908|>",
1792
+ "<|special_1909|>",
1793
+ "<|special_1910|>",
1794
+ "<|special_1911|>",
1795
+ "<|special_1912|>",
1796
+ "<|special_1913|>",
1797
+ "<|special_1914|>",
1798
+ "<|special_1915|>",
1799
+ "<|special_1916|>",
1800
+ "<|special_1917|>",
1801
+ "<|special_1918|>",
1802
+ "<|special_1919|>",
1803
+ "<|special_1920|>",
1804
+ "<|special_1921|>",
1805
+ "<|special_1922|>",
1806
+ "<|special_1923|>",
1807
+ "<|special_1924|>",
1808
+ "<|special_1925|>",
1809
+ "<|special_1926|>",
1810
+ "<|special_1927|>",
1811
+ "<|special_1928|>",
1812
+ "<|special_1929|>",
1813
+ "<|special_1930|>",
1814
+ "<|special_1931|>",
1815
+ "<|special_1932|>",
1816
+ "<|special_1933|>",
1817
+ "<|special_1934|>",
1818
+ "<|special_1935|>",
1819
+ "<|special_1936|>",
1820
+ "<|special_1937|>",
1821
+ "<|special_1938|>",
1822
+ "<|special_1939|>",
1823
+ "<|special_1940|>",
1824
+ "<|special_1941|>",
1825
+ "<|special_1942|>",
1826
+ "<|special_1943|>",
1827
+ "<|special_1944|>",
1828
+ "<|special_1945|>",
1829
+ "<|special_1946|>",
1830
+ "<|special_1947|>",
1831
+ "<|special_1948|>",
1832
+ "<|special_1949|>",
1833
+ "<|special_1950|>",
1834
+ "<|special_1951|>",
1835
+ "<|special_1952|>",
1836
+ "<|special_1953|>",
1837
+ "<|special_1954|>",
1838
+ "<|special_1955|>",
1839
+ "<|special_1956|>",
1840
+ "<|special_1957|>",
1841
+ "<|special_1958|>",
1842
+ "<|special_1959|>",
1843
+ "<|special_1960|>",
1844
+ "<|special_1961|>",
1845
+ "<|special_1962|>",
1846
+ "<|special_1963|>",
1847
+ "<|special_1964|>",
1848
+ "<|special_1965|>",
1849
+ "<|special_1966|>",
1850
+ "<|special_1967|>",
1851
+ "<|special_1968|>",
1852
+ "<|special_1969|>",
1853
+ "<|special_1970|>",
1854
+ "<|special_1971|>",
1855
+ "<|special_1972|>",
1856
+ "<|special_1973|>",
1857
+ "<|special_1974|>",
1858
+ "<|special_1975|>",
1859
+ "<|special_1976|>",
1860
+ "<|special_1977|>",
1861
+ "<|special_1978|>",
1862
+ "<|special_1979|>",
1863
+ "<|special_1980|>",
1864
+ "<|special_1981|>",
1865
+ "<|special_1982|>",
1866
+ "<|special_1983|>",
1867
+ "<|special_1984|>",
1868
+ "<|special_1985|>",
1869
+ "<|special_1986|>",
1870
+ "<|special_1987|>",
1871
+ "<|special_1988|>",
1872
+ "<|special_1989|>",
1873
+ "<|special_1990|>",
1874
+ "<|special_1991|>",
1875
+ "<|special_1992|>",
1876
+ "<|special_1993|>",
1877
+ "<|special_1994|>",
1878
+ "<|special_1995|>",
1879
+ "<|special_1996|>",
1880
+ "<|special_1997|>",
1881
+ "<|special_1998|>",
1882
+ "<|special_1999|>",
1883
+ "<|special_2000|>",
1884
+ "<|special_2001|>",
1885
+ "<|special_2002|>",
1886
+ "<|special_2003|>",
1887
+ "<|special_2004|>",
1888
+ "<|special_2005|>",
1889
+ "<|special_2006|>",
1890
+ "<|special_2007|>",
1891
+ "<|special_2008|>",
1892
+ "<|special_2009|>",
1893
+ "<|special_2010|>",
1894
+ "<|special_2011|>",
1895
+ "<|special_2012|>",
1896
+ "<|special_2013|>",
1897
+ "<|special_2014|>",
1898
+ "<|special_2015|>",
1899
+ "<|special_2016|>",
1900
+ "<|special_2017|>",
1901
+ "<|special_2018|>",
1902
+ "<|special_2019|>",
1903
+ "<|special_2020|>",
1904
+ "<|special_2021|>",
1905
+ "<|special_2022|>",
1906
+ "<|special_2023|>",
1907
+ "<|special_2024|>",
1908
+ "<|special_2025|>",
1909
+ "<|special_2026|>",
1910
+ "<|special_2027|>",
1911
+ "<|special_2028|>",
1912
+ "<|special_2029|>",
1913
+ "<|special_2030|>",
1914
+ "<|special_2031|>",
1915
+ "<|special_2032|>",
1916
+ "<|special_2033|>",
1917
+ "<|special_2034|>",
1918
+ "<|special_2035|>",
1919
+ "<|special_2036|>",
1920
+ "<|special_2037|>",
1921
+ "<|special_2038|>",
1922
+ "<|special_2039|>",
1923
+ "<|special_2040|>",
1924
+ "<|special_2041|>",
1925
+ "<|special_2042|>",
1926
+ "<|special_2043|>",
1927
+ "<|special_2044|>",
1928
+ "<|special_2045|>",
1929
+ "<|special_2046|>",
1930
+ "<|special_2047|>",
1931
+ "<|special_2048|>",
1932
+ "<|special_2049|>",
1933
+ "<|special_2050|>",
1934
+ "<|special_2051|>",
1935
+ "<|special_2052|>",
1936
+ "<|special_2053|>",
1937
+ "<|special_2054|>",
1938
+ "<|special_2055|>",
1939
+ "<|special_2056|>",
1940
+ "<|special_2057|>",
1941
+ "<|special_2058|>",
1942
+ "<|special_2059|>",
1943
+ "<|special_2060|>",
1944
+ "<|special_2061|>",
1945
+ "<|special_2062|>",
1946
+ "<|special_2063|>",
1947
+ "<|special_2064|>",
1948
+ "<|special_2065|>",
1949
+ "<|special_2066|>",
1950
+ "<|special_2067|>",
1951
+ "<|special_2068|>",
1952
+ "<|special_2069|>",
1953
+ "<|special_2070|>",
1954
+ "<|special_2071|>",
1955
+ "<|special_2072|>",
1956
+ "<|special_2073|>",
1957
+ "<|special_2074|>",
1958
+ "<|special_2075|>",
1959
+ "<|special_2076|>",
1960
+ "<|special_2077|>",
1961
+ "<|special_2078|>",
1962
+ "<|special_2079|>",
1963
+ "<|special_2080|>",
1964
+ "<|special_2081|>",
1965
+ "<|special_2082|>",
1966
+ "<|special_2083|>",
1967
+ "<|special_2084|>",
1968
+ "<|special_2085|>",
1969
+ "<|special_2086|>",
1970
+ "<|special_2087|>",
1971
+ "<|special_2088|>",
1972
+ "<|special_2089|>",
1973
+ "<|special_2090|>",
1974
+ "<|special_2091|>",
1975
+ "<|special_2092|>",
1976
+ "<|special_2093|>",
1977
+ "<|special_2094|>",
1978
+ "<|special_2095|>",
1979
+ "<|special_2096|>",
1980
+ "<|special_2097|>",
1981
+ "<|special_2098|>",
1982
+ "<|special_2099|>",
1983
+ "<|special_2100|>",
1984
+ "<|special_2101|>",
1985
+ "<|special_2102|>",
1986
+ "<|special_2103|>",
1987
+ "<|special_2104|>",
1988
+ "<|special_2105|>",
1989
+ "<|special_2106|>",
1990
+ "<|special_2107|>",
1991
+ "<|special_2108|>",
1992
+ "<|special_2109|>",
1993
+ "<|special_2110|>",
1994
+ "<|special_2111|>",
1995
+ "<|special_2112|>",
1996
+ "<|special_2113|>",
1997
+ "<|special_2114|>",
1998
+ "<|special_2115|>",
1999
+ "<|special_2116|>",
2000
+ "<|special_2117|>",
2001
+ "<|special_2118|>",
2002
+ "<|special_2119|>",
2003
+ "<|special_2120|>",
2004
+ "<|special_2121|>",
2005
+ "<|special_2122|>",
2006
+ "<|special_2123|>",
2007
+ "<|special_2124|>",
2008
+ "<|special_2125|>",
2009
+ "<|special_2126|>",
2010
+ "<|special_2127|>",
2011
+ "<|special_2128|>",
2012
+ "<|special_2129|>",
2013
+ "<|special_2130|>",
2014
+ "<|special_2131|>",
2015
+ "<|special_2132|>",
2016
+ "<|special_2133|>",
2017
+ "<|special_2134|>",
2018
+ "<|special_2135|>",
2019
+ "<|special_2136|>",
2020
+ "<|special_2137|>",
2021
+ "<|special_2138|>",
2022
+ "<|special_2139|>",
2023
+ "<|special_2140|>",
2024
+ "<|special_2141|>",
2025
+ "<|special_2142|>",
2026
+ "<|special_2143|>",
2027
+ "<|special_2144|>",
2028
+ "<|special_2145|>",
2029
+ "<|special_2146|>",
2030
+ "<|special_2147|>",
2031
+ "<|special_2148|>",
2032
+ "<|special_2149|>",
2033
+ "<|special_2150|>",
2034
+ "<|special_2151|>",
2035
+ "<|special_2152|>",
2036
+ "<|special_2153|>",
2037
+ "<|special_2154|>",
2038
+ "<|special_2155|>",
2039
+ "<|special_2156|>",
2040
+ "<|special_2157|>",
2041
+ "<|special_2158|>",
2042
+ "<|special_2159|>",
2043
+ "<|special_2160|>",
2044
+ "<|special_2161|>",
2045
+ "<|special_2162|>",
2046
+ "<|special_2163|>",
2047
+ "<|special_2164|>",
2048
+ "<|special_2165|>",
2049
+ "<|special_2166|>",
2050
+ "<|special_2167|>",
2051
+ "<|special_2168|>",
2052
+ "<|special_2169|>",
2053
+ "<|special_2170|>",
2054
+ "<|special_2171|>",
2055
+ "<|special_2172|>",
2056
+ "<|special_2173|>",
2057
+ "<|special_2174|>",
2058
+ "<|special_2175|>",
2059
+ "<|special_2176|>",
2060
+ "<|special_2177|>",
2061
+ "<|special_2178|>",
2062
+ "<|special_2179|>",
2063
+ "<|special_2180|>",
2064
+ "<|special_2181|>",
2065
+ "<|special_2182|>",
2066
+ "<|special_2183|>",
2067
+ "<|special_2184|>",
2068
+ "<|special_2185|>",
2069
+ "<|special_2186|>",
2070
+ "<|special_2187|>",
2071
+ "<|special_2188|>",
2072
+ "<|special_2189|>",
2073
+ "<|special_2190|>",
2074
+ "<|special_2191|>",
2075
+ "<|special_2192|>",
2076
+ "<|special_2193|>",
2077
+ "<|special_2194|>",
2078
+ "<|special_2195|>",
2079
+ "<|special_2196|>",
2080
+ "<|special_2197|>",
2081
+ "<|special_2198|>",
2082
+ "<|special_2199|>",
2083
+ "<|special_2200|>",
2084
+ "<|special_2201|>",
2085
+ "<|special_2202|>",
2086
+ "<|special_2203|>",
2087
+ "<|special_2204|>",
2088
+ "<|special_2205|>",
2089
+ "<|special_2206|>",
2090
+ "<|special_2207|>",
2091
+ "<|special_2208|>",
2092
+ "<|special_2209|>",
2093
+ "<|special_2210|>",
2094
+ "<|special_2211|>",
2095
+ "<|special_2212|>",
2096
+ "<|special_2213|>",
2097
+ "<|special_2214|>",
2098
+ "<|special_2215|>",
2099
+ "<|special_2216|>",
2100
+ "<|special_2217|>",
2101
+ "<|special_2218|>",
2102
+ "<|special_2219|>",
2103
+ "<|special_2220|>",
2104
+ "<|special_2221|>",
2105
+ "<|special_2222|>",
2106
+ "<|special_2223|>",
2107
+ "<|special_2224|>",
2108
+ "<|special_2225|>",
2109
+ "<|special_2226|>",
2110
+ "<|special_2227|>",
2111
+ "<|special_2228|>",
2112
+ "<|special_2229|>",
2113
+ "<|special_2230|>",
2114
+ "<|special_2231|>",
2115
+ "<|special_2232|>",
2116
+ "<|special_2233|>",
2117
+ "<|special_2234|>",
2118
+ "<|special_2235|>",
2119
+ "<|special_2236|>",
2120
+ "<|special_2237|>",
2121
+ "<|special_2238|>",
2122
+ "<|special_2239|>",
2123
+ "<|special_2240|>",
2124
+ "<|special_2241|>",
2125
+ "<|special_2242|>",
2126
+ "<|special_2243|>",
2127
+ "<|special_2244|>",
2128
+ "<|special_2245|>",
2129
+ "<|special_2246|>",
2130
+ "<|special_2247|>",
2131
+ "<|special_2248|>",
2132
+ "<|special_2249|>",
2133
+ "<|special_2250|>",
2134
+ "<|special_2251|>",
2135
+ "<|special_2252|>",
2136
+ "<|special_2253|>",
2137
+ "<|special_2254|>",
2138
+ "<|special_2255|>",
2139
+ "<|special_2256|>",
2140
+ "<|special_2257|>",
2141
+ "<|special_2258|>",
2142
+ "<|special_2259|>",
2143
+ "<|special_2260|>",
2144
+ "<|special_2261|>",
2145
+ "<|special_2262|>",
2146
+ "<|special_2263|>",
2147
+ "<|special_2264|>",
2148
+ "<|special_2265|>",
2149
+ "<|special_2266|>",
2150
+ "<|special_2267|>",
2151
+ "<|special_2268|>",
2152
+ "<|special_2269|>",
2153
+ "<|special_2270|>",
2154
+ "<|special_2271|>",
2155
+ "<|special_2272|>",
2156
+ "<|special_2273|>",
2157
+ "<|special_2274|>",
2158
+ "<|special_2275|>",
2159
+ "<|special_2276|>",
2160
+ "<|special_2277|>",
2161
+ "<|special_2278|>",
2162
+ "<|special_2279|>",
2163
+ "<|special_2280|>",
2164
+ "<|special_2281|>",
2165
+ "<|special_2282|>",
2166
+ "<|special_2283|>",
2167
+ "<|special_2284|>",
2168
+ "<|special_2285|>",
2169
+ "<|special_2286|>",
2170
+ "<|special_2287|>",
2171
+ "<|special_2288|>",
2172
+ "<|special_2289|>",
2173
+ "<|special_2290|>",
2174
+ "<|special_2291|>",
2175
+ "<|special_2292|>",
2176
+ "<|special_2293|>",
2177
+ "<|special_2294|>",
2178
+ "<|special_2295|>",
2179
+ "<|special_2296|>",
2180
+ "<|special_2297|>",
2181
+ "<|special_2298|>",
2182
+ "<|special_2299|>",
2183
+ "<|special_2300|>",
2184
+ "<|special_2301|>",
2185
+ "<|special_2302|>",
2186
+ "<|special_2303|>",
2187
+ "<|special_2304|>",
2188
+ "<|special_2305|>",
2189
+ "<|special_2306|>",
2190
+ "<|special_2307|>",
2191
+ "<|special_2308|>",
2192
+ "<|special_2309|>",
2193
+ "<|special_2310|>",
2194
+ "<|special_2311|>",
2195
+ "<|special_2312|>",
2196
+ "<|special_2313|>",
2197
+ "<|special_2314|>",
2198
+ "<|special_2315|>",
2199
+ "<|special_2316|>",
2200
+ "<|special_2317|>",
2201
+ "<|special_2318|>",
2202
+ "<|special_2319|>",
2203
+ "<|special_2320|>",
2204
+ "<|special_2321|>",
2205
+ "<|special_2322|>",
2206
+ "<|special_2323|>",
2207
+ "<|special_2324|>",
2208
+ "<|special_2325|>",
2209
+ "<|special_2326|>",
2210
+ "<|special_2327|>",
2211
+ "<|special_2328|>",
2212
+ "<|special_2329|>",
2213
+ "<|special_2330|>",
2214
+ "<|special_2331|>",
2215
+ "<|special_2332|>",
2216
+ "<|special_2333|>",
2217
+ "<|special_2334|>",
2218
+ "<|special_2335|>",
2219
+ "<|special_2336|>",
2220
+ "<|special_2337|>",
2221
+ "<|special_2338|>",
2222
+ "<|special_2339|>",
2223
+ "<|special_2340|>",
2224
+ "<|special_2341|>",
2225
+ "<|special_2342|>",
2226
+ "<|special_2343|>",
2227
+ "<|special_2344|>",
2228
+ "<|special_2345|>",
2229
+ "<|special_2346|>",
2230
+ "<|special_2347|>",
2231
+ "<|special_2348|>",
2232
+ "<|special_2349|>",
2233
+ "<|special_2350|>",
2234
+ "<|special_2351|>",
2235
+ "<|special_2352|>",
2236
+ "<|special_2353|>",
2237
+ "<|special_2354|>",
2238
+ "<|special_2355|>",
2239
+ "<|special_2356|>",
2240
+ "<|special_2357|>",
2241
+ "<|special_2358|>",
2242
+ "<|special_2359|>",
2243
+ "<|special_2360|>",
2244
+ "<|special_2361|>",
2245
+ "<|special_2362|>",
2246
+ "<|special_2363|>",
2247
+ "<|special_2364|>",
2248
+ "<|special_2365|>",
2249
+ "<|special_2366|>",
2250
+ "<|special_2367|>",
2251
+ "<|special_2368|>",
2252
+ "<|special_2369|>",
2253
+ "<|special_2370|>",
2254
+ "<|special_2371|>",
2255
+ "<|special_2372|>",
2256
+ "<|special_2373|>",
2257
+ "<|special_2374|>",
2258
+ "<|special_2375|>",
2259
+ "<|special_2376|>",
2260
+ "<|special_2377|>",
2261
+ "<|special_2378|>",
2262
+ "<|special_2379|>",
2263
+ "<|special_2380|>",
2264
+ "<|special_2381|>",
2265
+ "<|special_2382|>",
2266
+ "<|special_2383|>",
2267
+ "<|special_2384|>",
2268
+ "<|special_2385|>",
2269
+ "<|special_2386|>",
2270
+ "<|special_2387|>",
2271
+ "<|special_2388|>",
2272
+ "<|special_2389|>",
2273
+ "<|special_2390|>",
2274
+ "<|special_2391|>",
2275
+ "<|special_2392|>",
2276
+ "<|special_2393|>",
2277
+ "<|special_2394|>",
2278
+ "<|special_2395|>",
2279
+ "<|special_2396|>",
2280
+ "<|special_2397|>",
2281
+ "<|special_2398|>",
2282
+ "<|special_2399|>",
2283
+ "<|special_2400|>",
2284
+ "<|special_2401|>",
2285
+ "<|special_2402|>",
2286
+ "<|special_2403|>",
2287
+ "<|special_2404|>",
2288
+ "<|special_2405|>",
2289
+ "<|special_2406|>",
2290
+ "<|special_2407|>",
2291
+ "<|special_2408|>",
2292
+ "<|special_2409|>",
2293
+ "<|special_2410|>",
2294
+ "<|special_2411|>",
2295
+ "<|special_2412|>",
2296
+ "<|special_2413|>",
2297
+ "<|special_2414|>",
2298
+ "<|special_2415|>",
2299
+ "<|special_2416|>",
2300
+ "<|special_2417|>",
2301
+ "<|special_2418|>",
2302
+ "<|special_2419|>",
2303
+ "<|special_2420|>",
2304
+ "<|special_2421|>",
2305
+ "<|special_2422|>",
2306
+ "<|special_2423|>",
2307
+ "<|special_2424|>",
2308
+ "<|special_2425|>",
2309
+ "<|special_2426|>",
2310
+ "<|special_2427|>",
2311
+ "<|special_2428|>",
2312
+ "<|special_2429|>",
2313
+ "<|special_2430|>",
2314
+ "<|special_2431|>",
2315
+ "<|special_2432|>",
2316
+ "<|special_2433|>",
2317
+ "<|special_2434|>",
2318
+ "<|special_2435|>",
2319
+ "<|special_2436|>",
2320
+ "<|special_2437|>",
2321
+ "<|special_2438|>",
2322
+ "<|special_2439|>",
2323
+ "<|special_2440|>",
2324
+ "<|special_2441|>",
2325
+ "<|special_2442|>",
2326
+ "<|special_2443|>",
2327
+ "<|special_2444|>",
2328
+ "<|special_2445|>",
2329
+ "<|special_2446|>",
2330
+ "<|special_2447|>",
2331
+ "<|special_2448|>",
2332
+ "<|special_2449|>",
2333
+ "<|special_2450|>",
2334
+ "<|special_2451|>",
2335
+ "<|special_2452|>",
2336
+ "<|special_2453|>",
2337
+ "<|special_2454|>",
2338
+ "<|special_2455|>",
2339
+ "<|special_2456|>",
2340
+ "<|special_2457|>",
2341
+ "<|special_2458|>",
2342
+ "<|special_2459|>",
2343
+ "<|special_2460|>",
2344
+ "<|special_2461|>",
2345
+ "<|special_2462|>",
2346
+ "<|special_2463|>",
2347
+ "<|special_2464|>",
2348
+ "<|special_2465|>",
2349
+ "<|special_2466|>",
2350
+ "<|special_2467|>",
2351
+ "<|special_2468|>",
2352
+ "<|special_2469|>",
2353
+ "<|special_2470|>",
2354
+ "<|special_2471|>",
2355
+ "<|special_2472|>",
2356
+ "<|special_2473|>",
2357
+ "<|special_2474|>",
2358
+ "<|special_2475|>",
2359
+ "<|special_2476|>",
2360
+ "<|special_2477|>",
2361
+ "<|special_2478|>",
2362
+ "<|special_2479|>",
2363
+ "<|special_2480|>",
2364
+ "<|special_2481|>",
2365
+ "<|special_2482|>",
2366
+ "<|special_2483|>",
2367
+ "<|special_2484|>",
2368
+ "<|special_2485|>",
2369
+ "<|special_2486|>",
2370
+ "<|special_2487|>",
2371
+ "<|special_2488|>",
2372
+ "<|special_2489|>",
2373
+ "<|special_2490|>",
2374
+ "<|special_2491|>",
2375
+ "<|special_2492|>",
2376
+ "<|special_2493|>",
2377
+ "<|special_2494|>",
2378
+ "<|special_2495|>",
2379
+ "<|special_2496|>",
2380
+ "<|special_2497|>",
2381
+ "<|special_2498|>",
2382
+ "<|special_2499|>",
2383
+ "<|special_2500|>",
2384
+ "<|special_2501|>",
2385
+ "<|special_2502|>",
2386
+ "<|special_2503|>",
2387
+ "<|special_2504|>",
2388
+ "<|special_2505|>",
2389
+ "<|special_2506|>",
2390
+ "<|special_2507|>",
2391
+ "<|special_2508|>",
2392
+ "<|special_2509|>",
2393
+ "<|special_2510|>",
2394
+ "<|special_2511|>",
2395
+ "<|special_2512|>",
2396
+ "<|special_2513|>",
2397
+ "<|special_2514|>",
2398
+ "<|special_2515|>",
2399
+ "<|special_2516|>",
2400
+ "<|special_2517|>",
2401
+ "<|special_2518|>",
2402
+ "<|special_2519|>",
2403
+ "<|special_2520|>",
2404
+ "<|special_2521|>",
2405
+ "<|special_2522|>",
2406
+ "<|special_2523|>",
2407
+ "<|special_2524|>",
2408
+ "<|special_2525|>",
2409
+ "<|special_2526|>",
2410
+ "<|special_2527|>",
2411
+ "<|special_2528|>",
2412
+ "<|special_2529|>",
2413
+ "<|special_2530|>",
2414
+ "<|special_2531|>",
2415
+ "<|special_2532|>",
2416
+ "<|special_2533|>",
2417
+ "<|special_2534|>",
2418
+ "<|special_2535|>",
2419
+ "<|special_2536|>",
2420
+ "<|special_2537|>",
2421
+ "<|special_2538|>",
2422
+ "<|special_2539|>",
2423
+ "<|special_2540|>",
2424
+ "<|special_2541|>",
2425
+ "<|special_2542|>",
2426
+ "<|special_2543|>",
2427
+ "<|special_2544|>",
2428
+ "<|special_2545|>",
2429
+ "<|special_2546|>",
2430
+ "<|special_2547|>",
2431
+ "<|special_2548|>",
2432
+ "<|special_2549|>",
2433
+ "<|special_2550|>",
2434
+ "<|special_2551|>",
2435
+ "<|special_2552|>",
2436
+ "<|special_2553|>",
2437
+ "<|special_2554|>",
2438
+ "<|special_2555|>",
2439
+ "<|special_2556|>",
2440
+ "<|special_2557|>",
2441
+ "<|special_2558|>",
2442
+ "<|special_2559|>",
2443
+ "<|special_2560|>",
2444
+ "<|special_2561|>",
2445
+ "<|special_2562|>",
2446
+ "<|special_2563|>",
2447
+ "<|special_2564|>",
2448
+ "<|special_2565|>",
2449
+ "<|special_2566|>",
2450
+ "<|special_2567|>",
2451
+ "<|special_2568|>",
2452
+ "<|special_2569|>",
2453
+ "<|special_2570|>",
2454
+ "<|special_2571|>",
2455
+ "<|special_2572|>",
2456
+ "<|special_2573|>",
2457
+ "<|special_2574|>",
2458
+ "<|special_2575|>",
2459
+ "<|special_2576|>",
2460
+ "<|special_2577|>",
2461
+ "<|special_2578|>",
2462
+ "<|special_2579|>",
2463
+ "<|special_2580|>",
2464
+ "<|special_2581|>",
2465
+ "<|special_2582|>",
2466
+ "<|special_2583|>",
2467
+ "<|special_2584|>",
2468
+ "<|special_2585|>",
2469
+ "<|special_2586|>",
2470
+ "<|special_2587|>",
2471
+ "<|special_2588|>",
2472
+ "<|special_2589|>",
2473
+ "<|special_2590|>",
2474
+ "<|special_2591|>",
2475
+ "<|special_2592|>",
2476
+ "<|special_2593|>",
2477
+ "<|special_2594|>",
2478
+ "<|special_2595|>",
2479
+ "<|special_2596|>",
2480
+ "<|special_2597|>",
2481
+ "<|special_2598|>",
2482
+ "<|special_2599|>",
2483
+ "<|special_2600|>",
2484
+ "<|special_2601|>",
2485
+ "<|special_2602|>",
2486
+ "<|special_2603|>",
2487
+ "<|special_2604|>",
2488
+ "<|special_2605|>",
2489
+ "<|special_2606|>",
2490
+ "<|special_2607|>",
2491
+ "<|special_2608|>",
2492
+ "<|special_2609|>",
2493
+ "<|special_2610|>",
2494
+ "<|special_2611|>",
2495
+ "<|special_2612|>",
2496
+ "<|special_2613|>",
2497
+ "<|special_2614|>",
2498
+ "<|special_2615|>",
2499
+ "<|special_2616|>",
2500
+ "<|special_2617|>",
2501
+ "<|special_2618|>",
2502
+ "<|special_2619|>",
2503
+ "<|special_2620|>",
2504
+ "<|special_2621|>",
2505
+ "<|special_2622|>",
2506
+ "<|special_2623|>",
2507
+ "<|special_2624|>",
2508
+ "<|special_2625|>",
2509
+ "<|special_2626|>",
2510
+ "<|special_2627|>",
2511
+ "<|special_2628|>",
2512
+ "<|special_2629|>",
2513
+ "<|special_2630|>",
2514
+ "<|special_2631|>",
2515
+ "<|special_2632|>",
2516
+ "<|special_2633|>",
2517
+ "<|special_2634|>",
2518
+ "<|special_2635|>",
2519
+ "<|special_2636|>",
2520
+ "<|special_2637|>",
2521
+ "<|special_2638|>",
2522
+ "<|special_2639|>",
2523
+ "<|special_2640|>",
2524
+ "<|special_2641|>",
2525
+ "<|special_2642|>",
2526
+ "<|special_2643|>",
2527
+ "<|special_2644|>",
2528
+ "<|special_2645|>",
2529
+ "<|special_2646|>",
2530
+ "<|special_2647|>",
2531
+ "<|special_2648|>",
2532
+ "<|special_2649|>",
2533
+ "<|special_2650|>",
2534
+ "<|special_2651|>",
2535
+ "<|special_2652|>",
2536
+ "<|special_2653|>",
2537
+ "<|special_2654|>",
2538
+ "<|special_2655|>",
2539
+ "<|special_2656|>",
2540
+ "<|special_2657|>",
2541
+ "<|special_2658|>",
2542
+ "<|special_2659|>",
2543
+ "<|special_2660|>",
2544
+ "<|special_2661|>",
2545
+ "<|special_2662|>",
2546
+ "<|special_2663|>",
2547
+ "<|special_2664|>",
2548
+ "<|special_2665|>",
2549
+ "<|special_2666|>",
2550
+ "<|special_2667|>",
2551
+ "<|special_2668|>",
2552
+ "<|special_2669|>",
2553
+ "<|special_2670|>",
2554
+ "<|special_2671|>",
2555
+ "<|special_2672|>",
2556
+ "<|special_2673|>",
2557
+ "<|special_2674|>",
2558
+ "<|special_2675|>",
2559
+ "<|special_2676|>",
2560
+ "<|special_2677|>",
2561
+ "<|special_2678|>",
2562
+ "<|special_2679|>",
2563
+ "<|special_2680|>",
2564
+ "<|special_2681|>",
2565
+ "<|special_2682|>",
2566
+ "<|special_2683|>",
2567
+ "<|special_2684|>",
2568
+ "<|special_2685|>",
2569
+ "<|special_2686|>",
2570
+ "<|special_2687|>",
2571
+ "<|special_2688|>",
2572
+ "<|special_2689|>",
2573
+ "<|special_2690|>",
2574
+ "<|special_2691|>",
2575
+ "<|special_2692|>",
2576
+ "<|special_2693|>",
2577
+ "<|special_2694|>",
2578
+ "<|special_2695|>",
2579
+ "<|special_2696|>",
2580
+ "<|special_2697|>",
2581
+ "<|special_2698|>",
2582
+ "<|special_2699|>",
2583
+ "<|special_2700|>",
2584
+ "<|special_2701|>",
2585
+ "<|special_2702|>",
2586
+ "<|special_2703|>",
2587
+ "<|special_2704|>",
2588
+ "<|special_2705|>",
2589
+ "<|special_2706|>",
2590
+ "<|special_2707|>",
2591
+ "<|special_2708|>",
2592
+ "<|special_2709|>",
2593
+ "<|special_2710|>",
2594
+ "<|special_2711|>",
2595
+ "<|special_2712|>",
2596
+ "<|special_2713|>",
2597
+ "<|special_2714|>",
2598
+ "<|special_2715|>",
2599
+ "<|special_2716|>",
2600
+ "<|special_2717|>",
2601
+ "<|special_2718|>",
2602
+ "<|special_2719|>",
2603
+ "<|special_2720|>",
2604
+ "<|special_2721|>",
2605
+ "<|special_2722|>",
2606
+ "<|special_2723|>",
2607
+ "<|special_2724|>",
2608
+ "<|special_2725|>",
2609
+ "<|special_2726|>",
2610
+ "<|special_2727|>",
2611
+ "<|special_2728|>",
2612
+ "<|special_2729|>",
2613
+ "<|special_2730|>",
2614
+ "<|special_2731|>",
2615
+ "<|special_2732|>",
2616
+ "<|special_2733|>",
2617
+ "<|special_2734|>",
2618
+ "<|special_2735|>",
2619
+ "<|special_2736|>",
2620
+ "<|special_2737|>",
2621
+ "<|special_2738|>",
2622
+ "<|special_2739|>",
2623
+ "<|special_2740|>",
2624
+ "<|special_2741|>",
2625
+ "<|special_2742|>",
2626
+ "<|special_2743|>",
2627
+ "<|special_2744|>",
2628
+ "<|special_2745|>",
2629
+ "<|special_2746|>",
2630
+ "<|special_2747|>",
2631
+ "<|special_2748|>",
2632
+ "<|special_2749|>",
2633
+ "<|special_2750|>",
2634
+ "<|special_2751|>",
2635
+ "<|special_2752|>",
2636
+ "<|special_2753|>",
2637
+ "<|special_2754|>",
2638
+ "<|special_2755|>",
2639
+ "<|special_2756|>",
2640
+ "<|special_2757|>",
2641
+ "<|special_2758|>",
2642
+ "<|special_2759|>",
2643
+ "<|special_2760|>",
2644
+ "<|special_2761|>",
2645
+ "<|special_2762|>",
2646
+ "<|special_2763|>",
2647
+ "<|special_2764|>",
2648
+ "<|special_2765|>",
2649
+ "<|special_2766|>",
2650
+ "<|special_2767|>",
2651
+ "<|special_2768|>",
2652
+ "<|special_2769|>",
2653
+ "<|special_2770|>",
2654
+ "<|special_2771|>",
2655
+ "<|special_2772|>",
2656
+ "<|special_2773|>",
2657
+ "<|special_2774|>",
2658
+ "<|special_2775|>",
2659
+ "<|special_2776|>",
2660
+ "<|special_2777|>",
2661
+ "<|special_2778|>",
2662
+ "<|special_2779|>",
2663
+ "<|special_2780|>",
2664
+ "<|special_2781|>",
2665
+ "<|special_2782|>",
2666
+ "<|special_2783|>",
2667
+ "<|special_2784|>",
2668
+ "<|special_2785|>",
2669
+ "<|special_2786|>",
2670
+ "<|special_2787|>",
2671
+ "<|special_2788|>",
2672
+ "<|special_2789|>",
2673
+ "<|special_2790|>",
2674
+ "<|special_2791|>",
2675
+ "<|special_2792|>",
2676
+ "<|special_2793|>",
2677
+ "<|special_2794|>",
2678
+ "<|special_2795|>",
2679
+ "<|special_2796|>",
2680
+ "<|special_2797|>",
2681
+ "<|special_2798|>",
2682
+ "<|special_2799|>",
2683
+ "<|special_2800|>",
2684
+ "<|special_2801|>",
2685
+ "<|special_2802|>",
2686
+ "<|special_2803|>",
2687
+ "<|special_2804|>",
2688
+ "<|special_2805|>",
2689
+ "<|special_2806|>",
2690
+ "<|special_2807|>",
2691
+ "<|special_2808|>",
2692
+ "<|special_2809|>",
2693
+ "<|special_2810|>",
2694
+ "<|special_2811|>",
2695
+ "<|special_2812|>",
2696
+ "<|special_2813|>",
2697
+ "<|special_2814|>",
2698
+ "<|special_2815|>",
2699
+ "<|special_2816|>",
2700
+ "<|special_2817|>",
2701
+ "<|special_2818|>",
2702
+ "<|special_2819|>",
2703
+ "<|special_2820|>",
2704
+ "<|special_2821|>",
2705
+ "<|special_2822|>",
2706
+ "<|special_2823|>",
2707
+ "<|special_2824|>",
2708
+ "<|special_2825|>",
2709
+ "<|special_2826|>",
2710
+ "<|special_2827|>",
2711
+ "<|special_2828|>",
2712
+ "<|special_2829|>",
2713
+ "<|special_2830|>",
2714
+ "<|special_2831|>",
2715
+ "<|special_2832|>",
2716
+ "<|special_2833|>",
2717
+ "<|special_2834|>",
2718
+ "<|special_2835|>",
2719
+ "<|special_2836|>",
2720
+ "<|special_2837|>",
2721
+ "<|special_2838|>",
2722
+ "<|special_2839|>",
2723
+ "<|special_2840|>",
2724
+ "<|special_2841|>",
2725
+ "<|special_2842|>",
2726
+ "<|special_2843|>",
2727
+ "<|special_2844|>",
2728
+ "<|special_2845|>",
2729
+ "<|special_2846|>",
2730
+ "<|special_2847|>",
2731
+ "<|special_2848|>",
2732
+ "<|special_2849|>",
2733
+ "<|special_2850|>",
2734
+ "<|special_2851|>",
2735
+ "<|special_2852|>",
2736
+ "<|special_2853|>",
2737
+ "<|special_2854|>",
2738
+ "<|special_2855|>",
2739
+ "<|special_2856|>",
2740
+ "<|special_2857|>",
2741
+ "<|special_2858|>",
2742
+ "<|special_2859|>",
2743
+ "<|special_2860|>",
2744
+ "<|special_2861|>",
2745
+ "<|special_2862|>",
2746
+ "<|special_2863|>",
2747
+ "<|special_2864|>",
2748
+ "<|special_2865|>",
2749
+ "<|special_2866|>",
2750
+ "<|special_2867|>",
2751
+ "<|special_2868|>",
2752
+ "<|special_2869|>",
2753
+ "<|special_2870|>",
2754
+ "<|special_2871|>",
2755
+ "<|special_2872|>",
2756
+ "<|special_2873|>",
2757
+ "<|special_2874|>",
2758
+ "<|special_2875|>",
2759
+ "<|special_2876|>",
2760
+ "<|special_2877|>",
2761
+ "<|special_2878|>",
2762
+ "<|special_2879|>",
2763
+ "<|special_2880|>",
2764
+ "<|special_2881|>",
2765
+ "<|special_2882|>",
2766
+ "<|special_2883|>",
2767
+ "<|special_2884|>",
2768
+ "<|special_2885|>",
2769
+ "<|special_2886|>",
2770
+ "<|special_2887|>",
2771
+ "<|special_2888|>",
2772
+ "<|special_2889|>",
2773
+ "<|special_2890|>",
2774
+ "<|special_2891|>",
2775
+ "<|special_2892|>",
2776
+ "<|special_2893|>",
2777
+ "<|special_2894|>",
2778
+ "<|special_2895|>",
2779
+ "<|special_2896|>",
2780
+ "<|special_2897|>",
2781
+ "<|special_2898|>",
2782
+ "<|special_2899|>",
2783
+ "<|special_2900|>",
2784
+ "<|special_2901|>",
2785
+ "<|special_2902|>",
2786
+ "<|special_2903|>",
2787
+ "<|special_2904|>",
2788
+ "<|special_2905|>",
2789
+ "<|special_2906|>",
2790
+ "<|special_2907|>",
2791
+ "<|special_2908|>",
2792
+ "<|special_2909|>",
2793
+ "<|special_2910|>",
2794
+ "<|special_2911|>",
2795
+ "<|special_2912|>",
2796
+ "<|special_2913|>",
2797
+ "<|special_2914|>",
2798
+ "<|special_2915|>",
2799
+ "<|special_2916|>",
2800
+ "<|special_2917|>",
2801
+ "<|special_2918|>",
2802
+ "<|special_2919|>",
2803
+ "<|special_2920|>",
2804
+ "<|special_2921|>",
2805
+ "<|special_2922|>",
2806
+ "<|special_2923|>",
2807
+ "<|special_2924|>",
2808
+ "<|special_2925|>",
2809
+ "<|special_2926|>",
2810
+ "<|special_2927|>",
2811
+ "<|special_2928|>",
2812
+ "<|special_2929|>",
2813
+ "<|special_2930|>",
2814
+ "<|special_2931|>",
2815
+ "<|special_2932|>",
2816
+ "<|special_2933|>",
2817
+ "<|special_2934|>",
2818
+ "<|special_2935|>",
2819
+ "<|special_2936|>",
2820
+ "<|special_2937|>",
2821
+ "<|special_2938|>",
2822
+ "<|special_2939|>",
2823
+ "<|special_2940|>",
2824
+ "<|special_2941|>",
2825
+ "<|special_2942|>",
2826
+ "<|special_2943|>",
2827
+ "<|special_2944|>",
2828
+ "<|special_2945|>",
2829
+ "<|special_2946|>",
2830
+ "<|special_2947|>",
2831
+ "<|special_2948|>",
2832
+ "<|special_2949|>",
2833
+ "<|special_2950|>",
2834
+ "<|special_2951|>",
2835
+ "<|special_2952|>",
2836
+ "<|special_2953|>",
2837
+ "<|special_2954|>",
2838
+ "<|special_2955|>",
2839
+ "<|special_2956|>",
2840
+ "<|special_2957|>",
2841
+ "<|special_2958|>",
2842
+ "<|special_2959|>",
2843
+ "<|special_2960|>",
2844
+ "<|special_2961|>",
2845
+ "<|special_2962|>",
2846
+ "<|special_2963|>",
2847
+ "<|special_2964|>",
2848
+ "<|special_2965|>",
2849
+ "<|special_2966|>",
2850
+ "<|special_2967|>",
2851
+ "<|special_2968|>",
2852
+ "<|special_2969|>",
2853
+ "<|special_2970|>",
2854
+ "<|special_2971|>",
2855
+ "<|special_2972|>",
2856
+ "<|special_2973|>",
2857
+ "<|special_2974|>",
2858
+ "<|special_2975|>",
2859
+ "<|special_2976|>",
2860
+ "<|special_2977|>",
2861
+ "<|special_2978|>",
2862
+ "<|special_2979|>",
2863
+ "<|special_2980|>",
2864
+ "<|special_2981|>",
2865
+ "<|special_2982|>",
2866
+ "<|special_2983|>",
2867
+ "<|special_2984|>",
2868
+ "<|special_2985|>",
2869
+ "<|special_2986|>",
2870
+ "<|special_2987|>",
2871
+ "<|special_2988|>",
2872
+ "<|special_2989|>",
2873
+ "<|special_2990|>",
2874
+ "<|special_2991|>",
2875
+ "<|special_2992|>",
2876
+ "<|special_2993|>",
2877
+ "<|special_2994|>",
2878
+ "<|special_2995|>",
2879
+ "<|special_2996|>",
2880
+ "<|special_2997|>",
2881
+ "<|special_2998|>",
2882
+ "<|special_2999|>",
2883
+ "<|special_3000|>",
2884
+ "<|special_3001|>",
2885
+ "<|special_3002|>",
2886
+ "<|special_3003|>",
2887
+ "<|special_3004|>",
2888
+ "<|special_3005|>",
2889
+ "<|special_3006|>",
2890
+ "<|special_3007|>",
2891
+ "<|special_3008|>",
2892
+ "<|special_3009|>",
2893
+ "<|special_3010|>",
2894
+ "<|special_3011|>",
2895
+ "<|special_3012|>",
2896
+ "<|special_3013|>",
2897
+ "<|special_3014|>",
2898
+ "<|special_3015|>",
2899
+ "<|special_3016|>",
2900
+ "<|special_3017|>",
2901
+ "<|special_3018|>",
2902
+ "<|special_3019|>",
2903
+ "<|special_3020|>",
2904
+ "<|special_3021|>",
2905
+ "<|special_3022|>",
2906
+ "<|special_3023|>",
2907
+ "<|special_3024|>",
2908
+ "<|special_3025|>",
2909
+ "<|special_3026|>",
2910
+ "<|special_3027|>",
2911
+ "<|special_3028|>",
2912
+ "<|special_3029|>",
2913
+ "<|special_3030|>",
2914
+ "<|special_3031|>",
2915
+ "<|special_3032|>",
2916
+ "<|special_3033|>",
2917
+ "<|special_3034|>",
2918
+ "<|special_3035|>",
2919
+ "<|special_3036|>",
2920
+ "<|special_3037|>",
2921
+ "<|special_3038|>",
2922
+ "<|special_3039|>",
2923
+ "<|special_3040|>",
2924
+ "<|special_3041|>",
2925
+ "<|special_3042|>",
2926
+ "<|special_3043|>",
2927
+ "<|special_3044|>",
2928
+ "<|special_3045|>",
2929
+ "<|special_3046|>",
2930
+ "<|special_3047|>",
2931
+ "<|special_3048|>",
2932
+ "<|special_3049|>",
2933
+ "<|special_3050|>",
2934
+ "<|special_3051|>",
2935
+ "<|special_3052|>",
2936
+ "<|special_3053|>",
2937
+ "<|special_3054|>",
2938
+ "<|special_3055|>",
2939
+ "<|special_3056|>",
2940
+ "<|special_3057|>",
2941
+ "<|special_3058|>",
2942
+ "<|special_3059|>",
2943
+ "<|special_3060|>",
2944
+ "<|special_3061|>",
2945
+ "<|special_3062|>",
2946
+ "<|special_3063|>",
2947
+ "<|special_3064|>",
2948
+ "<|special_3065|>",
2949
+ "<|special_3066|>",
2950
+ "<|special_3067|>",
2951
+ "<|special_3068|>",
2952
+ "<|special_3069|>",
2953
+ "<|special_3070|>",
2954
+ "<|special_3071|>",
2955
+ "<|special_3072|>",
2956
+ "<|special_3073|>",
2957
+ "<|special_3074|>",
2958
+ "<|special_3075|>",
2959
+ "<|special_3076|>",
2960
+ "<|special_3077|>",
2961
+ "<|special_3078|>",
2962
+ "<|special_3079|>",
2963
+ "<|special_3080|>",
2964
+ "<|special_3081|>",
2965
+ "<|special_3082|>",
2966
+ "<|special_3083|>",
2967
+ "<|special_3084|>",
2968
+ "<|special_3085|>",
2969
+ "<|special_3086|>",
2970
+ "<|special_3087|>",
2971
+ "<|special_3088|>",
2972
+ "<|special_3089|>",
2973
+ "<|special_3090|>",
2974
+ "<|special_3091|>",
2975
+ "<|special_3092|>",
2976
+ "<|special_3093|>",
2977
+ "<|special_3094|>",
2978
+ "<|special_3095|>",
2979
+ "<|special_3096|>",
2980
+ "<|special_3097|>",
2981
+ "<|special_3098|>",
2982
+ "<|special_3099|>",
2983
+ "<|special_3100|>",
2984
+ "<|special_3101|>",
2985
+ "<|special_3102|>",
2986
+ "<|special_3103|>",
2987
+ "<|special_3104|>",
2988
+ "<|special_3105|>",
2989
+ "<|special_3106|>",
2990
+ "<|special_3107|>",
2991
+ "<|special_3108|>",
2992
+ "<|special_3109|>",
2993
+ "<|special_3110|>",
2994
+ "<|special_3111|>",
2995
+ "<|special_3112|>",
2996
+ "<|special_3113|>",
2997
+ "<|special_3114|>",
2998
+ "<|special_3115|>",
2999
+ "<|special_3116|>",
3000
+ "<|special_3117|>",
3001
+ "<|special_3118|>",
3002
+ "<|special_3119|>",
3003
+ "<|special_3120|>",
3004
+ "<|special_3121|>",
3005
+ "<|special_3122|>",
3006
+ "<|special_3123|>",
3007
+ "<|special_3124|>",
3008
+ "<|special_3125|>",
3009
+ "<|special_3126|>",
3010
+ "<|special_3127|>",
3011
+ "<|special_3128|>",
3012
+ "<|special_3129|>",
3013
+ "<|special_3130|>",
3014
+ "<|special_3131|>",
3015
+ "<|special_3132|>",
3016
+ "<|special_3133|>",
3017
+ "<|special_3134|>",
3018
+ "<|special_3135|>",
3019
+ "<|special_3136|>",
3020
+ "<|special_3137|>",
3021
+ "<|special_3138|>",
3022
+ "<|special_3139|>",
3023
+ "<|special_3140|>",
3024
+ "<|special_3141|>",
3025
+ "<|special_3142|>",
3026
+ "<|special_3143|>",
3027
+ "<|special_3144|>",
3028
+ "<|special_3145|>",
3029
+ "<|special_3146|>",
3030
+ "<|special_3147|>",
3031
+ "<|special_3148|>",
3032
+ "<|special_3149|>",
3033
+ "<|special_3150|>",
3034
+ "<|special_3151|>",
3035
+ "<|special_3152|>",
3036
+ "<|special_3153|>",
3037
+ "<|special_3154|>",
3038
+ "<|special_3155|>",
3039
+ "<|special_3156|>",
3040
+ "<|special_3157|>",
3041
+ "<|special_3158|>",
3042
+ "<|special_3159|>",
3043
+ "<|special_3160|>",
3044
+ "<|special_3161|>",
3045
+ "<|special_3162|>",
3046
+ "<|special_3163|>",
3047
+ "<|special_3164|>",
3048
+ "<|special_3165|>",
3049
+ "<|special_3166|>",
3050
+ "<|special_3167|>",
3051
+ "<|special_3168|>",
3052
+ "<|special_3169|>",
3053
+ "<|special_3170|>",
3054
+ "<|special_3171|>",
3055
+ "<|special_3172|>",
3056
+ "<|special_3173|>",
3057
+ "<|special_3174|>",
3058
+ "<|special_3175|>",
3059
+ "<|special_3176|>",
3060
+ "<|special_3177|>",
3061
+ "<|special_3178|>",
3062
+ "<|special_3179|>",
3063
+ "<|special_3180|>",
3064
+ "<|special_3181|>",
3065
+ "<|special_3182|>",
3066
+ "<|special_3183|>",
3067
+ "<|special_3184|>",
3068
+ "<|special_3185|>",
3069
+ "<|special_3186|>",
3070
+ "<|special_3187|>",
3071
+ "<|special_3188|>",
3072
+ "<|special_3189|>",
3073
+ "<|special_3190|>",
3074
+ "<|special_3191|>",
3075
+ "<|special_3192|>",
3076
+ "<|special_3193|>",
3077
+ "<|special_3194|>",
3078
+ "<|special_3195|>",
3079
+ "<|special_3196|>",
3080
+ "<|special_3197|>",
3081
+ "<|special_3198|>",
3082
+ "<|special_3199|>",
3083
+ "<|special_3200|>",
3084
+ "<|special_3201|>",
3085
+ "<|special_3202|>",
3086
+ "<|special_3203|>",
3087
+ "<|special_3204|>",
3088
+ "<|special_3205|>",
3089
+ "<|special_3206|>",
3090
+ "<|special_3207|>",
3091
+ "<|special_3208|>",
3092
+ "<|special_3209|>",
3093
+ "<|special_3210|>",
3094
+ "<|special_3211|>",
3095
+ "<|special_3212|>",
3096
+ "<|special_3213|>",
3097
+ "<|special_3214|>",
3098
+ "<|special_3215|>",
3099
+ "<|special_3216|>",
3100
+ "<|special_3217|>",
3101
+ "<|special_3218|>",
3102
+ "<|special_3219|>",
3103
+ "<|special_3220|>",
3104
+ "<|special_3221|>",
3105
+ "<|special_3222|>",
3106
+ "<|special_3223|>",
3107
+ "<|special_3224|>",
3108
+ "<|special_3225|>",
3109
+ "<|special_3226|>",
3110
+ "<|special_3227|>",
3111
+ "<|special_3228|>",
3112
+ "<|special_3229|>",
3113
+ "<|special_3230|>",
3114
+ "<|special_3231|>",
3115
+ "<|special_3232|>",
3116
+ "<|special_3233|>",
3117
+ "<|special_3234|>",
3118
+ "<|special_3235|>",
3119
+ "<|special_3236|>",
3120
+ "<|special_3237|>",
3121
+ "<|special_3238|>",
3122
+ "<|special_3239|>",
3123
+ "<|special_3240|>",
3124
+ "<|special_3241|>",
3125
+ "<|special_3242|>",
3126
+ "<|special_3243|>",
3127
+ "<|special_3244|>",
3128
+ "<|special_3245|>",
3129
+ "<|special_3246|>",
3130
+ "<|special_3247|>",
3131
+ "<|special_3248|>",
3132
+ "<|special_3249|>",
3133
+ "<|special_3250|>",
3134
+ "<|special_3251|>",
3135
+ "<|special_3252|>",
3136
+ "<|special_3253|>",
3137
+ "<|special_3254|>",
3138
+ "<|special_3255|>",
3139
+ "<|special_3256|>",
3140
+ "<|special_3257|>",
3141
+ "<|special_3258|>",
3142
+ "<|special_3259|>",
3143
+ "<|special_3260|>",
3144
+ "<|special_3261|>",
3145
+ "<|special_3262|>",
3146
+ "<|special_3263|>",
3147
+ "<|special_3264|>",
3148
+ "<|special_3265|>",
3149
+ "<|special_3266|>",
3150
+ "<|special_3267|>",
3151
+ "<|special_3268|>",
3152
+ "<|special_3269|>",
3153
+ "<|special_3270|>",
3154
+ "<|special_3271|>",
3155
+ "<|special_3272|>",
3156
+ "<|special_3273|>",
3157
+ "<|special_3274|>",
3158
+ "<|special_3275|>",
3159
+ "<|special_3276|>",
3160
+ "<|special_3277|>",
3161
+ "<|special_3278|>",
3162
+ "<|special_3279|>",
3163
+ "<|special_3280|>",
3164
+ "<|special_3281|>",
3165
+ "<|special_3282|>",
3166
+ "<|special_3283|>",
3167
+ "<|special_3284|>",
3168
+ "<|special_3285|>",
3169
+ "<|special_3286|>",
3170
+ "<|special_3287|>",
3171
+ "<|special_3288|>",
3172
+ "<|special_3289|>",
3173
+ "<|special_3290|>",
3174
+ "<|special_3291|>",
3175
+ "<|special_3292|>",
3176
+ "<|special_3293|>",
3177
+ "<|special_3294|>",
3178
+ "<|special_3295|>",
3179
+ "<|special_3296|>",
3180
+ "<|special_3297|>",
3181
+ "<|special_3298|>",
3182
+ "<|special_3299|>",
3183
+ "<|special_3300|>",
3184
+ "<|special_3301|>",
3185
+ "<|special_3302|>",
3186
+ "<|special_3303|>",
3187
+ "<|special_3304|>",
3188
+ "<|special_3305|>",
3189
+ "<|special_3306|>",
3190
+ "<|special_3307|>",
3191
+ "<|special_3308|>",
3192
+ "<|special_3309|>",
3193
+ "<|special_3310|>",
3194
+ "<|special_3311|>",
3195
+ "<|special_3312|>",
3196
+ "<|special_3313|>",
3197
+ "<|special_3314|>",
3198
+ "<|special_3315|>",
3199
+ "<|special_3316|>",
3200
+ "<|special_3317|>",
3201
+ "<|special_3318|>",
3202
+ "<|special_3319|>",
3203
+ "<|special_3320|>",
3204
+ "<|special_3321|>",
3205
+ "<|special_3322|>",
3206
+ "<|special_3323|>",
3207
+ "<|special_3324|>",
3208
+ "<|special_3325|>",
3209
+ "<|special_3326|>",
3210
+ "<|special_3327|>",
3211
+ "<|special_3328|>",
3212
+ "<|special_3329|>",
3213
+ "<|special_3330|>",
3214
+ "<|special_3331|>",
3215
+ "<|special_3332|>",
3216
+ "<|special_3333|>",
3217
+ "<|special_3334|>",
3218
+ "<|special_3335|>",
3219
+ "<|special_3336|>",
3220
+ "<|special_3337|>",
3221
+ "<|special_3338|>",
3222
+ "<|special_3339|>",
3223
+ "<|special_3340|>",
3224
+ "<|special_3341|>",
3225
+ "<|special_3342|>",
3226
+ "<|special_3343|>",
3227
+ "<|special_3344|>",
3228
+ "<|special_3345|>",
3229
+ "<|special_3346|>",
3230
+ "<|special_3347|>",
3231
+ "<|special_3348|>",
3232
+ "<|special_3349|>",
3233
+ "<|special_3350|>",
3234
+ "<|special_3351|>",
3235
+ "<|special_3352|>",
3236
+ "<|special_3353|>",
3237
+ "<|special_3354|>",
3238
+ "<|special_3355|>",
3239
+ "<|special_3356|>",
3240
+ "<|special_3357|>",
3241
+ "<|special_3358|>",
3242
+ "<|special_3359|>",
3243
+ "<|special_3360|>",
3244
+ "<|special_3361|>",
3245
+ "<|special_3362|>",
3246
+ "<|special_3363|>",
3247
+ "<|special_3364|>",
3248
+ "<|special_3365|>",
3249
+ "<|special_3366|>",
3250
+ "<|special_3367|>",
3251
+ "<|special_3368|>",
3252
+ "<|special_3369|>",
3253
+ "<|special_3370|>",
3254
+ "<|special_3371|>",
3255
+ "<|special_3372|>",
3256
+ "<|special_3373|>",
3257
+ "<|special_3374|>",
3258
+ "<|special_3375|>",
3259
+ "<|special_3376|>",
3260
+ "<|special_3377|>",
3261
+ "<|special_3378|>",
3262
+ "<|special_3379|>",
3263
+ "<|special_3380|>",
3264
+ "<|special_3381|>",
3265
+ "<|special_3382|>",
3266
+ "<|special_3383|>",
3267
+ "<|special_3384|>",
3268
+ "<|special_3385|>",
3269
+ "<|special_3386|>",
3270
+ "<|special_3387|>",
3271
+ "<|special_3388|>",
3272
+ "<|special_3389|>",
3273
+ "<|special_3390|>",
3274
+ "<|special_3391|>",
3275
+ "<|special_3392|>",
3276
+ "<|special_3393|>",
3277
+ "<|special_3394|>",
3278
+ "<|special_3395|>",
3279
+ "<|special_3396|>",
3280
+ "<|special_3397|>",
3281
+ "<|special_3398|>",
3282
+ "<|special_3399|>",
3283
+ "<|special_3400|>",
3284
+ "<|special_3401|>",
3285
+ "<|special_3402|>",
3286
+ "<|special_3403|>",
3287
+ "<|special_3404|>",
3288
+ "<|special_3405|>",
3289
+ "<|special_3406|>",
3290
+ "<|special_3407|>",
3291
+ "<|special_3408|>",
3292
+ "<|special_3409|>",
3293
+ "<|special_3410|>",
3294
+ "<|special_3411|>",
3295
+ "<|special_3412|>",
3296
+ "<|special_3413|>",
3297
+ "<|special_3414|>",
3298
+ "<|special_3415|>",
3299
+ "<|special_3416|>",
3300
+ "<|special_3417|>",
3301
+ "<|special_3418|>",
3302
+ "<|special_3419|>",
3303
+ "<|special_3420|>",
3304
+ "<|special_3421|>",
3305
+ "<|special_3422|>",
3306
+ "<|special_3423|>",
3307
+ "<|special_3424|>",
3308
+ "<|special_3425|>",
3309
+ "<|special_3426|>",
3310
+ "<|special_3427|>",
3311
+ "<|special_3428|>",
3312
+ "<|special_3429|>",
3313
+ "<|special_3430|>",
3314
+ "<|special_3431|>",
3315
+ "<|special_3432|>",
3316
+ "<|special_3433|>",
3317
+ "<|special_3434|>",
3318
+ "<|special_3435|>",
3319
+ "<|special_3436|>",
3320
+ "<|special_3437|>",
3321
+ "<|special_3438|>",
3322
+ "<|special_3439|>",
3323
+ "<|special_3440|>",
3324
+ "<|special_3441|>",
3325
+ "<|special_3442|>",
3326
+ "<|special_3443|>",
3327
+ "<|special_3444|>",
3328
+ "<|special_3445|>",
3329
+ "<|special_3446|>",
3330
+ "<|special_3447|>",
3331
+ "<|special_3448|>",
3332
+ "<|special_3449|>",
3333
+ "<|special_3450|>",
3334
+ "<|special_3451|>",
3335
+ "<|special_3452|>",
3336
+ "<|special_3453|>",
3337
+ "<|special_3454|>",
3338
+ "<|special_3455|>",
3339
+ "<|special_3456|>",
3340
+ "<|special_3457|>",
3341
+ "<|special_3458|>",
3342
+ "<|special_3459|>",
3343
+ "<|special_3460|>",
3344
+ "<|special_3461|>",
3345
+ "<|special_3462|>",
3346
+ "<|special_3463|>",
3347
+ "<|special_3464|>",
3348
+ "<|special_3465|>",
3349
+ "<|special_3466|>",
3350
+ "<|special_3467|>",
3351
+ "<|special_3468|>",
3352
+ "<|special_3469|>",
3353
+ "<|special_3470|>",
3354
+ "<|special_3471|>",
3355
+ "<|special_3472|>",
3356
+ "<|special_3473|>",
3357
+ "<|special_3474|>",
3358
+ "<|special_3475|>",
3359
+ "<|special_3476|>",
3360
+ "<|special_3477|>",
3361
+ "<|special_3478|>",
3362
+ "<|special_3479|>",
3363
+ "<|special_3480|>",
3364
+ "<|special_3481|>",
3365
+ "<|special_3482|>",
3366
+ "<|special_3483|>",
3367
+ "<|special_3484|>",
3368
+ "<|special_3485|>",
3369
+ "<|special_3486|>",
3370
+ "<|special_3487|>",
3371
+ "<|special_3488|>",
3372
+ "<|special_3489|>",
3373
+ "<|special_3490|>",
3374
+ "<|special_3491|>",
3375
+ "<|special_3492|>",
3376
+ "<|special_3493|>",
3377
+ "<|special_3494|>",
3378
+ "<|special_3495|>",
3379
+ "<|special_3496|>",
3380
+ "<|special_3497|>",
3381
+ "<|special_3498|>",
3382
+ "<|special_3499|>",
3383
+ "<|special_3500|>",
3384
+ "<|special_3501|>",
3385
+ "<|special_3502|>",
3386
+ "<|special_3503|>",
3387
+ "<|special_3504|>",
3388
+ "<|special_3505|>",
3389
+ "<|special_3506|>",
3390
+ "<|special_3507|>",
3391
+ "<|special_3508|>",
3392
+ "<|special_3509|>",
3393
+ "<|special_3510|>",
3394
+ "<|special_3511|>",
3395
+ "<|special_3512|>",
3396
+ "<|special_3513|>",
3397
+ "<|special_3514|>",
3398
+ "<|special_3515|>",
3399
+ "<|special_3516|>",
3400
+ "<|special_3517|>",
3401
+ "<|special_3518|>",
3402
+ "<|special_3519|>",
3403
+ "<|special_3520|>",
3404
+ "<|special_3521|>",
3405
+ "<|special_3522|>",
3406
+ "<|special_3523|>",
3407
+ "<|special_3524|>",
3408
+ "<|special_3525|>",
3409
+ "<|special_3526|>",
3410
+ "<|special_3527|>",
3411
+ "<|special_3528|>",
3412
+ "<|special_3529|>",
3413
+ "<|special_3530|>",
3414
+ "<|special_3531|>",
3415
+ "<|special_3532|>",
3416
+ "<|special_3533|>",
3417
+ "<|special_3534|>",
3418
+ "<|special_3535|>",
3419
+ "<|special_3536|>",
3420
+ "<|special_3537|>",
3421
+ "<|special_3538|>",
3422
+ "<|special_3539|>",
3423
+ "<|special_3540|>",
3424
+ "<|special_3541|>",
3425
+ "<|special_3542|>",
3426
+ "<|special_3543|>",
3427
+ "<|special_3544|>",
3428
+ "<|special_3545|>",
3429
+ "<|special_3546|>",
3430
+ "<|special_3547|>",
3431
+ "<|special_3548|>",
3432
+ "<|special_3549|>",
3433
+ "<|special_3550|>",
3434
+ "<|special_3551|>",
3435
+ "<|special_3552|>",
3436
+ "<|special_3553|>",
3437
+ "<|special_3554|>",
3438
+ "<|special_3555|>",
3439
+ "<|special_3556|>",
3440
+ "<|special_3557|>",
3441
+ "<|special_3558|>",
3442
+ "<|special_3559|>",
3443
+ "<|special_3560|>",
3444
+ "<|special_3561|>",
3445
+ "<|special_3562|>",
3446
+ "<|special_3563|>",
3447
+ "<|special_3564|>",
3448
+ "<|special_3565|>",
3449
+ "<|special_3566|>",
3450
+ "<|special_3567|>",
3451
+ "<|special_3568|>",
3452
+ "<|special_3569|>",
3453
+ "<|special_3570|>",
3454
+ "<|special_3571|>",
3455
+ "<|special_3572|>",
3456
+ "<|special_3573|>",
3457
+ "<|special_3574|>",
3458
+ "<|special_3575|>",
3459
+ "<|special_3576|>",
3460
+ "<|special_3577|>",
3461
+ "<|special_3578|>",
3462
+ "<|special_3579|>",
3463
+ "<|special_3580|>",
3464
+ "<|special_3581|>",
3465
+ "<|special_3582|>",
3466
+ "<|special_3583|>",
3467
+ "<|special_3584|>",
3468
+ "<|special_3585|>",
3469
+ "<|special_3586|>",
3470
+ "<|special_3587|>",
3471
+ "<|special_3588|>",
3472
+ "<|special_3589|>",
3473
+ "<|special_3590|>",
3474
+ "<|special_3591|>",
3475
+ "<|special_3592|>",
3476
+ "<|special_3593|>",
3477
+ "<|special_3594|>",
3478
+ "<|special_3595|>",
3479
+ "<|special_3596|>",
3480
+ "<|special_3597|>",
3481
+ "<|special_3598|>",
3482
+ "<|special_3599|>",
3483
+ "<|special_3600|>",
3484
+ "<|special_3601|>",
3485
+ "<|special_3602|>",
3486
+ "<|special_3603|>",
3487
+ "<|special_3604|>",
3488
+ "<|special_3605|>",
3489
+ "<|special_3606|>",
3490
+ "<|special_3607|>",
3491
+ "<|special_3608|>",
3492
+ "<|special_3609|>",
3493
+ "<|special_3610|>",
3494
+ "<|special_3611|>",
3495
+ "<|special_3612|>",
3496
+ "<|special_3613|>",
3497
+ "<|special_3614|>",
3498
+ "<|special_3615|>",
3499
+ "<|special_3616|>",
3500
+ "<|special_3617|>",
3501
+ "<|special_3618|>",
3502
+ "<|special_3619|>",
3503
+ "<|special_3620|>",
3504
+ "<|special_3621|>",
3505
+ "<|special_3622|>",
3506
+ "<|special_3623|>",
3507
+ "<|special_3624|>",
3508
+ "<|special_3625|>",
3509
+ "<|special_3626|>",
3510
+ "<|special_3627|>",
3511
+ "<|special_3628|>",
3512
+ "<|special_3629|>",
3513
+ "<|special_3630|>",
3514
+ "<|special_3631|>",
3515
+ "<|special_3632|>",
3516
+ "<|special_3633|>",
3517
+ "<|special_3634|>",
3518
+ "<|special_3635|>",
3519
+ "<|special_3636|>",
3520
+ "<|special_3637|>",
3521
+ "<|special_3638|>",
3522
+ "<|special_3639|>",
3523
+ "<|special_3640|>",
3524
+ "<|special_3641|>",
3525
+ "<|special_3642|>",
3526
+ "<|special_3643|>",
3527
+ "<|special_3644|>",
3528
+ "<|special_3645|>",
3529
+ "<|special_3646|>",
3530
+ "<|special_3647|>",
3531
+ "<|special_3648|>",
3532
+ "<|special_3649|>",
3533
+ "<|special_3650|>",
3534
+ "<|special_3651|>",
3535
+ "<|special_3652|>",
3536
+ "<|special_3653|>",
3537
+ "<|special_3654|>",
3538
+ "<|special_3655|>",
3539
+ "<|special_3656|>",
3540
+ "<|special_3657|>",
3541
+ "<|special_3658|>",
3542
+ "<|special_3659|>",
3543
+ "<|special_3660|>",
3544
+ "<|special_3661|>",
3545
+ "<|special_3662|>",
3546
+ "<|special_3663|>",
3547
+ "<|special_3664|>",
3548
+ "<|special_3665|>",
3549
+ "<|special_3666|>",
3550
+ "<|special_3667|>",
3551
+ "<|special_3668|>",
3552
+ "<|special_3669|>",
3553
+ "<|special_3670|>",
3554
+ "<|special_3671|>",
3555
+ "<|special_3672|>",
3556
+ "<|special_3673|>",
3557
+ "<|special_3674|>",
3558
+ "<|special_3675|>",
3559
+ "<|special_3676|>",
3560
+ "<|special_3677|>",
3561
+ "<|special_3678|>",
3562
+ "<|special_3679|>",
3563
+ "<|special_3680|>",
3564
+ "<|special_3681|>",
3565
+ "<|special_3682|>",
3566
+ "<|special_3683|>",
3567
+ "<|special_3684|>",
3568
+ "<|special_3685|>",
3569
+ "<|special_3686|>",
3570
+ "<|special_3687|>",
3571
+ "<|special_3688|>",
3572
+ "<|special_3689|>",
3573
+ "<|special_3690|>",
3574
+ "<|special_3691|>",
3575
+ "<|special_3692|>",
3576
+ "<|special_3693|>",
3577
+ "<|special_3694|>",
3578
+ "<|special_3695|>",
3579
+ "<|special_3696|>",
3580
+ "<|special_3697|>",
3581
+ "<|special_3698|>",
3582
+ "<|special_3699|>",
3583
+ "<|special_3700|>",
3584
+ "<|special_3701|>",
3585
+ "<|special_3702|>",
3586
+ "<|special_3703|>",
3587
+ "<|special_3704|>",
3588
+ "<|special_3705|>",
3589
+ "<|special_3706|>",
3590
+ "<|special_3707|>",
3591
+ "<|special_3708|>",
3592
+ "<|special_3709|>",
3593
+ "<|special_3710|>",
3594
+ "<|special_3711|>",
3595
+ "<|special_3712|>",
3596
+ "<|special_3713|>",
3597
+ "<|special_3714|>",
3598
+ "<|special_3715|>",
3599
+ "<|special_3716|>",
3600
+ "<|special_3717|>",
3601
+ "<|special_3718|>",
3602
+ "<|special_3719|>",
3603
+ "<|special_3720|>",
3604
+ "<|special_3721|>",
3605
+ "<|special_3722|>",
3606
+ "<|special_3723|>",
3607
+ "<|special_3724|>",
3608
+ "<|special_3725|>",
3609
+ "<|special_3726|>",
3610
+ "<|special_3727|>",
3611
+ "<|special_3728|>",
3612
+ "<|special_3729|>",
3613
+ "<|special_3730|>",
3614
+ "<|special_3731|>",
3615
+ "<|special_3732|>",
3616
+ "<|special_3733|>",
3617
+ "<|special_3734|>",
3618
+ "<|special_3735|>",
3619
+ "<|special_3736|>",
3620
+ "<|special_3737|>",
3621
+ "<|special_3738|>",
3622
+ "<|special_3739|>",
3623
+ "<|special_3740|>",
3624
+ "<|special_3741|>",
3625
+ "<|special_3742|>",
3626
+ "<|special_3743|>",
3627
+ "<|special_3744|>",
3628
+ "<|special_3745|>",
3629
+ "<|special_3746|>",
3630
+ "<|special_3747|>",
3631
+ "<|special_3748|>",
3632
+ "<|special_3749|>",
3633
+ "<|special_3750|>",
3634
+ "<|special_3751|>",
3635
+ "<|special_3752|>",
3636
+ "<|special_3753|>",
3637
+ "<|special_3754|>",
3638
+ "<|special_3755|>",
3639
+ "<|special_3756|>",
3640
+ "<|special_3757|>",
3641
+ "<|special_3758|>",
3642
+ "<|special_3759|>",
3643
+ "<|special_3760|>",
3644
+ "<|special_3761|>",
3645
+ "<|special_3762|>",
3646
+ "<|special_3763|>",
3647
+ "<|special_3764|>",
3648
+ "<|special_3765|>",
3649
+ "<|special_3766|>",
3650
+ "<|special_3767|>",
3651
+ "<|special_3768|>",
3652
+ "<|special_3769|>",
3653
+ "<|special_3770|>",
3654
+ "<|special_3771|>",
3655
+ "<|special_3772|>",
3656
+ "<|special_3773|>",
3657
+ "<|special_3774|>",
3658
+ "<|special_3775|>",
3659
+ "<|special_3776|>",
3660
+ "<|special_3777|>",
3661
+ "<|special_3778|>",
3662
+ "<|special_3779|>",
3663
+ "<|special_3780|>",
3664
+ "<|special_3781|>",
3665
+ "<|special_3782|>",
3666
+ "<|special_3783|>",
3667
+ "<|special_3784|>",
3668
+ "<|special_3785|>",
3669
+ "<|special_3786|>",
3670
+ "<|special_3787|>",
3671
+ "<|special_3788|>",
3672
+ "<|special_3789|>",
3673
+ "<|special_3790|>",
3674
+ "<|special_3791|>",
3675
+ "<|special_3792|>",
3676
+ "<|special_3793|>",
3677
+ "<|special_3794|>",
3678
+ "<|special_3795|>",
3679
+ "<|special_3796|>",
3680
+ "<|special_3797|>",
3681
+ "<|special_3798|>",
3682
+ "<|special_3799|>",
3683
+ "<|special_3800|>",
3684
+ "<|special_3801|>",
3685
+ "<|special_3802|>",
3686
+ "<|special_3803|>",
3687
+ "<|special_3804|>",
3688
+ "<|special_3805|>",
3689
+ "<|special_3806|>",
3690
+ "<|special_3807|>",
3691
+ "<|special_3808|>",
3692
+ "<|special_3809|>",
3693
+ "<|special_3810|>",
3694
+ "<|special_3811|>",
3695
+ "<|special_3812|>",
3696
+ "<|special_3813|>",
3697
+ "<|special_3814|>",
3698
+ "<|special_3815|>",
3699
+ "<|special_3816|>",
3700
+ "<|special_3817|>",
3701
+ "<|special_3818|>",
3702
+ "<|special_3819|>",
3703
+ "<|special_3820|>",
3704
+ "<|special_3821|>",
3705
+ "<|special_3822|>",
3706
+ "<|special_3823|>",
3707
+ "<|special_3824|>",
3708
+ "<|special_3825|>",
3709
+ "<|special_3826|>",
3710
+ "<|special_3827|>",
3711
+ "<|special_3828|>",
3712
+ "<|special_3829|>",
3713
+ "<|special_3830|>",
3714
+ "<|special_3831|>",
3715
+ "<|special_3832|>",
3716
+ "<|special_3833|>",
3717
+ "<|special_3834|>",
3718
+ "<|special_3835|>",
3719
+ "<|special_3836|>",
3720
+ "<|special_3837|>",
3721
+ "<|special_3838|>",
3722
+ "<|special_3839|>",
3723
+ "<|special_3840|>",
3724
+ "<|special_3841|>",
3725
+ "<|special_3842|>",
3726
+ "<|special_3843|>",
3727
+ "<|special_3844|>",
3728
+ "<|special_3845|>",
3729
+ "<|special_3846|>",
3730
+ "<|special_3847|>",
3731
+ "<|special_3848|>",
3732
+ "<|special_3849|>",
3733
+ "<|special_3850|>",
3734
+ "<|special_3851|>",
3735
+ "<|special_3852|>",
3736
+ "<|special_3853|>",
3737
+ "<|special_3854|>",
3738
+ "<|special_3855|>",
3739
+ "<|special_3856|>",
3740
+ "<|special_3857|>",
3741
+ "<|special_3858|>",
3742
+ "<|special_3859|>",
3743
+ "<|special_3860|>",
3744
+ "<|special_3861|>",
3745
+ "<|special_3862|>",
3746
+ "<|special_3863|>",
3747
+ "<|special_3864|>",
3748
+ "<|special_3865|>",
3749
+ "<|special_3866|>",
3750
+ "<|special_3867|>",
3751
+ "<|special_3868|>",
3752
+ "<|special_3869|>",
3753
+ "<|special_3870|>",
3754
+ "<|special_3871|>",
3755
+ "<|special_3872|>",
3756
+ "<|special_3873|>",
3757
+ "<|special_3874|>",
3758
+ "<|special_3875|>",
3759
+ "<|special_3876|>",
3760
+ "<|special_3877|>",
3761
+ "<|special_3878|>",
3762
+ "<|special_3879|>",
3763
+ "<|special_3880|>",
3764
+ "<|special_3881|>",
3765
+ "<|special_3882|>",
3766
+ "<|special_3883|>",
3767
+ "<|special_3884|>",
3768
+ "<|special_3885|>",
3769
+ "<|special_3886|>",
3770
+ "<|special_3887|>",
3771
+ "<|special_3888|>",
3772
+ "<|special_3889|>",
3773
+ "<|special_3890|>",
3774
+ "<|special_3891|>",
3775
+ "<|special_3892|>",
3776
+ "<|special_3893|>",
3777
+ "<|special_3894|>",
3778
+ "<|special_3895|>",
3779
+ "<|special_3896|>",
3780
+ "<|special_3897|>",
3781
+ "<|special_3898|>",
3782
+ "<|special_3899|>",
3783
+ "<|special_3900|>",
3784
+ "<|special_3901|>",
3785
+ "<|special_3902|>",
3786
+ "<|special_3903|>",
3787
+ "<|special_3904|>",
3788
+ "<|special_3905|>",
3789
+ "<|special_3906|>",
3790
+ "<|special_3907|>",
3791
+ "<|special_3908|>",
3792
+ "<|special_3909|>",
3793
+ "<|special_3910|>",
3794
+ "<|special_3911|>",
3795
+ "<|special_3912|>",
3796
+ "<|special_3913|>",
3797
+ "<|special_3914|>",
3798
+ "<|special_3915|>",
3799
+ "<|special_3916|>",
3800
+ "<|special_3917|>",
3801
+ "<|special_3918|>",
3802
+ "<|special_3919|>",
3803
+ "<|special_3920|>",
3804
+ "<|special_3921|>",
3805
+ "<|special_3922|>",
3806
+ "<|special_3923|>",
3807
+ "<|special_3924|>",
3808
+ "<|special_3925|>",
3809
+ "<|special_3926|>",
3810
+ "<|special_3927|>",
3811
+ "<|special_3928|>",
3812
+ "<|special_3929|>",
3813
+ "<|special_3930|>",
3814
+ "<|special_3931|>",
3815
+ "<|special_3932|>",
3816
+ "<|special_3933|>",
3817
+ "<|special_3934|>",
3818
+ "<|special_3935|>",
3819
+ "<|special_3936|>",
3820
+ "<|special_3937|>",
3821
+ "<|special_3938|>",
3822
+ "<|special_3939|>",
3823
+ "<|special_3940|>",
3824
+ "<|special_3941|>",
3825
+ "<|special_3942|>",
3826
+ "<|special_3943|>",
3827
+ "<|special_3944|>",
3828
+ "<|special_3945|>",
3829
+ "<|special_3946|>",
3830
+ "<|special_3947|>",
3831
+ "<|special_3948|>",
3832
+ "<|special_3949|>",
3833
+ "<|special_3950|>",
3834
+ "<|special_3951|>",
3835
+ "<|special_3952|>",
3836
+ "<|special_3953|>",
3837
+ "<|special_3954|>",
3838
+ "<|special_3955|>",
3839
+ "<|special_3956|>",
3840
+ "<|special_3957|>",
3841
+ "<|special_3958|>",
3842
+ "<|special_3959|>",
3843
+ "<|special_3960|>",
3844
+ "<|special_3961|>",
3845
+ "<|special_3962|>",
3846
+ "<|special_3963|>",
3847
+ "<|special_3964|>",
3848
+ "<|special_3965|>",
3849
+ "<|special_3966|>",
3850
+ "<|special_3967|>",
3851
+ "<|special_3968|>",
3852
+ "<|special_3969|>",
3853
+ "<|special_3970|>",
3854
+ "<|special_3971|>",
3855
+ "<|special_3972|>",
3856
+ "<|special_3973|>",
3857
+ "<|special_3974|>",
3858
+ "<|special_3975|>",
3859
+ "<|special_3976|>",
3860
+ "<|special_3977|>",
3861
+ "<|special_3978|>",
3862
+ "<|special_3979|>",
3863
+ "<|special_3980|>",
3864
+ "<|special_3981|>",
3865
+ "<|special_3982|>",
3866
+ "<|special_3983|>",
3867
+ "<|special_3984|>",
3868
+ "<|special_3985|>",
3869
+ "<|special_3986|>",
3870
+ "<|special_3987|>",
3871
+ "<|special_3988|>",
3872
+ "<|special_3989|>",
3873
+ "<|special_3990|>",
3874
+ "<|special_3991|>",
3875
+ "<|special_3992|>",
3876
+ "<|special_3993|>",
3877
+ "<|special_3994|>",
3878
+ "<|special_3995|>",
3879
+ "<|special_3996|>",
3880
+ "<|special_3997|>",
3881
+ "<|special_3998|>",
3882
+ "<|special_3999|>",
3883
+ "<|special_4000|>",
3884
+ "<|special_4001|>",
3885
+ "<|special_4002|>",
3886
+ "<|special_4003|>",
3887
+ "<|special_4004|>",
3888
+ "<|special_4005|>",
3889
+ "<|special_4006|>",
3890
+ "<|special_4007|>",
3891
+ "<|special_4008|>",
3892
+ "<|special_4009|>",
3893
+ "<|special_4010|>",
3894
+ "<|special_4011|>",
3895
+ "<|special_4012|>",
3896
+ "<|special_4013|>",
3897
+ "<|special_4014|>",
3898
+ "<|special_4015|>",
3899
+ "<|special_4016|>",
3900
+ "<|special_4017|>",
3901
+ "<|special_4018|>",
3902
+ "<|special_4019|>",
3903
+ "<|special_4020|>",
3904
+ "<|special_4021|>",
3905
+ "<|special_4022|>",
3906
+ "<|special_4023|>",
3907
+ "<|special_4024|>",
3908
+ "<|special_4025|>",
3909
+ "<|special_4026|>",
3910
+ "<|special_4027|>",
3911
+ "<|special_4028|>",
3912
+ "<|special_4029|>",
3913
+ "<|special_4030|>",
3914
+ "<|special_4031|>",
3915
+ "<|special_4032|>",
3916
+ "<|special_4033|>",
3917
+ "<|special_4034|>",
3918
+ "<|special_4035|>",
3919
+ "<|special_4036|>",
3920
+ "<|special_4037|>",
3921
+ "<|special_4038|>",
3922
+ "<|special_4039|>",
3923
+ "<|special_4040|>",
3924
+ "<|special_4041|>",
3925
+ "<|special_4042|>",
3926
+ "<|special_4043|>",
3927
+ "<|special_4044|>",
3928
+ "<|special_4045|>",
3929
+ "<|special_4046|>",
3930
+ "<|special_4047|>",
3931
+ "<|special_4048|>",
3932
+ "<|special_4049|>",
3933
+ "<|special_4050|>",
3934
+ "<|special_4051|>",
3935
+ "<|special_4052|>",
3936
+ "<|special_4053|>",
3937
+ "<|special_4054|>",
3938
+ "<|special_4055|>",
3939
+ "<|special_4056|>",
3940
+ "<|special_4057|>",
3941
+ "<|special_4058|>",
3942
+ "<|special_4059|>",
3943
+ "<|special_4060|>",
3944
+ "<|special_4061|>",
3945
+ "<|special_4062|>",
3946
+ "<|special_4063|>",
3947
+ "<|special_4064|>",
3948
+ "<|special_4065|>",
3949
+ "<|special_4066|>",
3950
+ "<|special_4067|>",
3951
+ "<|special_4068|>",
3952
+ "<|special_4069|>",
3953
+ "<|special_4070|>",
3954
+ "<|special_4071|>",
3955
+ "<|special_4072|>",
3956
+ "<|special_4073|>",
3957
+ "<|special_4074|>",
3958
+ "<|special_4075|>",
3959
+ "<|special_4076|>",
3960
+ "<|special_4077|>",
3961
+ "<|special_4078|>",
3962
+ "<|special_4079|>",
3963
+ "<|special_4080|>",
3964
+ "<|special_4081|>",
3965
+ "<|special_4082|>",
3966
+ "<|special_4083|>",
3967
+ "<|special_4084|>",
3968
+ "<|special_4085|>",
3969
+ "<|special_4086|>",
3970
+ "<|special_4087|>",
3971
+ "<|special_4088|>",
3972
+ "<|special_4089|>",
3973
+ "<|special_4090|>",
3974
+ "<|special_4091|>",
3975
+ "<|special_4092|>",
3976
+ "<|special_4093|>",
3977
+ "<|special_4094|>",
3978
+ "<|special_4095|>"
3979
+ ],
3980
+ "bos_token": {
3981
+ "content": "<|startoftext|>",
3982
+ "lstrip": false,
3983
+ "normalized": false,
3984
+ "rstrip": false,
3985
+ "single_word": false
3986
+ },
3987
+ "eos_token": {
3988
+ "content": "<|endoftext|>",
3989
+ "lstrip": false,
3990
+ "normalized": false,
3991
+ "rstrip": false,
3992
+ "single_word": false
3993
+ },
3994
+ "pad_token": {
3995
+ "content": "<|endoftext|>",
3996
+ "lstrip": false,
3997
+ "normalized": false,
3998
+ "rstrip": false,
3999
+ "single_word": false
4000
+ },
4001
+ "unk_token": {
4002
+ "content": "<unk>",
4003
+ "lstrip": false,
4004
+ "normalized": false,
4005
+ "rstrip": false,
4006
+ "single_word": false
4007
+ }
4008
+ }
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c7b7cb72447bbe1f8d856adaf34933acfdedfa89c7fe1d95d0fdabedf9d19a94
3
+ size 16473585
tokenizer_config.json ADDED
The diff for this file is too large to render. See raw diff