Upload folder using huggingface_hub

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,96 @@
+# Upstage Solar License
+
+# Preamble
+
+The 'Upstage Solar License' (hereinafter referred to as "this License") was established by Upstage Co., Ltd., incorporated under the laws of the Republic of Korea, to encourage the development of open-source software using Solar AI models, and is not affiliated with the Apache Software Foundation.
+
+This License basically adopts all provisions of the Apache License, Version 2.0 (hereinafter referred to as "Apache License 2.0"), including the principle of allowing commercial use, but prescribes minimum strategic conditions for the global expansion of AI technology and the sustainable development of the AI ecosystem.
+
+The key additional condition, as specified in Section 4(e), is that if you distribute a "Derivative AI Model" based on the "Work", you must specify the 'Solar' brand. This applies as an exception to Section 6 (Trademarks) of the Apache License 2.0.
+
+# TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1\. Definitions.
+
+"License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document.
+
+"Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License.
+
+"Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity.
+
+"You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License.
+
+"Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files.
+
+"Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types.
+
+"Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below).
+
+"Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof.
+
+"Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution."
+
+"Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work.
+
+2\. Grant of Copyright License.  
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form.
+
+3\. Grant of Patent License.  
+Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed.
+
+4\. Redistribution.  
+You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions:
+
+(a) You must give any other recipients of the Work or Derivative Works a copy of this License; and  
+(b) You must cause any modified files to carry prominent notices stating that You changed the files; and  
+(c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and  
+(d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License.  
+(e) If You distribute or make available a Derivative Work that is an artificial intelligence model created, trained, fine-tuned, or otherwise improved using the Work (the "Derivative AI Model"), You must adhere to the following conditions:  
+(i) The name of such Derivative AI Model must begin with "Solar" (e.g., "Solar-MyModel-v1"); and  
+(ii) You must prominently display the phrase "Built with Solar" in any related websites, user interfaces, or documentation associated with the Derivative AI Model; and  
+(iii) You must provide a copy of this License, including the original copyright notice and NOTICE file as included in the distribution of the Works, alongside the Derivative AI Model.
+
+You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License.
+
+5\. Submission of Contributions.  
+Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions.
+
+6\. Trademarks.  
+This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work, reproducing the content of the NOTICE file, and as explicitly required for attribution in Section 4(e) of this License.  
+Any use of the “Solar” name under this License must not imply any sponsorship, endorsement, certification, or official relationship with the Licensor, nor mislead users into believing that a Derivative AI Model is an official product of the Licensor.
+
+7\. Disclaimer of Warranty.  
+Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License.
+
+8\. Limitation of Liability.  
+In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages.
+
+9\. Accepting Warranty or Additional Liability.  
+While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+# APPENDIX: How to apply the Upstage Solar License to your work.
+
+To apply the Upstage Solar License to your work, attach the following boilerplate notice, with the fields enclosed by brackets replaced with your own identifying information. (Don't include the brackets\!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives.
+
+Copyright \[yyyy\] \[Upstage AI (or other copyright owner)\]
+
+Licensed under the Upstage Solar License (the "License");  
+you may not use this file except in compliance with the License.  
+You may obtain a copy of the License at
+
+   https://huggingface.co/Upstage/Solar-Open-100B/blob/main/LICENSE
+
+Unless required by applicable law or agreed to in writing, software  
+distributed under the License is distributed on an "AS IS" BASIS,  
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  
+See the License for the specific language governing permissions and  
+limitations under the License.
+
+This work is based on or derived from materials licensed under the  
+Upstage Solar License.  
+If you distribute or make available a Derivative AI Model (as defined  
+in Section 4(e) of the License), your model name must begin with "Solar"  
+and you must prominently display "Built with Solar" in associated  
+documentation or interfaces.  

README.md

@@ -1,5 +1,157 @@
----
-license: other
-license_name: upstage-solar-license
-license_link: LICENSE
----
+---
+language:
+- en
+- ko
+library_name: transformers
+license: other
+license_name: upstage-solar-license
+pipeline_tag: text-generation
+tags:
+- upstage
+- solar
+- moe
+- 100b
+- llm
+- nota
+- quantization
+---
+
+# **Solar-Open-100B-Nota-FP8**
+
+This repository provides an FP8-quantized version of **Upstage’s flagship model, [Solar-Open-100B](https://huggingface.co/upstage/Solar-Open-100B)**.
+
+## Overview
+
+- **Base model:** [Solar-Open-100B](https://huggingface.co/upstage/Solar-Open-100B) 
+- **Quantization:** FP8 (weight: per-channel/static, activation: per-token/dynamic)
+- **Hardware Requirements:** 
+    * **Minimum:** 2 x NVIDIA A100 (80GB)    
+
+## License
+This repository contains both model weights and code,
+which are licensed under different terms:
+
+1. MODEL WEIGHTS (*.safetensors)
+   Licensed under **Upstage Solar License**
+   See: https://huggingface.co/upstage/Solar-Open-100B/blob/main/LICENSE
+
+2. CODE (*.py, *.json, *.jinja files)
+   Licensed under **Apache License 2.0**
+   See: https://www.apache.org/licenses/LICENSE-2.0
+
+
+## Performance
+
+- English
+
+|                 |**Solar-Open-100B**|**Nota FP8 (Ours)**|
+|---              | ---               | ---               |
+|PPL (WikiText-2)↓|6.06               |6.06               |
+|PPL (C4)↓        |20.37              |20.62              |
+|PIQA↑            |82.37              |81.94              |
+|BoolQ↑           |84.89              |85.14              |
+|ARC-E↑           |87.25              |87.08              |
+|ARC-C↑           |61.43              |61.60              |
+|TruthfulQA↑      |59.25              |59.29              |
+|WinoGrande↑      |76.09              |76.01              |
+
+- Korean
+
+|                 |**Solar-Open-100B**|**Nota FP8 (Ours)**|
+|---              | ---               | ---               |
+|HRM8K↑           |81.52              |81.54              |
+|MMLU-ProX-Lite↑  |55.44              |54.86              |
+|KoBEST↑          |62.00              |62.00              |
+|CLiCK↑           |71.33              |71.28              |
+
+- Model weigth memory footprint
+
+|**Solar-Open-100B**|**Nota MoE Quantization (Ours)**|
+| ---               | ---                            |
+|191.2 GB           |97.2 GB                         |
+
+
+* Note 
+  - ↑ / ↓ denote the direction of improvement: higher is better (↑), lower is better (↓).
+  - Because we used a smaller thinking budget, the results for HRM8K and CLiCK are slightly lower than the numbers reported in the original Solar-Open-100B repository.
+  - Memory refers to the pure VRAM footprint occupied only by the model weights.
+
+## Inference
+### Transformers
+
+Install the required dependencies:
+
+```bash
+pip install -U transformers kernels torch accelerate
+```
+
+Run inference with the following code:
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+MODEL_ID = "nota-ai/Solar-Open-100B-Nota-FP8"
+
+# Load model and tokenizer
+tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
+
+model = AutoModelForCausalLM.from_pretrained(
+    pretrained_model_name_or_path=MODEL_ID,
+    torch_dtype=torch.bfloat16,
+    device_map="auto",
+    trust_remote_code=True,
+)
+
+# Prepare input
+messages = [{"role": "user", "content": "who are you?"}]
+inputs = tokenizer.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_dict=True,
+    return_tensors="pt",
+)
+inputs = inputs.to(model.device)
+
+# Generate response
+generated_ids = model.generate(
+    **inputs,
+    max_new_tokens=4096,
+    temperature=0.8,
+    top_p=0.95,
+    top_k=50,
+    do_sample=True,
+)
+generated_text = tokenizer.decode(generated_ids[0][inputs.input_ids.shape[1] :])
+print(generated_text)
+```
+
+### vLLM
+Create and activate a Python virtual environment
+```bash
+uv venv --python 3.12 --seed
+source .venv/bin/activate
+```
+
+Install Solar Open's optimized vLLM
+```bash
+VLLM_PRECOMPILED_WHEEL_LOCATION="https://github.com/vllm-project/vllm/releases/download/v0.12.0/vllm-0.12.0-cp38-abi3-manylinux_2_31_x86_64.whl" \
+VLLM_USE_PRECOMPILED=1 \
+uv pip install git+https://github.com/UpstageAI/vllm.git@v0.12.0-solar-open
+```
+
+Start the vLLM server (For 2 GPUs)
+```bash
+PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
+vllm serve nota-ai/Solar-Open-100B-Nota-FP8 \
+    --trust-remote-code \
+    --enable-auto-tool-choice \
+    --tool-call-parser solar_open \
+    --reasoning-parser solar_open \
+    --logits-processors vllm.model_executor.models.parallel_tool_call_logits_processor:ParallelToolCallLogitsProcessor \
+    --logits-processors vllm.model_executor.models.solar_open_logits_processor:SolarOpenTemplateLogitsProcessor \
+    --tensor-parallel-size 2 \
+    --max-num-seqs 64 \
+    --gpu-memory-utilization 0.8
+```

chat_template.jinja

@@ -0,0 +1,156 @@
+{#- ======== Template Parameters ========  #}
+{%- set add_generation_prompt = add_generation_prompt if add_generation_prompt is defined else true %}
+{%- set default_system_prompt = default_system_prompt if default_system_prompt is defined else true %}
+{%- set reasoning_effort = reasoning_effort if reasoning_effort is defined else "high" %}
+{%- set think_render_option = think_render_option if think_render_option is defined else "lastthink" %}
+
+{#- ======== System Block State ========  #}
+{%- set sys_ns = namespace(is_first_block=true) -%}
+
+{#- ======== Find last user message index ========  #}
+{%- set last_user_idx = namespace(value=-1) -%}
+{%- for message in messages -%}
+    {%- if message.role == 'user' -%}
+        {%- set last_user_idx.value = loop.index0 -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{#- ======== System messages renderers ========  #}
+{%- macro render_system_message(user_system_messages) %}
+    {%- if default_system_prompt %}
+        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+        {%- set sys_ns.is_first_block = false %}
+        {{- "## Provider System Prompt\n\nYou are Solar Open 100B, a large language model trained by Upstage AI, a Korean startup. Your knowledge cutoff is 2025-07. The current date is " + strftime_now("%Y-%m-%d") + "." }}
+    {%- endif -%}
+    {%- if user_system_messages %}
+        {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+        {%- set sys_ns.is_first_block = false %}
+        {{- "## System Prompt" }}
+        {%- for system_message in user_system_messages %}
+            {{- "\n\n" }}
+            {{- system_message }}
+        {%- endfor %}
+    {%- endif -%}
+{%- endmacro %}
+
+{%- macro render_tool_instruction(tools) %}
+    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+    {%- set sys_ns.is_first_block = false %}
+    {{- "## Tools\n\n### Tool Call Instruction" }}
+    {{- "\nYou may invoke one or more tools to assist with the user's query. Available tools are provided in JSON Schema format: <|tools:begin|><|tool:begin|><tools-json-object><|tool:end|>...<|tools:end|>\n" }}
+    {{- "\n### Available Tools\n" }}
+    {{- "<|tools:begin|>" }}
+    {%- for tool in tools %}
+        {{- "<|tool:begin|>" }}
+        {{- tool.function | tojson }}
+        {{- "<|tool:end|>" }}
+    {%- endfor %}
+    {{- "<|tools:end|>\n" }}
+    {{- "\n### Tool Call Format\n" }}
+    {{- "For each tool call, return a JSON object with the following structure, enclosed within <|tool_call:begin|> and <|tool_call:end|> tags: \n<|tool_call:begin|><tool-call-id><|tool_call:name|><tool-name><|tool_call:args|><args-json-object><|tool_call:end|>\n" }}
+    {{- "- The <tool-call-id> must be a randomly generated string consisting of 10 lowercase letters (a-z) and/or digits (0-9) (e.g., a1b2c3d4e5)\n" }}
+    {{- "\n### Tool Response Format\n" }}
+    {{- "Each tool is responded by `tool` with the following structure:\n<|tool_response:id|><tool-call-id><|tool_response:name|><tool-name><|tool_response:result|><results><|tool_response:end|>\n" }}
+    {{- "- Ensure the <tool-call-id> matches the corresponding tool call" -}}
+{%- endmacro %}
+
+{%- macro render_json_response_format_instruction(response_format) %}
+    {%- if not sys_ns.is_first_block %}{{- "\n\n" }}{%- endif %}
+    {%- set sys_ns.is_first_block = false %}
+    {{- "## Output Format Constraint" }}
+    {{- "\n\nYour final response should follow the JSON schema: \n[Start of schema]" }}
+    {{- response_format }}
+    {{- "\n[End of schema]\nPlease ensure your answers adhere to this format and do not contain any unnecessary text." }}
+{%- endmacro %}
+
+{%- macro get_tool_name(messages, tool_call_id) %}
+    {%- for msg in messages -%}
+        {%- if msg.role == 'assistant' and msg.tool_calls -%}
+            {%- for tool_call in msg.tool_calls -%}
+                {%- if tool_call.id == tool_call_id -%}
+                    {{- tool_call.function.name }}
+                {%- endif -%}
+            {%- endfor -%}
+        {%- endif -%}
+    {%- endfor -%}
+{%- endmacro %}
+
+{%- macro render_tool_arguments(tool_arguments) %}
+    {%- if tool_arguments is mapping -%}
+        {{- tool_arguments | tojson }}
+    {%- else -%}
+        {{- tool_arguments }}
+    {%- endif -%}
+{%- endmacro %}
+
+{#- ======== Render system message ========  #}
+{%- set ns = namespace(system_messages=[]) -%}
+{%- for message in messages -%}
+    {%- if message.role == 'system' -%}
+        {%- set ns.system_messages = ns.system_messages + [message.content] -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if ns.system_messages or default_system_prompt or tools or response_format -%}
+    {{- "<|begin|>system<|content|>" }}
+        {{- render_system_message(ns.system_messages) }}
+        {%- if tools -%}
+            {{- render_tool_instruction(tools) }}
+        {%- endif %}
+        {%- if response_format -%}
+            {{- render_json_response_format_instruction(response_format) }}
+        {%- endif %}
+    {{- "<|end|>" }}
+{%- endif -%}
+
+{#- ======== Render main messages ========  #}
+{%- for message in messages -%}
+    {%- if message.role == 'user' -%}
+         {{- "<|begin|>user<|content|>" + message.content + "<|end|>" }}
+    {%- elif message.role == 'tool' -%}
+        {%- set prev_is_tool = loop.index0 > 0 and messages[loop.index0 - 1].role == 'tool' -%}
+        {%- set next_is_tool = loop.index0 < (messages | length - 1) and messages[loop.index0 + 1].role == 'tool' -%}
+        {%- if not prev_is_tool -%}
+            {{- "<|begin|>tool<|tool_response|>" }}
+        {%- endif -%}
+        {{- "<|tool_response:begin|>" + message.tool_call_id + "<|tool_response:name|>" }}
+        {{- get_tool_name(messages, message.tool_call_id) }}
+        {{- "<|tool_response:result|>" }}
+        {{- message.content }}
+        {{- "<|tool_response:end|>" }}
+        {%- if not next_is_tool -%}
+            {{- "<|end|>" }}
+        {%- endif -%}
+    {%- elif message.role == 'assistant' -%}
+        {#- ======== Assistant Thinking ========  #}
+        {%- if think_render_option == "all" -%}
+            {%- if message.reasoning -%}
+                {{- "<|begin|>assistant<|think|>" + message.reasoning + "<|end|>" }}
+            {%- endif -%}
+        {%- elif think_render_option == "lastthink" -%}
+            {%- if message.reasoning and loop.index0 > last_user_idx.value -%}
+                {{- "<|begin|>assistant<|think|>" + message.reasoning + "<|end|>" }}
+            {%- endif -%}
+        {%- endif -%}
+
+        {#- ======== Assistant Messages ========  #}
+        {%- if message.tool_calls -%}
+            {{- "<|begin|>assistant<|tool_calls|>" }}
+            {%- for tool_call in message.tool_calls -%}
+                {{- "<|tool_call:begin|>" + tool_call.id +"<|tool_call:name|>" + tool_call.function.name + "<|tool_call:args|>" }}
+                {{- render_tool_arguments(tool_call.function.arguments) }}
+                {{- "<|tool_call:end|>" }}
+            {%- endfor -%}
+            {{- "<|calls|>" }}
+        {%- else -%}
+            {{- "<|begin|>assistant<|content|>" + message.content + "<|end|>" }}
+        {%- endif -%}
+    {%- endif -%}
+{%- endfor -%}
+
+{%- if add_generation_prompt -%}
+    {%- if reasoning_effort in ["low", "minimal"] -%}
+        {{- "<|begin|>assistant<|think|><|end|>" }}
+    {%- endif -%}
+    {{- "<|begin|>assistant" }}
+{%- endif -%}

config.json

@@ -0,0 +1,95 @@
+{
+  "architectures": [
+    "SolarOpenForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "auto_map": {
+    "AutoConfig": "configuration_solar_open.SolarOpenConfig",
+    "AutoModel": "modeling_solar_open.SolarOpenModel",
+    "AutoModelForCausalLM": "modeling_solar_open.SolarOpenForCausalLM"
+  },
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "first_k_dense_replace": 0,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 10240,
+  "max_position_embeddings": 131072,
+  "model_type": "solar_open",
+  "moe_intermediate_size": 1280,
+  "n_group": 1,
+  "n_routed_experts": 128,
+  "n_shared_experts": 1,
+  "norm_topk_prob": true,
+  "num_attention_heads": 64,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 48,
+  "num_key_value_heads": 8,
+  "pad_token_id": 2,
+  "partial_rotary_factor": 1.0,
+  "quantization_config": {
+    "config_groups": {
+      "group_0": {
+        "format": "float-quantized",
+        "input_activations": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": true,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": null,
+          "observer_kwargs": {},
+          "strategy": "token",
+          "symmetric": true,
+          "type": "float"
+        },
+        "output_activations": null,
+        "targets": [
+          "Linear"
+        ],
+        "weights": {
+          "actorder": null,
+          "block_structure": null,
+          "dynamic": false,
+          "group_size": null,
+          "num_bits": 8,
+          "observer": "minmax",
+          "observer_kwargs": {},
+          "strategy": "channel",
+          "symmetric": true,
+          "type": "float"
+        }
+      }
+    },
+    "format": "float-quantized",
+    "global_compression_ratio": null,
+    "ignore": [
+      "lm_head"
+    ],
+    "kv_cache_scheme": null,
+    "quant_method": "compressed-tensors",
+    "quantization_status": "compressed",
+    "sparsity_config": {},
+    "transform_config": {},
+    "version": "0.12.2"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": {
+    "factor": 2.0,
+    "original_max_position_embeddings": 65536,
+    "rope_type": "yarn",
+    "type": "yarn"
+  },
+  "rope_theta": 1000000,
+  "routed_scaling_factor": 1.0,
+  "tie_word_embeddings": false,
+  "topk_group": 1,
+  "transformers_version": "4.57.3",
+  "use_cache": true,
+  "use_qk_norm": false,
+  "vocab_size": 196608
+}

configuration_solar_open.py

@@ -0,0 +1,242 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+# Copyright 2025 The ZhipuAI Inc. team and HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file has been modified by Upstage AI including
+# -  Hyperparameter Adjustments: Modified the model architecture by increasing vocab_size and num_hidden_layers, while decreasing num_attention_heads, intermediate_size, and moe_intermediate_size.
+# RoPE Configuration: Replaced the generic rope_parameters argument with explicit rope_theta and rope_scaling parameters to define Rotary Positional Embeddings settings.
+#
+# Based on code from: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4_moe/configuration_glm4_moe.py
+
+from transformers.configuration_utils import PretrainedConfig
+from transformers.modeling_rope_utils import rope_config_validation
+
+
+class SolarOpenConfig(PretrainedConfig):
+    r"""
+    This is the configuration class to store the configuration of a [`SolarOpenModel`]. It is used to instantiate a
+    SolarOpen model according to the specified arguments, defining the model architecture.
+
+    Configuration objects inherit from [`PretrainedConfig`] and can be used to control the model outputs. Read the
+    documentation from [`PretrainedConfig`] for more information.
+
+
+    Args:
+        vocab_size (`int`, *optional*, defaults to 196608):
+            Vocabulary size of the SolarOpen model. Defines the number of different tokens that can be represented by the
+            `inputs_ids` passed when calling [`SolarOpenModel`]
+        hidden_size (`int`, *optional*, defaults to 4096):
+            Dimension of the hidden representations.
+        intermediate_size (`int`, *optional*, defaults to 10240):
+            Dimension of the MLP representations.
+        num_hidden_layers (`int`, *optional*, defaults to 48):
+            Number of hidden layers in the Transformer encoder.
+        num_attention_heads (`int`, *optional*, defaults to 64):
+            Number of attention heads for each attention layer in the Transformer encoder.
+        partial_rotary_factor (`float`, *optional*, defaults to 1.0):
+            The factor of the partial rotary position.
+        num_key_value_heads (`int`, *optional*, defaults to 8):
+            This is the number of key_value heads that should be used to implement Grouped Query Attention. If
+            `num_key_value_heads=num_attention_heads`, the model will use Multi Head Attention (MHA), if
+            `num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When
+            converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed
+            by meanpooling all the original heads within that group. For more details, check out [this
+            paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `32`.
+
+        hidden_act (`str` or `function`, *optional*, defaults to `"silu"`):
+            The non-linear activation function (function or string) in the decoder.
+        max_position_embeddings (`int`, *optional*, defaults to 131072):
+            The maximum sequence length that this model might ever be used with.
+        initializer_range (`float`, *optional*, defaults to 0.02):
+            The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
+        rms_norm_eps (`float`, *optional*, defaults to 1e-05):
+            The epsilon used by the rms normalization layers.
+        use_cache (`bool`, *optional*, defaults to `True`):
+            Whether or not the model should return the last key/values attentions (not used by all models). Only
+            relevant if `config.is_decoder=True`.
+        tie_word_embeddings (`bool`, *optional*, defaults to `False`):
+            Whether the model's input and output word embeddings should be tied.
+        rope_theta (`float`, *optional*, defaults to 1000000.0):
+            The base period of the RoPE embeddings.
+        rope_scaling (`Dict`, *optional*):
+            Dictionary containing the scaling configuration for the RoPE embeddings. NOTE: if you apply new rope type
+            and you expect the model to work on longer `max_position_embeddings`, we recommend you to update this value
+            accordingly.
+            Expected contents:
+                `rope_type` (`str`):
+                    The sub-variant of RoPE to use. Can be one of ['default', 'linear', 'dynamic', 'yarn', 'longrope',
+                    'llama3'], with 'default' being the original RoPE implementation.
+                `factor` (`float`, *optional*):
+                    Used with all rope types except 'default'. The scaling factor to apply to the RoPE embeddings. In
+                    most scaling types, a `factor` of x will enable the model to handle sequences of length x *
+                    original maximum pre-trained length.
+                `original_max_position_embeddings` (`int`, *optional*):
+                    Used with 'dynamic', 'longrope' and 'llama3'. The original max position embeddings used during
+                    pretraining.
+                `attention_factor` (`float`, *optional*):
+                    Used with 'yarn' and 'longrope'. The scaling factor to be applied on the attention
+                    computation. If unspecified, it defaults to value recommended by the implementation, using the
+                    `factor` field to infer the suggested value.
+                `beta_fast` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for extrapolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 32.
+                `beta_slow` (`float`, *optional*):
+                    Only used with 'yarn'. Parameter to set the boundary for interpolation (only) in the linear
+                    ramp function. If unspecified, it defaults to 1.
+                `short_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to short contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `long_factor` (`list[float]`, *optional*):
+                    Only used with 'longrope'. The scaling factor to be applied to long contexts (<
+                    `original_max_position_embeddings`). Must be a list of numbers with the same length as the hidden
+                    size divided by the number of attention heads divided by 2
+                `low_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to low frequency components of the RoPE
+                `high_freq_factor` (`float`, *optional*):
+                    Only used with 'llama3'. Scaling factor applied to high frequency components of the RoPE
+        attention_bias (`bool`, defaults to `False`, *optional*, defaults to `False`):
+            Whether to use a bias in the query, key, value and output projection layers during self-attention.
+        attention_dropout (`float`, *optional*, defaults to 0.0):
+            The dropout ratio for the attention probabilities.
+        moe_intermediate_size (`int`, *optional*, defaults to 1280):
+            Intermediate size of the routed expert.
+        num_experts_per_tok (`int`, *optional*, defaults to 8):
+            number of experts per token.
+        n_shared_experts (`int`, *optional*, defaults to 1):
+            Number of shared experts.
+        n_routed_experts (`int`, *optional*, defaults to 128):
+            Number of routed experts.
+        routed_scaling_factor (`float`, *optional*, defaults to 1.0):
+            Scaling factor or routed experts.
+        n_group (`int`, *optional*, defaults to 1):
+            Number of groups for routed experts.
+        topk_group (`int`, *optional*, defaults to 1):
+            Number of selected groups for each token(for each token, ensuring the selected experts is only within `topk_group` groups).
+        first_k_dense_replace (`int`, *optional*, defaults to 0):
+            Number of dense layers in shallow layers(embed->dense->dense->...->dense->moe->moe...->lm_head).
+                                                            \--k dense layers--/
+        norm_topk_prob (`bool`, *optional*, defaults to `True`):
+            Whether to normalize the topk probabilities.
+        use_qk_norm (`bool`, *optional*, defaults to `False`):
+            Whether to use query-key normalization in the attention
+    ```python
+    >>> from transformers import SolarOpenModel, SolarOpenConfig
+
+    >>> # Initializing a SolarOpen style configuration
+    >>> configuration = SolarOpenConfig()
+
+    >>> # Initializing a model from the SolarOpen style configuration
+    >>> model = SolarOpenModel(configuration)
+
+    >>> # Accessing the model configuration
+    >>> configuration = model.config
+    ```"""
+
+    model_type = "solar_open"
+    keys_to_ignore_at_inference = ["past_key_values"]
+
+    # Default tensor parallel plan for base model `SolarOpen`
+    base_model_tp_plan = {
+        "layers.*.self_attn.q_proj": "colwise",
+        "layers.*.self_attn.k_proj": "colwise",
+        "layers.*.self_attn.v_proj": "colwise",
+        "layers.*.self_attn.o_proj": "rowwise",
+        "layers.*.mlp.experts.*.gate_proj": "colwise",
+        "layers.*.mlp.experts.*.up_proj": "colwise",
+        "layers.*.mlp.experts.*.down_proj": "rowwise",
+        "layers.*.mlp.gate_proj": "colwise",
+        "layers.*.mlp.up_proj": "colwise",
+        "layers.*.mlp.down_proj": "rowwise",
+    }
+    base_model_pp_plan = {
+        "embed_tokens": (["input_ids"], ["inputs_embeds"]),
+        "layers": (["hidden_states", "attention_mask"], ["hidden_states"]),
+        "norm": (["hidden_states"], ["hidden_states"]),
+    }
+
+    def __init__(
+        self,
+        vocab_size=196608,
+        hidden_size=4096,
+        intermediate_size=10240,
+        num_hidden_layers=48,
+        num_attention_heads=64,
+        partial_rotary_factor=1.0,
+        num_key_value_heads=8,
+        hidden_act="silu",
+        max_position_embeddings=131072,
+        initializer_range=0.02,
+        rms_norm_eps=1e-5,
+        use_cache=True,
+        tie_word_embeddings=False,
+        rope_theta=1000000.0,
+        rope_scaling=None,
+        attention_bias=False,
+        attention_dropout=0.0,
+        moe_intermediate_size=1280,
+        num_experts_per_tok=8,
+        n_shared_experts=1,
+        n_routed_experts=128,
+        routed_scaling_factor=1.0,
+        n_group=1,
+        topk_group=1,
+        first_k_dense_replace=0,
+        norm_topk_prob=True,
+        use_qk_norm=False,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.max_position_embeddings = max_position_embeddings
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.partial_rotary_factor = partial_rotary_factor
+
+        self.num_key_value_heads = num_key_value_heads
+        self.hidden_act = hidden_act
+        self.initializer_range = initializer_range
+        self.rms_norm_eps = rms_norm_eps
+        self.use_cache = use_cache
+        self.rope_theta = rope_theta
+        self.rope_scaling = rope_scaling
+        self.attention_bias = attention_bias
+        self.attention_dropout = attention_dropout
+        # Validate the correctness of rotary position embeddings parameters
+        # BC: if there is a 'type' field, move it to 'rope_type'.
+        if self.rope_scaling is not None and "type" in self.rope_scaling:
+            self.rope_scaling["rope_type"] = self.rope_scaling["type"]
+        rope_config_validation(self)
+
+        # MoE arguments
+        self.moe_intermediate_size = moe_intermediate_size
+        self.num_experts_per_tok = num_experts_per_tok
+        self.n_group = n_group
+        self.topk_group = topk_group
+        self.n_shared_experts = n_shared_experts
+        self.n_routed_experts = n_routed_experts
+        self.routed_scaling_factor = routed_scaling_factor
+        self.first_k_dense_replace = first_k_dense_replace
+        self.norm_topk_prob = norm_topk_prob
+        self.use_qk_norm = use_qk_norm
+
+        super().__init__(
+            tie_word_embeddings=tie_word_embeddings,
+            **kwargs,
+        )
+
+
+__all__ = ["SolarOpenConfig"]

generation_config.json

@@ -0,0 +1,14 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "do_sample": true,
+  "eos_token_id": [
+    2,
+    24,
+    25
+  ],
+  "pad_token_id": 2,
+  "temperature": 0.8,
+  "top_p": 0.95,
+  "transformers_version": "4.57.3"
+}

model-00001-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9586b3be358f339676b372ed5f4289ff39380bbf92cb56dd0fef10242bcfaf60
+size 4995235552

model-00002-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1e9365d8a28629c2dee00b2c6317f0c41d9a09268ac0082989058728ff631e7
+size 4996690936

model-00003-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e82fd02c72f5eada7fe9ebbb98968f8d68790b3d3dfd43b63f76b072f5c85a59
+size 4999816264

model-00004-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7fa5fb7f30bfd8c8c53ff1b1864ca2ad70e8fbb952192f08a5d8f52b00c97d2a
+size 4996685016

model-00005-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f3ced072884417a61020aa20562bbfafb9661a39e45f948dcb014891e9db12d0
+size 4999816816

model-00006-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:13614da79690c9b219215ab82d31652e1f226f5e4a3feaed4151d17326453df1
+size 4996692512

model-00007-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4f1faa47c5e5ac2fee9e46a8f4aeb45dc42d88117eb2ff569443a15c3d6491b8
+size 4996692584

model-00008-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:efcfa17725c41cdfad05ae38375a16c8192f2732e4f746740b96a75e85f78a8d
+size 4999818096

model-00009-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3ef73bb9f0eb35a908b5b63159bae79d7d44cb3b02fa242e8c789eaf1b54d8dc
+size 4996686888

model-00010-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:276d8872f6cae6aff5eb4f7de66e23d750960c6be9945d7850f6413118b8d2af
+size 4996692768

model-00011-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:332accec25bc4250d1f23bf36888fb8a9829e8324fcbf66f38ad2f9806a64ca5
+size 4999818096

model-00012-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24d253fe8393b739286dfd383e5a9a60bdf07e5b6abe01f868bd5b2e6d67de19
+size 4996692520

model-00013-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f737fd8f8f4ae88e1f2cf0b990ed3c33d83b5302d81cf59f34c6fea2ab009a7a
+size 4999818056

model-00014-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:219d5cc7613b882d3ba9b94de2ff3eb0d30c9080fb22974779ceb4b743ae0313
+size 4996686848

model-00015-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ce00ebf3c5b756f7acc00be9449f6f1e3bf8bb35deb54717891ff50d55dcefdf
+size 4996692544

model-00016-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bb7820a429541a95fd49e62a9f52f81632a655eb0ea71a86e44746eb13b99a32
+size 4999818096

model-00017-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7c852d36b2e6a22d7241330d078bb352bf4599706a3bf03a6b0486cf6ba481b5
+size 4996692520

model-00018-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1a92366f4492dbcf37fb26558794a1da97683af125c90f9762ef2e754a186b5d
+size 4996687088

model-00019-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74cfe23838f84c6214f7565404dc2b93b5c4d1d81306a7ba4aaf95cacb9fa8bf
+size 4999818096

model-00020-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f60372452506d9f675834b2a538151d142e95a8e63920064224b2b55c5f15572
+size 4996692520

model-00021-of-00021.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e853d76650d786c88cd4baa699e2095191d022de358965409ef8c62f9e17be14
+size 4422268360

modeling_solar_open.py

@@ -0,0 +1,605 @@
+# coding=utf-8
+# Copyright 2025 Upstage AI.
+# Copyright 2025 The GLM4 & ZhipuAI team and HuggingFace Inc. team.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# This file has been modified by Upstage AI including:
+# - Hybrid MoE Architecture: Replaced the standard dense structure with a depth-dependent Hybrid MoE, adding `SolarOpenMoE` and `SolarOpenTopkRouter` classes.
+# - RoPE Strategy: Changed the rotary position embedding strategy from GLM4's interleaved rotation to Llama-style block rotation (via modified `rotate_half`).
+# - Normalization Logic: Simplified the layer normalization structure by removing GLM4's extra post-operation norms and adding optional Query-Key Normalization (`use_qk_norm`).
+#
+# Based on code from: https://github.com/huggingface/transformers/blob/main/src/transformers/models/glm4/modeling_glm4.py
+
+from typing import Callable, Optional, Union
+
+import torch
+import torch.nn.functional as F
+from torch import nn
+
+from transformers.activations import ACT2FN
+from transformers.cache_utils import Cache, DynamicCache
+from transformers.generation import GenerationMixin
+from transformers.integrations import use_kernel_forward_from_hub
+from transformers.masking_utils import create_causal_mask
+from transformers.modeling_flash_attention_utils import FlashAttentionKwargs
+from transformers.modeling_layers import GradientCheckpointingLayer
+from transformers.modeling_outputs import BaseModelOutputWithPast, CausalLMOutputWithPast
+from transformers.modeling_rope_utils import ROPE_INIT_FUNCTIONS, dynamic_rope_update
+from transformers.modeling_utils import ALL_ATTENTION_FUNCTIONS, PreTrainedModel
+from transformers.processing_utils import Unpack
+from transformers.utils import TransformersKwargs, auto_docstring, can_return_tuple
+from transformers.utils.deprecation import deprecate_kwarg
+from transformers.utils.generic import check_model_inputs
+from .configuration_solar_open import SolarOpenConfig
+
+
+def repeat_kv(hidden_states: torch.Tensor, n_rep: int) -> torch.Tensor:
+    """
+    This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch,
+    num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim)
+    """
+    batch, num_key_value_heads, slen, head_dim = hidden_states.shape
+    if n_rep == 1:
+        return hidden_states
+    hidden_states = hidden_states[:, :, None, :, :].expand(batch, num_key_value_heads, n_rep, slen, head_dim)
+    return hidden_states.reshape(batch, num_key_value_heads * n_rep, slen, head_dim)
+
+
+def eager_attention_forward(
+    module: nn.Module,
+    query: torch.Tensor,
+    key: torch.Tensor,
+    value: torch.Tensor,
+    attention_mask: Optional[torch.Tensor],
+    scaling: float,
+    dropout: float = 0.0,
+    **kwargs: Unpack[TransformersKwargs],
+):
+    key_states = repeat_kv(key, module.num_key_value_groups)
+    value_states = repeat_kv(value, module.num_key_value_groups)
+
+    attn_weights = torch.matmul(query, key_states.transpose(2, 3)) * scaling
+    if attention_mask is not None:
+        causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
+        attn_weights = attn_weights + causal_mask
+
+    attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query.dtype)
+    attn_weights = nn.functional.dropout(attn_weights, p=dropout, training=module.training)
+    attn_output = torch.matmul(attn_weights, value_states)
+    attn_output = attn_output.transpose(1, 2).contiguous()
+
+    return attn_output, attn_weights
+
+
+def rotate_half(x):
+    """Rotates half the hidden dims of the input."""
+    x1 = x[..., : x.shape[-1] // 2]
+    x2 = x[..., x.shape[-1] // 2 :]
+    return torch.cat((-x2, x1), dim=-1)
+
+
+def apply_rotary_pos_emb(q, k, cos, sin, position_ids=None, unsqueeze_dim=1):
+    """Applies Rotary Position Embedding to the query and key tensors.
+
+    Args:
+        q (`torch.Tensor`): The query tensor.
+        k (`torch.Tensor`): The key tensor.
+        cos (`torch.Tensor`): The cosine part of the rotary embedding.
+        sin (`torch.Tensor`): The sine part of the rotary embedding.
+        position_ids (`torch.Tensor`, *optional*):
+            Deprecated and unused.
+        unsqueeze_dim (`int`, *optional*, defaults to 1):
+            The 'unsqueeze_dim' argument specifies the dimension along which to unsqueeze cos[position_ids] and
+            sin[position_ids] so that they can be properly broadcasted to the dimensions of q and k. For example, note
+            that cos[position_ids] and sin[position_ids] have the shape [batch_size, seq_len, head_dim]. Then, if q and
+            k have the shape [batch_size, heads, seq_len, head_dim], then setting unsqueeze_dim=1 makes
+            cos[position_ids] and sin[position_ids] broadcastable to the shapes of q and k. Similarly, if q and k have
+            the shape [batch_size, seq_len, heads, head_dim], then set unsqueeze_dim=2.
+    Returns:
+        `tuple(torch.Tensor)` comprising of the query and key tensors rotated using the Rotary Position Embedding.
+    """
+    cos = cos.unsqueeze(unsqueeze_dim)
+    sin = sin.unsqueeze(unsqueeze_dim)
+
+    # Keep half or full tensor for later concatenation
+    rotary_dim = cos.shape[-1]
+    q_rot, q_pass = q[..., :rotary_dim], q[..., rotary_dim:]
+    k_rot, k_pass = k[..., :rotary_dim], k[..., rotary_dim:]
+
+    # Apply rotary embeddings on the first half or full tensor
+    q_embed = (q_rot * cos) + (rotate_half(q_rot) * sin)
+    k_embed = (k_rot * cos) + (rotate_half(k_rot) * sin)
+
+    # Concatenate back to full shape
+    q_embed = torch.cat([q_embed, q_pass], dim=-1)
+    k_embed = torch.cat([k_embed, k_pass], dim=-1)
+    return q_embed, k_embed
+
+
+class SolarOpenAttention(nn.Module):
+    """Multi-headed attention from 'Attention Is All You Need' paper"""
+
+    def __init__(self, config: SolarOpenConfig, layer_idx: Optional[int] = None):
+        super().__init__()
+        self.config = config
+        self.layer_idx = layer_idx
+        self.head_dim = getattr(config, "head_dim", config.hidden_size // config.num_attention_heads)
+        self.num_key_value_groups = config.num_attention_heads // config.num_key_value_heads
+        self.scaling = self.head_dim**-0.5
+        self.rope_scaling = config.rope_scaling
+        self.attention_dropout = config.attention_dropout
+        self.is_causal = True
+
+        self.q_proj = nn.Linear(
+            config.hidden_size, config.num_attention_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.k_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.v_proj = nn.Linear(
+            config.hidden_size, config.num_key_value_heads * self.head_dim, bias=config.attention_bias
+        )
+        self.o_proj = nn.Linear(config.num_attention_heads * self.head_dim, config.hidden_size, bias=False)
+        self.use_qk_norm = config.use_qk_norm
+        if self.use_qk_norm:
+            self.q_norm = SolarOpenRMSNorm(self.head_dim, eps=config.rms_norm_eps)
+            self.k_norm = SolarOpenRMSNorm(self.head_dim, eps=config.rms_norm_eps)
+
+    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        position_embeddings: tuple[torch.Tensor, torch.Tensor],
+        attention_mask: Optional[torch.Tensor],
+        past_key_values: Optional[Cache] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        **kwargs: Unpack[FlashAttentionKwargs],
+    ) -> tuple[torch.Tensor, Optional[torch.Tensor]]:
+        input_shape = hidden_states.shape[:-1]
+        hidden_shape = (*input_shape, -1, self.head_dim)
+
+        query_states = self.q_proj(hidden_states).view(hidden_shape)
+        key_states = self.k_proj(hidden_states).view(hidden_shape)
+        value_states = self.v_proj(hidden_states).view(hidden_shape)
+
+        if self.use_qk_norm:  # main diff from Llama
+            query_states = self.q_norm(query_states)
+            key_states = self.k_norm(key_states)
+
+        query_states = query_states.transpose(1, 2)
+        key_states = key_states.transpose(1, 2)
+        value_states = value_states.transpose(1, 2)
+
+        cos, sin = position_embeddings
+        query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin)
+
+        if past_key_values is not None:
+            # sin and cos are specific to RoPE models; position_ids needed for the static cache
+            cache_kwargs = {"sin": sin, "cos": cos, "cache_position": cache_position}
+            key_states, value_states = past_key_values.update(key_states, value_states, self.layer_idx, cache_kwargs)
+
+        attention_interface: Callable = eager_attention_forward
+        if self.config._attn_implementation != "eager":
+            attention_interface = ALL_ATTENTION_FUNCTIONS[self.config._attn_implementation]
+
+        attn_output, attn_weights = attention_interface(
+            self,
+            query_states,
+            key_states,
+            value_states,
+            attention_mask,
+            dropout=0.0 if not self.training else self.attention_dropout,
+            scaling=self.scaling,
+            **kwargs,
+        )
+
+        attn_output = attn_output.reshape(*input_shape, -1).contiguous()
+        attn_output = self.o_proj(attn_output)
+        return attn_output, attn_weights
+
+
+class SolarOpenMLP(nn.Module):
+    def __init__(self, config, hidden_size=None, intermediate_size=None):
+        super().__init__()
+        self.config = config
+        self.hidden_size = config.hidden_size if hidden_size is None else hidden_size
+        self.intermediate_size = config.intermediate_size if intermediate_size is None else intermediate_size
+
+        self.gate_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(self.hidden_size, self.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(self.intermediate_size, self.hidden_size, bias=False)
+        self.act_fn = ACT2FN[config.hidden_act]
+
+    def forward(self, x):
+        down_proj = self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))
+        return down_proj
+
+
+class SolarOpenTopkRouter(nn.Module):
+    def __init__(self, config: SolarOpenConfig):
+        super().__init__()
+        self.config = config
+        self.top_k = config.num_experts_per_tok
+        self.n_routed_experts = config.n_routed_experts
+        self.routed_scaling_factor = config.routed_scaling_factor
+        self.n_group = config.n_group
+        self.topk_group = config.topk_group
+        self.norm_topk_prob = config.norm_topk_prob
+
+        self.weight = nn.Parameter(torch.empty((self.n_routed_experts, config.hidden_size)))
+        self.e_score_correction_bias = nn.Parameter(
+            torch.zeros((self.n_routed_experts), dtype=torch.float32))
+
+    @torch.no_grad()
+    def get_topk_indices(self, scores):
+        scores_for_choice = scores.view(-1, self.n_routed_experts) + self.e_score_correction_bias.unsqueeze(0)
+        group_scores = (
+            scores_for_choice.view(-1, self.n_group, self.n_routed_experts // self.n_group)
+            .topk(2, dim=-1)[0]
+            .sum(dim=-1)
+        )
+        group_idx = torch.topk(group_scores, k=self.topk_group, dim=-1, sorted=False)[1]
+        group_mask = torch.zeros_like(group_scores)
+        group_mask.scatter_(1, group_idx, 1)
+        score_mask = (
+            group_mask.unsqueeze(-1)
+            .expand(-1, self.n_group, self.n_routed_experts // self.n_group)
+            .reshape(-1, self.n_routed_experts)
+        )
+        scores_for_choice = scores_for_choice.masked_fill(~score_mask.bool(), 0.0)
+        topk_indices = torch.topk(scores_for_choice, k=self.top_k, dim=-1, sorted=False)[1]
+        return topk_indices
+
+    def forward(self, hidden_states):
+        hidden_states = hidden_states.view(-1, self.config.hidden_size)
+        router_logits = F.linear(hidden_states.type(torch.float32), self.weight.type(torch.float32))
+        scores = router_logits.sigmoid()
+        topk_indices = self.get_topk_indices(scores)
+        topk_weights = scores.gather(1, topk_indices)
+        if self.norm_topk_prob:
+            denominator = topk_weights.sum(dim=-1, keepdim=True) + 1e-20
+            topk_weights /= denominator
+        topk_weights = topk_weights * self.routed_scaling_factor
+        return topk_indices, topk_weights
+
+
+@use_kernel_forward_from_hub("RMSNorm")
+class SolarOpenRMSNorm(nn.Module):
+    def __init__(self, hidden_size, eps=1e-6):
+        """
+        SolarOpenRMSNorm is equivalent to T5LayerNorm
+        """
+        super().__init__()
+        self.weight = nn.Parameter(torch.ones(hidden_size))
+        self.variance_epsilon = eps
+
+    def forward(self, hidden_states):
+        input_dtype = hidden_states.dtype
+        hidden_states = hidden_states.to(torch.float32)
+        variance = hidden_states.pow(2).mean(-1, keepdim=True)
+        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
+        return self.weight * hidden_states.to(input_dtype)
+
+    def extra_repr(self):
+        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
+
+
+class SolarOpenMoE(nn.Module):
+    """
+    A mixed expert module containing shared experts.
+    """
+
+    def __init__(self, config):
+        super().__init__()
+        self.config = config
+        self.experts = nn.ModuleList(
+            [
+                SolarOpenMLP(config, intermediate_size=config.moe_intermediate_size)
+                for _ in range(config.n_routed_experts)
+            ]
+        )
+        self.gate = SolarOpenTopkRouter(config)
+        self.shared_experts = SolarOpenMLP(
+            config=config, intermediate_size=config.moe_intermediate_size * config.n_shared_experts
+        )
+
+    @torch.compiler.disable()
+    def moe(self, hidden_states: torch.Tensor, topk_indices: torch.Tensor, topk_weights: torch.Tensor):
+        r"""
+        MoE forward pass that only executes selected experts.
+        Uses @torch.compiler.disable() to allow dynamic shape operations.
+        Requires --enforce-eager flag when serving with vLLM.
+        """
+        final_hidden_states = torch.zeros_like(hidden_states)
+
+        for expert_idx in range(len(self.experts)):
+            expert = self.experts[expert_idx]
+
+            # Find positions where this expert was selected
+            batch_idx, topk_pos = torch.where(topk_indices == expert_idx)
+
+            if batch_idx.numel() == 0:
+                continue
+
+            # Extract only the tokens routed to this expert
+            expert_input = hidden_states[batch_idx]
+            expert_output = expert(expert_input)
+
+            # Apply weights and accumulate results
+            weights = topk_weights[batch_idx, topk_pos].unsqueeze(-1)
+            final_hidden_states.index_add_(0, batch_idx, (expert_output * weights).to(hidden_states.dtype))
+
+        return final_hidden_states
+
+    def forward(self, hidden_states):
+        residuals = hidden_states
+        orig_shape = hidden_states.shape
+        topk_indices, topk_weights = self.gate(hidden_states)
+        hidden_states = hidden_states.view(-1, hidden_states.shape[-1])
+        hidden_states = self.moe(hidden_states, topk_indices, topk_weights).view(*orig_shape)
+        hidden_states = hidden_states + self.shared_experts(residuals)
+        return hidden_states
+
+
+class SolarOpenDecoderLayer(GradientCheckpointingLayer):
+    def __init__(self, config: SolarOpenConfig, layer_idx: int):
+        super().__init__()
+        self.hidden_size = config.hidden_size
+
+        self.self_attn = SolarOpenAttention(config=config, layer_idx=layer_idx)
+
+        if layer_idx >= config.first_k_dense_replace:
+            self.mlp = SolarOpenMoE(config)
+        else:
+            self.mlp = SolarOpenMLP(config)
+
+        self.input_layernorm = SolarOpenRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.post_attention_layernorm = SolarOpenRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+
+    @deprecate_kwarg("past_key_value", new_name="past_key_values", version="4.58")
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        use_cache: Optional[bool] = False,
+        cache_position: Optional[torch.LongTensor] = None,
+        position_embeddings: Optional[tuple[torch.Tensor, torch.Tensor]] = None,  # necessary, but kept here for BC
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> torch.Tensor:
+        residual = hidden_states
+        hidden_states = self.input_layernorm(hidden_states)
+        # Self Attention
+        hidden_states, _ = self.self_attn(
+            hidden_states=hidden_states,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            use_cache=use_cache,
+            cache_position=cache_position,
+            position_embeddings=position_embeddings,
+            **kwargs,
+        )
+        hidden_states = residual + hidden_states
+
+        # Fully Connected
+        residual = hidden_states
+        hidden_states = self.post_attention_layernorm(hidden_states)
+        hidden_states = self.mlp(hidden_states)
+        hidden_states = residual + hidden_states
+        return hidden_states
+
+
+@auto_docstring
+class SolarOpenPreTrainedModel(PreTrainedModel):
+    config: SolarOpenConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = True
+    _no_split_modules = ["SolarOpenDecoderLayer"]
+    _skip_keys_device_placement = ["past_key_values"]
+    _supports_flash_attn = True
+    _supports_sdpa = True
+    _supports_flex_attn = True
+    _can_compile_fullgraph = False
+    _supports_attention_backend = True
+    _can_record_outputs = {
+        "hidden_states": SolarOpenDecoderLayer,
+        "attentions": SolarOpenAttention,
+    }
+
+    def _init_weights(self, module):
+        super()._init_weights(module)
+        if isinstance(module, SolarOpenTopkRouter):
+            module.weight.data.normal_(mean=0.0, std=self.config.initializer_range)
+
+
+class SolarOpenRotaryEmbedding(nn.Module):
+    inv_freq: torch.Tensor  # fix linting for `register_buffer`
+
+    def __init__(self, config: SolarOpenConfig, device=None):
+        super().__init__()
+        # BC: "rope_type" was originally "type"
+        if hasattr(config, "rope_scaling") and isinstance(config.rope_scaling, dict):
+            self.rope_type = config.rope_scaling.get("rope_type", config.rope_scaling.get("type"))
+        else:
+            self.rope_type = "default"
+        self.max_seq_len_cached = config.max_position_embeddings
+        self.original_max_seq_len = config.max_position_embeddings
+
+        self.config = config
+        self.rope_init_fn = ROPE_INIT_FUNCTIONS[self.rope_type]
+
+        inv_freq, self.attention_scaling = self.rope_init_fn(self.config, device)
+        self.register_buffer("inv_freq", inv_freq, persistent=False)
+        self.original_inv_freq = self.inv_freq
+
+    @torch.no_grad()
+    @dynamic_rope_update  # power user: used with advanced RoPE types (e.g. dynamic rope)
+    def forward(self, x, position_ids):
+        inv_freq_expanded = self.inv_freq[None, :, None].float().expand(position_ids.shape[0], -1, 1).to(x.device)
+        position_ids_expanded = position_ids[:, None, :].float()
+
+        device_type = x.device.type if isinstance(x.device.type, str) and x.device.type != "mps" else "cpu"
+        with torch.autocast(device_type=device_type, enabled=False):  # Force float32
+            freqs = (inv_freq_expanded.float() @ position_ids_expanded.float()).transpose(1, 2)
+            emb = torch.cat((freqs, freqs), dim=-1)
+            cos = emb.cos() * self.attention_scaling
+            sin = emb.sin() * self.attention_scaling
+
+        return cos.to(dtype=x.dtype), sin.to(dtype=x.dtype)
+
+
+@auto_docstring
+class SolarOpenModel(SolarOpenPreTrainedModel):
+    _keys_to_ignore_on_load_unexpected = [r"model\.layers\.92.*", r"model\.layers\.46.*"]
+
+    def __init__(self, config: SolarOpenConfig):
+        super().__init__(config)
+        self.padding_idx = config.pad_token_id
+        self.vocab_size = config.vocab_size
+
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)
+        self.layers = nn.ModuleList(
+            [SolarOpenDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)]
+        )
+        self.norm = SolarOpenRMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.rotary_emb = SolarOpenRotaryEmbedding(config=config)
+        self.gradient_checkpointing = False
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @check_model_inputs()
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> BaseModelOutputWithPast:
+        if (input_ids is None) ^ (inputs_embeds is not None):
+            raise ValueError("You must specify exactly one of input_ids or inputs_embeds")
+
+        if inputs_embeds is None:
+            inputs_embeds: torch.Tensor = self.embed_tokens(input_ids)
+
+        if use_cache and past_key_values is None:
+            past_key_values = DynamicCache(config=self.config)
+
+        if cache_position is None:
+            past_seen_tokens = past_key_values.get_seq_length() if past_key_values is not None else 0
+            cache_position: torch.Tensor = torch.arange(
+                past_seen_tokens, past_seen_tokens + inputs_embeds.shape[1], device=inputs_embeds.device
+            )
+
+        if position_ids is None:
+            position_ids = cache_position.unsqueeze(0)
+
+        causal_mask = create_causal_mask(
+            config=self.config,
+            input_embeds=inputs_embeds,
+            attention_mask=attention_mask,
+            cache_position=cache_position,
+            past_key_values=past_key_values,
+            position_ids=position_ids,
+        )
+
+        hidden_states = inputs_embeds
+        position_embeddings = self.rotary_emb(hidden_states, position_ids)
+
+        for decoder_layer in self.layers[: self.config.num_hidden_layers]:
+            hidden_states = decoder_layer(
+                hidden_states,
+                attention_mask=causal_mask,
+                position_ids=position_ids,
+                past_key_values=past_key_values,
+                cache_position=cache_position,
+                position_embeddings=position_embeddings,
+                **kwargs,
+            )
+
+        hidden_states = self.norm(hidden_states)
+        return BaseModelOutputWithPast(
+            last_hidden_state=hidden_states,
+            past_key_values=past_key_values,
+        )
+
+
+@auto_docstring
+class SolarOpenForCausalLM(SolarOpenPreTrainedModel, GenerationMixin):
+    _tied_weights_keys = ["lm_head.weight"]
+    _tp_plan = {"lm_head": "colwise_rep"}
+    _pp_plan = {"lm_head": (["hidden_states"], ["logits"])}
+
+    def __init__(self, config):
+        super().__init__(config)
+        self.model = SolarOpenModel(config)
+        self.vocab_size = config.vocab_size
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+
+        # Initialize weights and apply final processing
+        self.post_init()
+
+    @can_return_tuple
+    @auto_docstring
+    def forward(
+        self,
+        input_ids: Optional[torch.LongTensor] = None,
+        attention_mask: Optional[torch.Tensor] = None,
+        position_ids: Optional[torch.LongTensor] = None,
+        past_key_values: Optional[Cache] = None,
+        inputs_embeds: Optional[torch.FloatTensor] = None,
+        labels: Optional[torch.LongTensor] = None,
+        use_cache: Optional[bool] = None,
+        cache_position: Optional[torch.LongTensor] = None,
+        logits_to_keep: Union[int, torch.Tensor] = 0,
+        **kwargs: Unpack[TransformersKwargs],
+    ) -> CausalLMOutputWithPast:
+
+        outputs: BaseModelOutputWithPast = self.model(
+            input_ids=input_ids,
+            attention_mask=attention_mask,
+            position_ids=position_ids,
+            past_key_values=past_key_values,
+            inputs_embeds=inputs_embeds,
+            use_cache=use_cache,
+            cache_position=cache_position,
+            **kwargs,
+        )
+
+        hidden_states = outputs.last_hidden_state
+        # Only compute necessary logits, and do not upcast them to float if we are not computing the loss
+        slice_indices = slice(-logits_to_keep, None) if isinstance(logits_to_keep, int) else logits_to_keep
+        logits = self.lm_head(hidden_states[:, slice_indices, :])
+
+        loss = None
+        if labels is not None:
+            loss = self.loss_function(logits=logits, labels=labels, vocab_size=self.config.vocab_size, **kwargs)
+
+        return CausalLMOutputWithPast(
+            loss=loss,
+            logits=logits,
+            past_key_values=outputs.past_key_values,
+            hidden_states=outputs.hidden_states,
+            attentions=outputs.attentions,
+        )
+
+
+__all__ = ["SolarOpenPreTrainedModel", "SolarOpenModel", "SolarOpenForCausalLM"]

recipe.yaml

@@ -0,0 +1,6 @@
+default_stage:
+  default_modifiers:
+    QuantizationModifier:
+      targets: [Linear]
+      ignore: [lm_head]
+      scheme: FP8_DYNAMIC

special_tokens_map.json

@@ -0,0 +1,4006 @@
+{
+  "additional_special_tokens": [
+    "<unk>",
+    "<|startoftext|>",
+    "<|endoftext|>",
+    "<|fim_prefix|>",
+    "<|fim_middle|>",
+    "<|fim_suffix|>",
+    "<|special_6|>",
+    "<|special_7|>",
+    "<|special_8|>",
+    "<|special_9|>",
+    "<|special_10|>",
+    "<|special_11|>",
+    "<|special_12|>",
+    "<|special_13|>",
+    "<|special_14|>",
+    "<|special_15|>",
+    "<|special_16|>",
+    "<|special_17|>",
+    "<|special_18|>",
+    "<|special_19|>",
+    "<|flush|>",
+    "<|calls|>",
+    "<|tools:begin|>",
+    "<|tools:end|>",
+    "<|tool:begin|>",
+    "<|tool:end|>",
+    "<|tool_response|>",
+    "<|tool_response:begin|>",
+    "<|tool_response:end|>",
+    "<|tool_response:name|>",
+    "<|tool_response:result|>",
+    "<|special_40|>",
+    "<|special_41|>",
+    "<|special_42|>",
+    "<|special_43|>",
+    "<|special_44|>",
+    "<|special_45|>",
+    "<|special_46|>",
+    "<|special_47|>",
+    "<|special_48|>",
+    "<|special_49|>",
+    "<|special_50|>",
+    "<|special_51|>",
+    "<|special_52|>",
+    "<|special_53|>",
+    "<|special_54|>",
+    "<|special_55|>",
+    "<|special_56|>",
+    "<|special_57|>",
+    "<|special_58|>",
+    "<|special_59|>",
+    "<|special_60|>",
+    "<|special_61|>",
+    "<|special_62|>",
+    "<|special_63|>",
+    "<|special_64|>",
+    "<|special_65|>",
+    "<|special_66|>",
+    "<|special_67|>",
+    "<|special_68|>",
+    "<|special_69|>",
+    "<|special_70|>",
+    "<|special_71|>",
+    "<|special_72|>",
+    "<|special_73|>",
+    "<|special_74|>",
+    "<|special_75|>",
+    "<|special_76|>",
+    "<|special_77|>",
+    "<|special_78|>",
+    "<|special_79|>",
+    "<|special_80|>",
+    "<|special_81|>",
+    "<|special_82|>",
+    "<|special_83|>",
+    "<|special_84|>",
+    "<|special_85|>",
+    "<|special_86|>",
+    "<|special_87|>",
+    "<|special_88|>",
+    "<|special_89|>",
+    "<|special_90|>",
+    "<|special_91|>",
+    "<|special_92|>",
+    "<|special_93|>",
+    "<|special_94|>",
+    "<|special_95|>",
+    "<|special_96|>",
+    "<|special_97|>",
+    "<|special_98|>",
+    "<|special_99|>",
+    "<|special_100|>",
+    "<|special_101|>",
+    "<|special_102|>",
+    "<|special_103|>",
+    "<|special_104|>",
+    "<|special_105|>",
+    "<|special_106|>",
+    "<|special_107|>",
+    "<|special_108|>",
+    "<|special_109|>",
+    "<|special_110|>",
+    "<|special_111|>",
+    "<|special_112|>",
+    "<|special_113|>",
+    "<|special_114|>",
+    "<|special_115|>",
+    "<|special_116|>",
+    "<|special_117|>",
+    "<|special_118|>",
+    "<|special_119|>",
+    "<|special_120|>",
+    "<|special_121|>",
+    "<|special_122|>",
+    "<|special_123|>",
+    "<|special_124|>",
+    "<|special_125|>",
+    "<|special_126|>",
+    "<|special_127|>",
+    "<|special_128|>",
+    "<|special_129|>",
+    "<|special_130|>",
+    "<|special_131|>",
+    "<|special_132|>",
+    "<|special_133|>",
+    "<|special_134|>",
+    "<|special_135|>",
+    "<|special_136|>",
+    "<|special_137|>",
+    "<|special_138|>",
+    "<|special_139|>",
+    "<|special_140|>",
+    "<|special_141|>",
+    "<|special_142|>",
+    "<|special_143|>",
+    "<|special_144|>",
+    "<|special_145|>",
+    "<|special_146|>",
+    "<|special_147|>",
+    "<|special_148|>",
+    "<|special_149|>",
+    "<|special_150|>",
+    "<|special_151|>",
+    "<|special_152|>",
+    "<|special_153|>",
+    "<|special_154|>",
+    "<|special_155|>",
+    "<|special_156|>",
+    "<|special_157|>",
+    "<|special_158|>",
+    "<|special_159|>",
+    "<|special_160|>",
+    "<|special_161|>",
+    "<|special_162|>",
+    "<|special_163|>",
+    "<|special_164|>",
+    "<|special_165|>",
+    "<|special_166|>",
+    "<|special_167|>",
+    "<|special_168|>",
+    "<|special_169|>",
+    "<|special_170|>",
+    "<|special_171|>",
+    "<|special_172|>",
+    "<|special_173|>",
+    "<|special_174|>",
+    "<|special_175|>",
+    "<|special_176|>",
+    "<|special_177|>",
+    "<|special_178|>",
+    "<|special_179|>",
+    "<|special_180|>",
+    "<|special_181|>",
+    "<|special_182|>",
+    "<|special_183|>",
+    "<|special_184|>",
+    "<|special_185|>",
+    "<|special_186|>",
+    "<|special_187|>",
+    "<|special_188|>",
+    "<|special_189|>",
+    "<|special_190|>",
+    "<|special_191|>",
+    "<|special_192|>",
+    "<|special_193|>",
+    "<|special_194|>",
+    "<|special_195|>",
+    "<|special_196|>",
+    "<|special_197|>",
+    "<|special_198|>",
+    "<|special_199|>",
+    "<|special_200|>",
+    "<|special_201|>",
+    "<|special_202|>",
+    "<|special_203|>",
+    "<|special_204|>",
+    "<|special_205|>",
+    "<|special_206|>",
+    "<|special_207|>",
+    "<|special_208|>",
+    "<|special_209|>",
+    "<|special_210|>",
+    "<|special_211|>",
+    "<|special_212|>",
+    "<|special_213|>",
+    "<|special_214|>",
+    "<|special_215|>",
+    "<|special_216|>",
+    "<|special_217|>",
+    "<|special_218|>",
+    "<|special_219|>",
+    "<|special_220|>",
+    "<|special_221|>",
+    "<|special_222|>",
+    "<|special_223|>",
+    "<|special_224|>",
+    "<|special_225|>",
+    "<|special_226|>",
+    "<|special_227|>",
+    "<|special_228|>",
+    "<|special_229|>",
+    "<|special_230|>",
+    "<|special_231|>",
+    "<|special_232|>",
+    "<|special_233|>",
+    "<|special_234|>",
+    "<|special_235|>",
+    "<|special_236|>",
+    "<|special_237|>",
+    "<|special_238|>",
+    "<|special_239|>",
+    "<|special_240|>",
+    "<|special_241|>",
+    "<|special_242|>",
+    "<|special_243|>",
+    "<|special_244|>",
+    "<|special_245|>",
+    "<|special_246|>",
+    "<|special_247|>",
+    "<|special_248|>",
+    "<|special_249|>",
+    "<|special_250|>",
+    "<|special_251|>",
+    "<|special_252|>",
+    "<|special_253|>",
+    "<|special_254|>",
+    "<|special_255|>",
+    "<|special_256|>",
+    "<|special_257|>",
+    "<|special_258|>",
+    "<|special_259|>",
+    "<|special_260|>",
+    "<|special_261|>",
+    "<|special_262|>",
+    "<|special_263|>",
+    "<|special_264|>",
+    "<|special_265|>",
+    "<|special_266|>",
+    "<|special_267|>",
+    "<|special_268|>",
+    "<|special_269|>",
+    "<|special_270|>",
+    "<|special_271|>",
+    "<|special_272|>",
+    "<|special_273|>",
+    "<|special_274|>",
+    "<|special_275|>",
+    "<|special_276|>",
+    "<|special_277|>",
+    "<|special_278|>",
+    "<|special_279|>",
+    "<|special_280|>",
+    "<|special_281|>",
+    "<|special_282|>",
+    "<|special_283|>",
+    "<|special_284|>",
+    "<|special_285|>",
+    "<|special_286|>",
+    "<|special_287|>",
+    "<|special_288|>",
+    "<|special_289|>",
+    "<|special_290|>",
+    "<|special_291|>",
+    "<|special_292|>",
+    "<|special_293|>",
+    "<|special_294|>",
+    "<|special_295|>",
+    "<|special_296|>",
+    "<|special_297|>",
+    "<|special_298|>",
+    "<|special_299|>",
+    "<|special_300|>",
+    "<|special_301|>",
+    "<|special_302|>",
+    "<|special_303|>",
+    "<|special_304|>",
+    "<|special_305|>",
+    "<|special_306|>",
+    "<|special_307|>",
+    "<|special_308|>",
+    "<|special_309|>",
+    "<|special_310|>",
+    "<|special_311|>",
+    "<|special_312|>",
+    "<|special_313|>",
+    "<|special_314|>",
+    "<|special_315|>",
+    "<|special_316|>",
+    "<|special_317|>",
+    "<|special_318|>",
+    "<|special_319|>",
+    "<|special_320|>",
+    "<|special_321|>",
+    "<|special_322|>",
+    "<|special_323|>",
+    "<|special_324|>",
+    "<|special_325|>",
+    "<|special_326|>",
+    "<|special_327|>",
+    "<|special_328|>",
+    "<|special_329|>",
+    "<|special_330|>",
+    "<|special_331|>",
+    "<|special_332|>",
+    "<|special_333|>",
+    "<|special_334|>",
+    "<|special_335|>",
+    "<|special_336|>",
+    "<|special_337|>",
+    "<|special_338|>",
+    "<|special_339|>",
+    "<|special_340|>",
+    "<|special_341|>",
+    "<|special_342|>",
+    "<|special_343|>",
+    "<|special_344|>",
+    "<|special_345|>",
+    "<|special_346|>",
+    "<|special_347|>",
+    "<|special_348|>",
+    "<|special_349|>",
+    "<|special_350|>",
+    "<|special_351|>",
+    "<|special_352|>",
+    "<|special_353|>",
+    "<|special_354|>",
+    "<|special_355|>",
+    "<|special_356|>",
+    "<|special_357|>",
+    "<|special_358|>",
+    "<|special_359|>",
+    "<|special_360|>",
+    "<|special_361|>",
+    "<|special_362|>",
+    "<|special_363|>",
+    "<|special_364|>",
+    "<|special_365|>",
+    "<|special_366|>",
+    "<|special_367|>",
+    "<|special_368|>",
+    "<|special_369|>",
+    "<|special_370|>",
+    "<|special_371|>",
+    "<|special_372|>",
+    "<|special_373|>",
+    "<|special_374|>",
+    "<|special_375|>",
+    "<|special_376|>",
+    "<|special_377|>",
+    "<|special_378|>",
+    "<|special_379|>",
+    "<|special_380|>",
+    "<|special_381|>",
+    "<|special_382|>",
+    "<|special_383|>",
+    "<|special_384|>",
+    "<|special_385|>",
+    "<|special_386|>",
+    "<|special_387|>",
+    "<|special_388|>",
+    "<|special_389|>",
+    "<|special_390|>",
+    "<|special_391|>",
+    "<|special_392|>",
+    "<|special_393|>",
+    "<|special_394|>",
+    "<|special_395|>",
+    "<|special_396|>",
+    "<|special_397|>",
+    "<|special_398|>",
+    "<|special_399|>",
+    "<|special_400|>",
+    "<|special_401|>",
+    "<|special_402|>",
+    "<|special_403|>",
+    "<|special_404|>",
+    "<|special_405|>",
+    "<|special_406|>",
+    "<|special_407|>",
+    "<|special_408|>",
+    "<|special_409|>",
+    "<|special_410|>",
+    "<|special_411|>",
+    "<|special_412|>",
+    "<|special_413|>",
+    "<|special_414|>",
+    "<|special_415|>",
+    "<|special_416|>",
+    "<|special_417|>",
+    "<|special_418|>",
+    "<|special_419|>",
+    "<|special_420|>",
+    "<|special_421|>",
+    "<|special_422|>",
+    "<|special_423|>",
+    "<|special_424|>",
+    "<|special_425|>",
+    "<|special_426|>",
+    "<|special_427|>",
+    "<|special_428|>",
+    "<|special_429|>",
+    "<|special_430|>",
+    "<|special_431|>",
+    "<|special_432|>",
+    "<|special_433|>",
+    "<|special_434|>",
+    "<|special_435|>",
+    "<|special_436|>",
+    "<|special_437|>",
+    "<|special_438|>",
+    "<|special_439|>",
+    "<|special_440|>",
+    "<|special_441|>",
+    "<|special_442|>",
+    "<|special_443|>",
+    "<|special_444|>",
+    "<|special_445|>",
+    "<|special_446|>",
+    "<|special_447|>",
+    "<|special_448|>",
+    "<|special_449|>",
+    "<|special_450|>",
+    "<|special_451|>",
+    "<|special_452|>",
+    "<|special_453|>",
+    "<|special_454|>",
+    "<|special_455|>",
+    "<|special_456|>",
+    "<|special_457|>",
+    "<|special_458|>",
+    "<|special_459|>",
+    "<|special_460|>",
+    "<|special_461|>",
+    "<|special_462|>",
+    "<|special_463|>",
+    "<|special_464|>",
+    "<|special_465|>",
+    "<|special_466|>",
+    "<|special_467|>",
+    "<|special_468|>",
+    "<|special_469|>",
+    "<|special_470|>",
+    "<|special_471|>",
+    "<|special_472|>",
+    "<|special_473|>",
+    "<|special_474|>",
+    "<|special_475|>",
+    "<|special_476|>",
+    "<|special_477|>",
+    "<|special_478|>",
+    "<|special_479|>",
+    "<|special_480|>",
+    "<|special_481|>",
+    "<|special_482|>",
+    "<|special_483|>",
+    "<|special_484|>",
+    "<|special_485|>",
+    "<|special_486|>",
+    "<|special_487|>",
+    "<|special_488|>",
+    "<|special_489|>",
+    "<|special_490|>",
+    "<|special_491|>",
+    "<|special_492|>",
+    "<|special_493|>",
+    "<|special_494|>",
+    "<|special_495|>",
+    "<|special_496|>",
+    "<|special_497|>",
+    "<|special_498|>",
+    "<|special_499|>",
+    "<|special_500|>",
+    "<|special_501|>",
+    "<|special_502|>",
+    "<|special_503|>",
+    "<|special_504|>",
+    "<|special_505|>",
+    "<|special_506|>",
+    "<|special_507|>",
+    "<|special_508|>",
+    "<|special_509|>",
+    "<|special_510|>",
+    "<|special_511|>",
+    "<|special_625|>",
+    "<|special_626|>",
+    "<|special_627|>",
+    "<|special_628|>",
+    "<|special_629|>",
+    "<|special_630|>",
+    "<|special_631|>",
+    "<|special_632|>",
+    "<|special_633|>",
+    "<|special_634|>",
+    "<|special_635|>",
+    "<|special_636|>",
+    "<|special_637|>",
+    "<|special_638|>",
+    "<|special_639|>",
+    "<|special_640|>",
+    "<|special_641|>",
+    "<|special_642|>",
+    "<|special_643|>",
+    "<|special_644|>",
+    "<|special_645|>",
+    "<|special_646|>",
+    "<|special_647|>",
+    "<|special_648|>",
+    "<|special_649|>",
+    "<|special_650|>",
+    "<|special_651|>",
+    "<|special_652|>",
+    "<|special_653|>",
+    "<|special_654|>",
+    "<|special_655|>",
+    "<|special_656|>",
+    "<|special_657|>",
+    "<|special_658|>",
+    "<|special_659|>",
+    "<|special_660|>",
+    "<|special_661|>",
+    "<|special_662|>",
+    "<|special_663|>",
+    "<|special_664|>",
+    "<|special_665|>",
+    "<|special_666|>",
+    "<|special_667|>",
+    "<|special_668|>",
+    "<|special_669|>",
+    "<|special_670|>",
+    "<|special_671|>",
+    "<|special_672|>",
+    "<|special_673|>",
+    "<|special_674|>",
+    "<|special_675|>",
+    "<|special_676|>",
+    "<|special_677|>",
+    "<|special_678|>",
+    "<|special_679|>",
+    "<|special_680|>",
+    "<|special_681|>",
+    "<|special_682|>",
+    "<|special_683|>",
+    "<|special_684|>",
+    "<|special_685|>",
+    "<|special_686|>",
+    "<|special_687|>",
+    "<|special_688|>",
+    "<|special_689|>",
+    "<|special_690|>",
+    "<|special_691|>",
+    "<|special_692|>",
+    "<|special_693|>",
+    "<|special_694|>",
+    "<|special_695|>",
+    "<|special_696|>",
+    "<|special_697|>",
+    "<|special_698|>",
+    "<|special_699|>",
+    "<|special_700|>",
+    "<|special_701|>",
+    "<|special_702|>",
+    "<|special_703|>",
+    "<|special_704|>",
+    "<|special_705|>",
+    "<|special_706|>",
+    "<|special_707|>",
+    "<|special_708|>",
+    "<|special_709|>",
+    "<|special_710|>",
+    "<|special_711|>",
+    "<|special_712|>",
+    "<|special_713|>",
+    "<|special_714|>",
+    "<|special_715|>",
+    "<|special_716|>",
+    "<|special_717|>",
+    "<|special_718|>",
+    "<|special_719|>",
+    "<|special_720|>",
+    "<|special_721|>",
+    "<|special_722|>",
+    "<|special_723|>",
+    "<|special_724|>",
+    "<|special_725|>",
+    "<|special_726|>",
+    "<|special_727|>",
+    "<|special_728|>",
+    "<|special_729|>",
+    "<|special_730|>",
+    "<|special_731|>",
+    "<|special_732|>",
+    "<|special_733|>",
+    "<|special_734|>",
+    "<|special_735|>",
+    "<|special_736|>",
+    "<|special_737|>",
+    "<|special_738|>",
+    "<|special_739|>",
+    "<|special_740|>",
+    "<|special_741|>",
+    "<|special_742|>",
+    "<|special_743|>",
+    "<|special_744|>",
+    "<|special_745|>",
+    "<|special_746|>",
+    "<|special_747|>",
+    "<|special_748|>",
+    "<|special_749|>",
+    "<|special_750|>",
+    "<|special_751|>",
+    "<|special_752|>",
+    "<|special_753|>",
+    "<|special_754|>",
+    "<|special_755|>",
+    "<|special_756|>",
+    "<|special_757|>",
+    "<|special_758|>",
+    "<|special_759|>",
+    "<|special_760|>",
+    "<|special_761|>",
+    "<|special_762|>",
+    "<|special_763|>",
+    "<|special_764|>",
+    "<|special_765|>",
+    "<|special_766|>",
+    "<|special_767|>",
+    "<|special_768|>",
+    "<|special_769|>",
+    "<|special_770|>",
+    "<|special_771|>",
+    "<|special_772|>",
+    "<|special_773|>",
+    "<|special_774|>",
+    "<|special_775|>",
+    "<|special_776|>",
+    "<|special_777|>",
+    "<|special_778|>",
+    "<|special_779|>",
+    "<|special_780|>",
+    "<|special_781|>",
+    "<|special_782|>",
+    "<|special_783|>",
+    "<|special_784|>",
+    "<|special_785|>",
+    "<|special_786|>",
+    "<|special_787|>",
+    "<|special_788|>",
+    "<|special_789|>",
+    "<|special_790|>",
+    "<|special_791|>",
+    "<|special_792|>",
+    "<|special_793|>",
+    "<|special_794|>",
+    "<|special_795|>",
+    "<|special_796|>",
+    "<|special_797|>",
+    "<|special_798|>",
+    "<|special_799|>",
+    "<|special_800|>",
+    "<|special_801|>",
+    "<|special_802|>",
+    "<|special_803|>",
+    "<|special_804|>",
+    "<|special_805|>",
+    "<|special_806|>",
+    "<|special_807|>",
+    "<|special_808|>",
+    "<|special_809|>",
+    "<|special_810|>",
+    "<|special_811|>",
+    "<|special_812|>",
+    "<|special_813|>",
+    "<|special_814|>",
+    "<|special_815|>",
+    "<|special_816|>",
+    "<|special_817|>",
+    "<|special_818|>",
+    "<|special_819|>",
+    "<|special_820|>",
+    "<|special_821|>",
+    "<|special_822|>",
+    "<|special_823|>",
+    "<|special_824|>",
+    "<|special_825|>",
+    "<|special_826|>",
+    "<|special_827|>",
+    "<|special_828|>",
+    "<|special_829|>",
+    "<|special_830|>",
+    "<|special_831|>",
+    "<|special_832|>",
+    "<|special_833|>",
+    "<|special_834|>",
+    "<|special_835|>",
+    "<|special_836|>",
+    "<|special_837|>",
+    "<|special_838|>",
+    "<|special_839|>",
+    "<|special_840|>",
+    "<|special_841|>",
+    "<|special_842|>",
+    "<|special_843|>",
+    "<|special_844|>",
+    "<|special_845|>",
+    "<|special_846|>",
+    "<|special_847|>",
+    "<|special_848|>",
+    "<|special_849|>",
+    "<|special_850|>",
+    "<|special_851|>",
+    "<|special_852|>",
+    "<|special_853|>",
+    "<|special_854|>",
+    "<|special_855|>",
+    "<|special_856|>",
+    "<|special_857|>",
+    "<|special_858|>",
+    "<|special_859|>",
+    "<|special_860|>",
+    "<|special_861|>",
+    "<|special_862|>",
+    "<|special_863|>",
+    "<|special_864|>",
+    "<|special_865|>",
+    "<|special_866|>",
+    "<|special_867|>",
+    "<|special_868|>",
+    "<|special_869|>",
+    "<|special_870|>",
+    "<|special_871|>",
+    "<|special_872|>",
+    "<|special_873|>",
+    "<|special_874|>",
+    "<|special_875|>",
+    "<|special_876|>",
+    "<|special_877|>",
+    "<|special_878|>",
+    "<|special_879|>",
+    "<|special_880|>",
+    "<|special_881|>",
+    "<|special_882|>",
+    "<|special_883|>",
+    "<|special_884|>",
+    "<|special_885|>",
+    "<|special_886|>",
+    "<|special_887|>",
+    "<|special_888|>",
+    "<|special_889|>",
+    "<|special_890|>",
+    "<|special_891|>",
+    "<|special_892|>",
+    "<|special_893|>",
+    "<|special_894|>",
+    "<|special_895|>",
+    "<|special_896|>",
+    "<|special_897|>",
+    "<|special_898|>",
+    "<|special_899|>",
+    "<|special_900|>",
+    "<|special_901|>",
+    "<|special_902|>",
+    "<|special_903|>",
+    "<|special_904|>",
+    "<|special_905|>",
+    "<|special_906|>",
+    "<|special_907|>",
+    "<|special_908|>",
+    "<|special_909|>",
+    "<|special_910|>",
+    "<|special_911|>",
+    "<|special_912|>",
+    "<|special_913|>",
+    "<|special_914|>",
+    "<|special_915|>",
+    "<|special_916|>",
+    "<|special_917|>",
+    "<|special_918|>",
+    "<|special_919|>",
+    "<|special_920|>",
+    "<|special_921|>",
+    "<|special_922|>",
+    "<|special_923|>",
+    "<|special_924|>",
+    "<|special_925|>",
+    "<|special_926|>",
+    "<|special_927|>",
+    "<|special_928|>",
+    "<|special_929|>",
+    "<|special_930|>",
+    "<|special_931|>",
+    "<|special_932|>",
+    "<|special_933|>",
+    "<|special_934|>",
+    "<|special_935|>",
+    "<|special_936|>",
+    "<|special_937|>",
+    "<|special_938|>",
+    "<|special_939|>",
+    "<|special_940|>",
+    "<|special_941|>",
+    "<|special_942|>",
+    "<|special_943|>",
+    "<|special_944|>",
+    "<|special_945|>",
+    "<|special_946|>",
+    "<|special_947|>",
+    "<|special_948|>",
+    "<|special_949|>",
+    "<|special_950|>",
+    "<|special_951|>",
+    "<|special_952|>",
+    "<|special_953|>",
+    "<|special_954|>",
+    "<|special_955|>",
+    "<|special_956|>",
+    "<|special_957|>",
+    "<|special_958|>",
+    "<|special_959|>",
+    "<|special_960|>",
+    "<|special_961|>",
+    "<|special_962|>",
+    "<|special_963|>",
+    "<|special_964|>",
+    "<|special_965|>",
+    "<|special_966|>",
+    "<|special_967|>",
+    "<|special_968|>",
+    "<|special_969|>",
+    "<|special_970|>",
+    "<|special_971|>",
+    "<|special_972|>",
+    "<|special_973|>",
+    "<|special_974|>",
+    "<|special_975|>",
+    "<|special_976|>",
+    "<|special_977|>",
+    "<|special_978|>",
+    "<|special_979|>",
+    "<|special_980|>",
+    "<|special_981|>",
+    "<|special_982|>",
+    "<|special_983|>",
+    "<|special_984|>",
+    "<|special_985|>",
+    "<|special_986|>",
+    "<|special_987|>",
+    "<|special_988|>",
+    "<|special_989|>",
+    "<|special_990|>",
+    "<|special_991|>",
+    "<|special_992|>",
+    "<|special_993|>",
+    "<|special_994|>",
+    "<|special_995|>",
+    "<|special_996|>",
+    "<|special_997|>",
+    "<|special_998|>",
+    "<|special_999|>",
+    "<|special_1000|>",
+    "<|special_1001|>",
+    "<|special_1002|>",
+    "<|special_1003|>",
+    "<|special_1004|>",
+    "<|special_1005|>",
+    "<|special_1006|>",
+    "<|special_1007|>",
+    "<|special_1008|>",
+    "<|special_1009|>",
+    "<|special_1010|>",
+    "<|special_1011|>",
+    "<|special_1012|>",
+    "<|special_1013|>",
+    "<|special_1014|>",
+    "<|special_1015|>",
+    "<|special_1016|>",
+    "<|special_1017|>",
+    "<|special_1018|>",
+    "<|special_1019|>",
+    "<|special_1020|>",
+    "<|special_1021|>",
+    "<|special_1022|>",
+    "<|special_1023|>",
+    "<|special_1024|>",
+    "<|special_1025|>",
+    "<|special_1026|>",
+    "<|special_1027|>",
+    "<|special_1028|>",
+    "<|special_1029|>",
+    "<|special_1030|>",
+    "<|special_1031|>",
+    "<|special_1032|>",
+    "<|special_1033|>",
+    "<|special_1034|>",
+    "<|special_1035|>",
+    "<|special_1036|>",
+    "<|special_1037|>",
+    "<|special_1038|>",
+    "<|special_1039|>",
+    "<|special_1040|>",
+    "<|special_1041|>",
+    "<|special_1042|>",
+    "<|special_1043|>",
+    "<|special_1044|>",
+    "<|special_1045|>",
+    "<|special_1046|>",
+    "<|special_1047|>",
+    "<|special_1048|>",
+    "<|special_1049|>",
+    "<|special_1050|>",
+    "<|special_1051|>",
+    "<|special_1052|>",
+    "<|special_1053|>",
+    "<|special_1054|>",
+    "<|special_1055|>",
+    "<|special_1056|>",
+    "<|special_1057|>",
+    "<|special_1058|>",
+    "<|special_1059|>",
+    "<|special_1060|>",
+    "<|special_1061|>",
+    "<|special_1062|>",
+    "<|special_1063|>",
+    "<|special_1064|>",
+    "<|special_1065|>",
+    "<|special_1066|>",
+    "<|special_1067|>",
+    "<|special_1068|>",
+    "<|special_1069|>",
+    "<|special_1070|>",
+    "<|special_1071|>",
+    "<|special_1072|>",
+    "<|special_1073|>",
+    "<|special_1074|>",
+    "<|special_1075|>",
+    "<|special_1076|>",
+    "<|special_1077|>",
+    "<|special_1078|>",
+    "<|special_1079|>",
+    "<|special_1080|>",
+    "<|special_1081|>",
+    "<|special_1082|>",
+    "<|special_1083|>",
+    "<|special_1084|>",
+    "<|special_1085|>",
+    "<|special_1086|>",
+    "<|special_1087|>",
+    "<|special_1088|>",
+    "<|special_1089|>",
+    "<|special_1090|>",
+    "<|special_1091|>",
+    "<|special_1092|>",
+    "<|special_1093|>",
+    "<|special_1094|>",
+    "<|special_1095|>",
+    "<|special_1096|>",
+    "<|special_1097|>",
+    "<|special_1098|>",
+    "<|special_1099|>",
+    "<|special_1100|>",
+    "<|special_1101|>",
+    "<|special_1102|>",
+    "<|special_1103|>",
+    "<|special_1104|>",
+    "<|special_1105|>",
+    "<|special_1106|>",
+    "<|special_1107|>",
+    "<|special_1108|>",
+    "<|special_1109|>",
+    "<|special_1110|>",
+    "<|special_1111|>",
+    "<|special_1112|>",
+    "<|special_1113|>",
+    "<|special_1114|>",
+    "<|special_1115|>",
+    "<|special_1116|>",
+    "<|special_1117|>",
+    "<|special_1118|>",
+    "<|special_1119|>",
+    "<|special_1120|>",
+    "<|special_1121|>",
+    "<|special_1122|>",
+    "<|special_1123|>",
+    "<|special_1124|>",
+    "<|special_1125|>",
+    "<|special_1126|>",
+    "<|special_1127|>",
+    "<|special_1128|>",
+    "<|special_1129|>",
+    "<|special_1130|>",
+    "<|special_1131|>",
+    "<|special_1132|>",
+    "<|special_1133|>",
+    "<|special_1134|>",
+    "<|special_1135|>",
+    "<|special_1136|>",
+    "<|special_1137|>",
+    "<|special_1138|>",
+    "<|special_1139|>",
+    "<|special_1140|>",
+    "<|special_1141|>",
+    "<|special_1142|>",
+    "<|special_1143|>",
+    "<|special_1144|>",
+    "<|special_1145|>",
+    "<|special_1146|>",
+    "<|special_1147|>",
+    "<|special_1148|>",
+    "<|special_1149|>",
+    "<|special_1150|>",
+    "<|special_1151|>",
+    "<|special_1152|>",
+    "<|special_1153|>",
+    "<|special_1154|>",
+    "<|special_1155|>",
+    "<|special_1156|>",
+    "<|special_1157|>",
+    "<|special_1158|>",
+    "<|special_1159|>",
+    "<|special_1160|>",
+    "<|special_1161|>",
+    "<|special_1162|>",
+    "<|special_1163|>",
+    "<|special_1164|>",
+    "<|special_1165|>",
+    "<|special_1166|>",
+    "<|special_1167|>",
+    "<|special_1168|>",
+    "<|special_1169|>",
+    "<|special_1170|>",
+    "<|special_1171|>",
+    "<|special_1172|>",
+    "<|special_1173|>",
+    "<|special_1174|>",
+    "<|special_1175|>",
+    "<|special_1176|>",
+    "<|special_1177|>",
+    "<|special_1178|>",
+    "<|special_1179|>",
+    "<|special_1180|>",
+    "<|special_1181|>",
+    "<|special_1182|>",
+    "<|special_1183|>",
+    "<|special_1184|>",
+    "<|special_1185|>",
+    "<|special_1186|>",
+    "<|special_1187|>",
+    "<|special_1188|>",
+    "<|special_1189|>",
+    "<|special_1190|>",
+    "<|special_1191|>",
+    "<|special_1192|>",
+    "<|special_1193|>",
+    "<|special_1194|>",
+    "<|special_1195|>",
+    "<|special_1196|>",
+    "<|special_1197|>",
+    "<|special_1198|>",
+    "<|special_1199|>",
+    "<|special_1200|>",
+    "<|special_1201|>",
+    "<|special_1202|>",
+    "<|special_1203|>",
+    "<|special_1204|>",
+    "<|special_1205|>",
+    "<|special_1206|>",
+    "<|special_1207|>",
+    "<|special_1208|>",
+    "<|special_1209|>",
+    "<|special_1210|>",
+    "<|special_1211|>",
+    "<|special_1212|>",
+    "<|special_1213|>",
+    "<|special_1214|>",
+    "<|special_1215|>",
+    "<|special_1216|>",
+    "<|special_1217|>",
+    "<|special_1218|>",
+    "<|special_1219|>",
+    "<|special_1220|>",
+    "<|special_1221|>",
+    "<|special_1222|>",
+    "<|special_1223|>",
+    "<|special_1224|>",
+    "<|special_1225|>",
+    "<|special_1226|>",
+    "<|special_1227|>",
+    "<|special_1228|>",
+    "<|special_1229|>",
+    "<|special_1230|>",
+    "<|special_1231|>",
+    "<|special_1232|>",
+    "<|special_1233|>",
+    "<|special_1234|>",
+    "<|special_1235|>",
+    "<|special_1236|>",
+    "<|special_1237|>",
+    "<|special_1238|>",
+    "<|special_1239|>",
+    "<|special_1240|>",
+    "<|special_1241|>",
+    "<|special_1242|>",
+    "<|special_1243|>",
+    "<|special_1244|>",
+    "<|special_1245|>",
+    "<|special_1246|>",
+    "<|special_1247|>",
+    "<|special_1248|>",
+    "<|special_1249|>",
+    "<|special_1250|>",
+    "<|special_1251|>",
+    "<|special_1252|>",
+    "<|special_1253|>",
+    "<|special_1254|>",
+    "<|special_1255|>",
+    "<|special_1256|>",
+    "<|special_1257|>",
+    "<|special_1258|>",
+    "<|special_1259|>",
+    "<|special_1260|>",
+    "<|special_1261|>",
+    "<|special_1262|>",
+    "<|special_1263|>",
+    "<|special_1264|>",
+    "<|special_1265|>",
+    "<|special_1266|>",
+    "<|special_1267|>",
+    "<|special_1268|>",
+    "<|special_1269|>",
+    "<|special_1270|>",
+    "<|special_1271|>",
+    "<|special_1272|>",
+    "<|special_1273|>",
+    "<|special_1274|>",
+    "<|special_1275|>",
+    "<|special_1276|>",
+    "<|special_1277|>",
+    "<|special_1278|>",
+    "<|special_1279|>",
+    "<|special_1280|>",
+    "<|special_1281|>",
+    "<|special_1282|>",
+    "<|special_1283|>",
+    "<|special_1284|>",
+    "<|special_1285|>",
+    "<|special_1286|>",
+    "<|special_1287|>",
+    "<|special_1288|>",
+    "<|special_1289|>",
+    "<|special_1290|>",
+    "<|special_1291|>",
+    "<|special_1292|>",
+    "<|special_1293|>",
+    "<|special_1294|>",
+    "<|special_1295|>",
+    "<|special_1296|>",
+    "<|special_1297|>",
+    "<|special_1298|>",
+    "<|special_1299|>",
+    "<|special_1300|>",
+    "<|special_1301|>",
+    "<|special_1302|>",
+    "<|special_1303|>",
+    "<|special_1304|>",
+    "<|special_1305|>",
+    "<|special_1306|>",
+    "<|special_1307|>",
+    "<|special_1308|>",
+    "<|special_1309|>",
+    "<|special_1310|>",
+    "<|special_1311|>",
+    "<|special_1312|>",
+    "<|special_1313|>",
+    "<|special_1314|>",
+    "<|special_1315|>",
+    "<|special_1316|>",
+    "<|special_1317|>",
+    "<|special_1318|>",
+    "<|special_1319|>",
+    "<|special_1320|>",
+    "<|special_1321|>",
+    "<|special_1322|>",
+    "<|special_1323|>",
+    "<|special_1324|>",
+    "<|special_1325|>",
+    "<|special_1326|>",
+    "<|special_1327|>",
+    "<|special_1328|>",
+    "<|special_1329|>",
+    "<|special_1330|>",
+    "<|special_1331|>",
+    "<|special_1332|>",
+    "<|special_1333|>",
+    "<|special_1334|>",
+    "<|special_1335|>",
+    "<|special_1336|>",
+    "<|special_1337|>",
+    "<|special_1338|>",
+    "<|special_1339|>",
+    "<|special_1340|>",
+    "<|special_1341|>",
+    "<|special_1342|>",
+    "<|special_1343|>",
+    "<|special_1344|>",
+    "<|special_1345|>",
+    "<|special_1346|>",
+    "<|special_1347|>",
+    "<|special_1348|>",
+    "<|special_1349|>",
+    "<|special_1350|>",
+    "<|special_1351|>",
+    "<|special_1352|>",
+    "<|special_1353|>",
+    "<|special_1354|>",
+    "<|special_1355|>",
+    "<|special_1356|>",
+    "<|special_1357|>",
+    "<|special_1358|>",
+    "<|special_1359|>",
+    "<|special_1360|>",
+    "<|special_1361|>",
+    "<|special_1362|>",
+    "<|special_1363|>",
+    "<|special_1364|>",
+    "<|special_1365|>",
+    "<|special_1366|>",
+    "<|special_1367|>",
+    "<|special_1368|>",
+    "<|special_1369|>",
+    "<|special_1370|>",
+    "<|special_1371|>",
+    "<|special_1372|>",
+    "<|special_1373|>",
+    "<|special_1374|>",
+    "<|special_1375|>",
+    "<|special_1376|>",
+    "<|special_1377|>",
+    "<|special_1378|>",
+    "<|special_1379|>",
+    "<|special_1380|>",
+    "<|special_1381|>",
+    "<|special_1382|>",
+    "<|special_1383|>",
+    "<|special_1384|>",
+    "<|special_1385|>",
+    "<|special_1386|>",
+    "<|special_1387|>",
+    "<|special_1388|>",
+    "<|special_1389|>",
+    "<|special_1390|>",
+    "<|special_1391|>",
+    "<|special_1392|>",
+    "<|special_1393|>",
+    "<|special_1394|>",
+    "<|special_1395|>",
+    "<|special_1396|>",
+    "<|special_1397|>",
+    "<|special_1398|>",
+    "<|special_1399|>",
+    "<|special_1400|>",
+    "<|special_1401|>",
+    "<|special_1402|>",
+    "<|special_1403|>",
+    "<|special_1404|>",
+    "<|special_1405|>",
+    "<|special_1406|>",
+    "<|special_1407|>",
+    "<|special_1408|>",
+    "<|special_1409|>",
+    "<|special_1410|>",
+    "<|special_1411|>",
+    "<|special_1412|>",
+    "<|special_1413|>",
+    "<|special_1414|>",
+    "<|special_1415|>",
+    "<|special_1416|>",
+    "<|special_1417|>",
+    "<|special_1418|>",
+    "<|special_1419|>",
+    "<|special_1420|>",
+    "<|special_1421|>",
+    "<|special_1422|>",
+    "<|special_1423|>",
+    "<|special_1424|>",
+    "<|special_1425|>",
+    "<|special_1426|>",
+    "<|special_1427|>",
+    "<|special_1428|>",
+    "<|special_1429|>",
+    "<|special_1430|>",
+    "<|special_1431|>",
+    "<|special_1432|>",
+    "<|special_1433|>",
+    "<|special_1434|>",
+    "<|special_1435|>",
+    "<|special_1436|>",
+    "<|special_1437|>",
+    "<|special_1438|>",
+    "<|special_1439|>",
+    "<|special_1440|>",
+    "<|special_1441|>",
+    "<|special_1442|>",
+    "<|special_1443|>",
+    "<|special_1444|>",
+    "<|special_1445|>",
+    "<|special_1446|>",
+    "<|special_1447|>",
+    "<|special_1448|>",
+    "<|special_1449|>",
+    "<|special_1450|>",
+    "<|special_1451|>",
+    "<|special_1452|>",
+    "<|special_1453|>",
+    "<|special_1454|>",
+    "<|special_1455|>",
+    "<|special_1456|>",
+    "<|special_1457|>",
+    "<|special_1458|>",
+    "<|special_1459|>",
+    "<|special_1460|>",
+    "<|special_1461|>",
+    "<|special_1462|>",
+    "<|special_1463|>",
+    "<|special_1464|>",
+    "<|special_1465|>",
+    "<|special_1466|>",
+    "<|special_1467|>",
+    "<|special_1468|>",
+    "<|special_1469|>",
+    "<|special_1470|>",
+    "<|special_1471|>",
+    "<|special_1472|>",
+    "<|special_1473|>",
+    "<|special_1474|>",
+    "<|special_1475|>",
+    "<|special_1476|>",
+    "<|special_1477|>",
+    "<|special_1478|>",
+    "<|special_1479|>",
+    "<|special_1480|>",
+    "<|special_1481|>",
+    "<|special_1482|>",
+    "<|special_1483|>",
+    "<|special_1484|>",
+    "<|special_1485|>",
+    "<|special_1486|>",
+    "<|special_1487|>",
+    "<|special_1488|>",
+    "<|special_1489|>",
+    "<|special_1490|>",
+    "<|special_1491|>",
+    "<|special_1492|>",
+    "<|special_1493|>",
+    "<|special_1494|>",
+    "<|special_1495|>",
+    "<|special_1496|>",
+    "<|special_1497|>",
+    "<|special_1498|>",
+    "<|special_1499|>",
+    "<|special_1500|>",
+    "<|special_1501|>",
+    "<|special_1502|>",
+    "<|special_1503|>",
+    "<|special_1504|>",
+    "<|special_1505|>",
+    "<|special_1506|>",
+    "<|special_1507|>",
+    "<|special_1508|>",
+    "<|special_1509|>",
+    "<|special_1510|>",
+    "<|special_1511|>",
+    "<|special_1512|>",
+    "<|special_1513|>",
+    "<|special_1514|>",
+    "<|special_1515|>",
+    "<|special_1516|>",
+    "<|special_1517|>",
+    "<|special_1518|>",
+    "<|special_1519|>",
+    "<|special_1520|>",
+    "<|special_1521|>",
+    "<|special_1522|>",
+    "<|special_1523|>",
+    "<|special_1524|>",
+    "<|special_1525|>",
+    "<|special_1526|>",
+    "<|special_1527|>",
+    "<|special_1528|>",
+    "<|special_1529|>",
+    "<|special_1530|>",
+    "<|special_1531|>",
+    "<|special_1532|>",
+    "<|special_1533|>",
+    "<|special_1534|>",
+    "<|special_1535|>",
+    "<|special_1536|>",
+    "<|special_1537|>",
+    "<|special_1538|>",
+    "<|special_1539|>",
+    "<|special_1540|>",
+    "<|special_1541|>",
+    "<|special_1542|>",
+    "<|special_1543|>",
+    "<|special_1544|>",
+    "<|special_1545|>",
+    "<|special_1546|>",
+    "<|special_1547|>",
+    "<|special_1548|>",
+    "<|special_1549|>",
+    "<|special_1550|>",
+    "<|special_1551|>",
+    "<|special_1552|>",
+    "<|special_1553|>",
+    "<|special_1554|>",
+    "<|special_1555|>",
+    "<|special_1556|>",
+    "<|special_1557|>",
+    "<|special_1558|>",
+    "<|special_1559|>",
+    "<|special_1560|>",
+    "<|special_1561|>",
+    "<|special_1562|>",
+    "<|special_1563|>",
+    "<|special_1564|>",
+    "<|special_1565|>",
+    "<|special_1566|>",
+    "<|special_1567|>",
+    "<|special_1568|>",
+    "<|special_1569|>",
+    "<|special_1570|>",
+    "<|special_1571|>",
+    "<|special_1572|>",
+    "<|special_1573|>",
+    "<|special_1574|>",
+    "<|special_1575|>",
+    "<|special_1576|>",
+    "<|special_1577|>",
+    "<|special_1578|>",
+    "<|special_1579|>",
+    "<|special_1580|>",
+    "<|special_1581|>",
+    "<|special_1582|>",
+    "<|special_1583|>",
+    "<|special_1584|>",
+    "<|special_1585|>",
+    "<|special_1586|>",
+    "<|special_1587|>",
+    "<|special_1588|>",
+    "<|special_1589|>",
+    "<|special_1590|>",
+    "<|special_1591|>",
+    "<|special_1592|>",
+    "<|special_1593|>",
+    "<|special_1594|>",
+    "<|special_1595|>",
+    "<|special_1596|>",
+    "<|special_1597|>",
+    "<|special_1598|>",
+    "<|special_1599|>",
+    "<|special_1600|>",
+    "<|special_1601|>",
+    "<|special_1602|>",
+    "<|special_1603|>",
+    "<|special_1604|>",
+    "<|special_1605|>",
+    "<|special_1606|>",
+    "<|special_1607|>",
+    "<|special_1608|>",
+    "<|special_1609|>",
+    "<|special_1610|>",
+    "<|special_1611|>",
+    "<|special_1612|>",
+    "<|special_1613|>",
+    "<|special_1614|>",
+    "<|special_1615|>",
+    "<|special_1616|>",
+    "<|special_1617|>",
+    "<|special_1618|>",
+    "<|special_1619|>",
+    "<|special_1620|>",
+    "<|special_1621|>",
+    "<|special_1622|>",
+    "<|special_1623|>",
+    "<|special_1624|>",
+    "<|special_1625|>",
+    "<|special_1626|>",
+    "<|special_1627|>",
+    "<|special_1628|>",
+    "<|special_1629|>",
+    "<|special_1630|>",
+    "<|special_1631|>",
+    "<|special_1632|>",
+    "<|special_1633|>",
+    "<|special_1634|>",
+    "<|special_1635|>",
+    "<|special_1636|>",
+    "<|special_1637|>",
+    "<|special_1638|>",
+    "<|special_1639|>",
+    "<|special_1640|>",
+    "<|special_1641|>",
+    "<|special_1642|>",
+    "<|special_1643|>",
+    "<|special_1644|>",
+    "<|special_1645|>",
+    "<|special_1646|>",
+    "<|special_1647|>",
+    "<|special_1648|>",
+    "<|special_1649|>",
+    "<|special_1650|>",
+    "<|special_1651|>",
+    "<|special_1652|>",
+    "<|special_1653|>",
+    "<|special_1654|>",
+    "<|special_1655|>",
+    "<|special_1656|>",
+    "<|special_1657|>",
+    "<|special_1658|>",
+    "<|special_1659|>",
+    "<|special_1660|>",
+    "<|special_1661|>",
+    "<|special_1662|>",
+    "<|special_1663|>",
+    "<|special_1664|>",
+    "<|special_1665|>",
+    "<|special_1666|>",
+    "<|special_1667|>",
+    "<|special_1668|>",
+    "<|special_1669|>",
+    "<|special_1670|>",
+    "<|special_1671|>",
+    "<|special_1672|>",
+    "<|special_1673|>",
+    "<|special_1674|>",
+    "<|special_1675|>",
+    "<|special_1676|>",
+    "<|special_1677|>",
+    "<|special_1678|>",
+    "<|special_1679|>",
+    "<|special_1680|>",
+    "<|special_1681|>",
+    "<|special_1682|>",
+    "<|special_1683|>",
+    "<|special_1684|>",
+    "<|special_1685|>",
+    "<|special_1686|>",
+    "<|special_1687|>",
+    "<|special_1688|>",
+    "<|special_1689|>",
+    "<|special_1690|>",
+    "<|special_1691|>",
+    "<|special_1692|>",
+    "<|special_1693|>",
+    "<|special_1694|>",
+    "<|special_1695|>",
+    "<|special_1696|>",
+    "<|special_1697|>",
+    "<|special_1698|>",
+    "<|special_1699|>",
+    "<|special_1700|>",
+    "<|special_1701|>",
+    "<|special_1702|>",
+    "<|special_1703|>",
+    "<|special_1704|>",
+    "<|special_1705|>",
+    "<|special_1706|>",
+    "<|special_1707|>",
+    "<|special_1708|>",
+    "<|special_1709|>",
+    "<|special_1710|>",
+    "<|special_1711|>",
+    "<|special_1712|>",
+    "<|special_1713|>",
+    "<|special_1714|>",
+    "<|special_1715|>",
+    "<|special_1716|>",
+    "<|special_1717|>",
+    "<|special_1718|>",
+    "<|special_1719|>",
+    "<|special_1720|>",
+    "<|special_1721|>",
+    "<|special_1722|>",
+    "<|special_1723|>",
+    "<|special_1724|>",
+    "<|special_1725|>",
+    "<|special_1726|>",
+    "<|special_1727|>",
+    "<|special_1728|>",
+    "<|special_1729|>",
+    "<|special_1730|>",
+    "<|special_1731|>",
+    "<|special_1732|>",
+    "<|special_1733|>",
+    "<|special_1734|>",
+    "<|special_1735|>",
+    "<|special_1736|>",
+    "<|special_1737|>",
+    "<|special_1738|>",
+    "<|special_1739|>",
+    "<|special_1740|>",
+    "<|special_1741|>",
+    "<|special_1742|>",
+    "<|special_1743|>",
+    "<|special_1744|>",
+    "<|special_1745|>",
+    "<|special_1746|>",
+    "<|special_1747|>",
+    "<|special_1748|>",
+    "<|special_1749|>",
+    "<|special_1750|>",
+    "<|special_1751|>",
+    "<|special_1752|>",
+    "<|special_1753|>",
+    "<|special_1754|>",
+    "<|special_1755|>",
+    "<|special_1756|>",
+    "<|special_1757|>",
+    "<|special_1758|>",
+    "<|special_1759|>",
+    "<|special_1760|>",
+    "<|special_1761|>",
+    "<|special_1762|>",
+    "<|special_1763|>",
+    "<|special_1764|>",
+    "<|special_1765|>",
+    "<|special_1766|>",
+    "<|special_1767|>",
+    "<|special_1768|>",
+    "<|special_1769|>",
+    "<|special_1770|>",
+    "<|special_1771|>",
+    "<|special_1772|>",
+    "<|special_1773|>",
+    "<|special_1774|>",
+    "<|special_1775|>",
+    "<|special_1776|>",
+    "<|special_1777|>",
+    "<|special_1778|>",
+    "<|special_1779|>",
+    "<|special_1780|>",
+    "<|special_1781|>",
+    "<|special_1782|>",
+    "<|special_1783|>",
+    "<|special_1784|>",
+    "<|special_1785|>",
+    "<|special_1786|>",
+    "<|special_1787|>",
+    "<|special_1788|>",
+    "<|special_1789|>",
+    "<|special_1790|>",
+    "<|special_1791|>",
+    "<|special_1792|>",
+    "<|special_1793|>",
+    "<|special_1794|>",
+    "<|special_1795|>",
+    "<|special_1796|>",
+    "<|special_1797|>",
+    "<|special_1798|>",
+    "<|special_1799|>",
+    "<|special_1800|>",
+    "<|special_1801|>",
+    "<|special_1802|>",
+    "<|special_1803|>",
+    "<|special_1804|>",
+    "<|special_1805|>",
+    "<|special_1806|>",
+    "<|special_1807|>",
+    "<|special_1808|>",
+    "<|special_1809|>",
+    "<|special_1810|>",
+    "<|special_1811|>",
+    "<|special_1812|>",
+    "<|special_1813|>",
+    "<|special_1814|>",
+    "<|special_1815|>",
+    "<|special_1816|>",
+    "<|special_1817|>",
+    "<|special_1818|>",
+    "<|special_1819|>",
+    "<|special_1820|>",
+    "<|special_1821|>",
+    "<|special_1822|>",
+    "<|special_1823|>",
+    "<|special_1824|>",
+    "<|special_1825|>",
+    "<|special_1826|>",
+    "<|special_1827|>",
+    "<|special_1828|>",
+    "<|special_1829|>",
+    "<|special_1830|>",
+    "<|special_1831|>",
+    "<|special_1832|>",
+    "<|special_1833|>",
+    "<|special_1834|>",
+    "<|special_1835|>",
+    "<|special_1836|>",
+    "<|special_1837|>",
+    "<|special_1838|>",
+    "<|special_1839|>",
+    "<|special_1840|>",
+    "<|special_1841|>",
+    "<|special_1842|>",
+    "<|special_1843|>",
+    "<|special_1844|>",
+    "<|special_1845|>",
+    "<|special_1846|>",
+    "<|special_1847|>",
+    "<|special_1848|>",
+    "<|special_1849|>",
+    "<|special_1850|>",
+    "<|special_1851|>",
+    "<|special_1852|>",
+    "<|special_1853|>",
+    "<|special_1854|>",
+    "<|special_1855|>",
+    "<|special_1856|>",
+    "<|special_1857|>",
+    "<|special_1858|>",
+    "<|special_1859|>",
+    "<|special_1860|>",
+    "<|special_1861|>",
+    "<|special_1862|>",
+    "<|special_1863|>",
+    "<|special_1864|>",
+    "<|special_1865|>",
+    "<|special_1866|>",
+    "<|special_1867|>",
+    "<|special_1868|>",
+    "<|special_1869|>",
+    "<|special_1870|>",
+    "<|special_1871|>",
+    "<|special_1872|>",
+    "<|special_1873|>",
+    "<|special_1874|>",
+    "<|special_1875|>",
+    "<|special_1876|>",
+    "<|special_1877|>",
+    "<|special_1878|>",
+    "<|special_1879|>",
+    "<|special_1880|>",
+    "<|special_1881|>",
+    "<|special_1882|>",
+    "<|special_1883|>",
+    "<|special_1884|>",
+    "<|special_1885|>",
+    "<|special_1886|>",
+    "<|special_1887|>",
+    "<|special_1888|>",
+    "<|special_1889|>",
+    "<|special_1890|>",
+    "<|special_1891|>",
+    "<|special_1892|>",
+    "<|special_1893|>",
+    "<|special_1894|>",
+    "<|special_1895|>",
+    "<|special_1896|>",
+    "<|special_1897|>",
+    "<|special_1898|>",
+    "<|special_1899|>",
+    "<|special_1900|>",
+    "<|special_1901|>",
+    "<|special_1902|>",
+    "<|special_1903|>",
+    "<|special_1904|>",
+    "<|special_1905|>",
+    "<|special_1906|>",
+    "<|special_1907|>",
+    "<|special_1908|>",
+    "<|special_1909|>",
+    "<|special_1910|>",
+    "<|special_1911|>",
+    "<|special_1912|>",
+    "<|special_1913|>",
+    "<|special_1914|>",
+    "<|special_1915|>",
+    "<|special_1916|>",
+    "<|special_1917|>",
+    "<|special_1918|>",
+    "<|special_1919|>",
+    "<|special_1920|>",
+    "<|special_1921|>",
+    "<|special_1922|>",
+    "<|special_1923|>",
+    "<|special_1924|>",
+    "<|special_1925|>",
+    "<|special_1926|>",
+    "<|special_1927|>",
+    "<|special_1928|>",
+    "<|special_1929|>",
+    "<|special_1930|>",
+    "<|special_1931|>",
+    "<|special_1932|>",
+    "<|special_1933|>",
+    "<|special_1934|>",
+    "<|special_1935|>",
+    "<|special_1936|>",
+    "<|special_1937|>",
+    "<|special_1938|>",
+    "<|special_1939|>",
+    "<|special_1940|>",
+    "<|special_1941|>",
+    "<|special_1942|>",
+    "<|special_1943|>",
+    "<|special_1944|>",
+    "<|special_1945|>",
+    "<|special_1946|>",
+    "<|special_1947|>",
+    "<|special_1948|>",
+    "<|special_1949|>",
+    "<|special_1950|>",
+    "<|special_1951|>",
+    "<|special_1952|>",
+    "<|special_1953|>",
+    "<|special_1954|>",
+    "<|special_1955|>",
+    "<|special_1956|>",
+    "<|special_1957|>",
+    "<|special_1958|>",
+    "<|special_1959|>",
+    "<|special_1960|>",
+    "<|special_1961|>",
+    "<|special_1962|>",
+    "<|special_1963|>",
+    "<|special_1964|>",
+    "<|special_1965|>",
+    "<|special_1966|>",
+    "<|special_1967|>",
+    "<|special_1968|>",
+    "<|special_1969|>",
+    "<|special_1970|>",
+    "<|special_1971|>",
+    "<|special_1972|>",
+    "<|special_1973|>",
+    "<|special_1974|>",
+    "<|special_1975|>",
+    "<|special_1976|>",
+    "<|special_1977|>",
+    "<|special_1978|>",
+    "<|special_1979|>",
+    "<|special_1980|>",
+    "<|special_1981|>",
+    "<|special_1982|>",
+    "<|special_1983|>",
+    "<|special_1984|>",
+    "<|special_1985|>",
+    "<|special_1986|>",
+    "<|special_1987|>",
+    "<|special_1988|>",
+    "<|special_1989|>",
+    "<|special_1990|>",
+    "<|special_1991|>",
+    "<|special_1992|>",
+    "<|special_1993|>",
+    "<|special_1994|>",
+    "<|special_1995|>",
+    "<|special_1996|>",
+    "<|special_1997|>",
+    "<|special_1998|>",
+    "<|special_1999|>",
+    "<|special_2000|>",
+    "<|special_2001|>",
+    "<|special_2002|>",
+    "<|special_2003|>",
+    "<|special_2004|>",
+    "<|special_2005|>",
+    "<|special_2006|>",
+    "<|special_2007|>",
+    "<|special_2008|>",
+    "<|special_2009|>",
+    "<|special_2010|>",
+    "<|special_2011|>",
+    "<|special_2012|>",
+    "<|special_2013|>",
+    "<|special_2014|>",
+    "<|special_2015|>",
+    "<|special_2016|>",
+    "<|special_2017|>",
+    "<|special_2018|>",
+    "<|special_2019|>",
+    "<|special_2020|>",
+    "<|special_2021|>",
+    "<|special_2022|>",
+    "<|special_2023|>",
+    "<|special_2024|>",
+    "<|special_2025|>",
+    "<|special_2026|>",
+    "<|special_2027|>",
+    "<|special_2028|>",
+    "<|special_2029|>",
+    "<|special_2030|>",
+    "<|special_2031|>",
+    "<|special_2032|>",
+    "<|special_2033|>",
+    "<|special_2034|>",
+    "<|special_2035|>",
+    "<|special_2036|>",
+    "<|special_2037|>",
+    "<|special_2038|>",
+    "<|special_2039|>",
+    "<|special_2040|>",
+    "<|special_2041|>",
+    "<|special_2042|>",
+    "<|special_2043|>",
+    "<|special_2044|>",
+    "<|special_2045|>",
+    "<|special_2046|>",
+    "<|special_2047|>",
+    "<|special_2048|>",
+    "<|special_2049|>",
+    "<|special_2050|>",
+    "<|special_2051|>",
+    "<|special_2052|>",
+    "<|special_2053|>",
+    "<|special_2054|>",
+    "<|special_2055|>",
+    "<|special_2056|>",
+    "<|special_2057|>",
+    "<|special_2058|>",
+    "<|special_2059|>",
+    "<|special_2060|>",
+    "<|special_2061|>",
+    "<|special_2062|>",
+    "<|special_2063|>",
+    "<|special_2064|>",
+    "<|special_2065|>",
+    "<|special_2066|>",
+    "<|special_2067|>",
+    "<|special_2068|>",
+    "<|special_2069|>",
+    "<|special_2070|>",
+    "<|special_2071|>",
+    "<|special_2072|>",
+    "<|special_2073|>",
+    "<|special_2074|>",
+    "<|special_2075|>",
+    "<|special_2076|>",
+    "<|special_2077|>",
+    "<|special_2078|>",
+    "<|special_2079|>",
+    "<|special_2080|>",
+    "<|special_2081|>",
+    "<|special_2082|>",
+    "<|special_2083|>",
+    "<|special_2084|>",
+    "<|special_2085|>",
+    "<|special_2086|>",
+    "<|special_2087|>",
+    "<|special_2088|>",
+    "<|special_2089|>",
+    "<|special_2090|>",
+    "<|special_2091|>",
+    "<|special_2092|>",
+    "<|special_2093|>",
+    "<|special_2094|>",
+    "<|special_2095|>",
+    "<|special_2096|>",
+    "<|special_2097|>",
+    "<|special_2098|>",
+    "<|special_2099|>",
+    "<|special_2100|>",
+    "<|special_2101|>",
+    "<|special_2102|>",
+    "<|special_2103|>",
+    "<|special_2104|>",
+    "<|special_2105|>",
+    "<|special_2106|>",
+    "<|special_2107|>",
+    "<|special_2108|>",
+    "<|special_2109|>",
+    "<|special_2110|>",
+    "<|special_2111|>",
+    "<|special_2112|>",
+    "<|special_2113|>",
+    "<|special_2114|>",
+    "<|special_2115|>",
+    "<|special_2116|>",
+    "<|special_2117|>",
+    "<|special_2118|>",
+    "<|special_2119|>",
+    "<|special_2120|>",
+    "<|special_2121|>",
+    "<|special_2122|>",
+    "<|special_2123|>",
+    "<|special_2124|>",
+    "<|special_2125|>",
+    "<|special_2126|>",
+    "<|special_2127|>",
+    "<|special_2128|>",
+    "<|special_2129|>",
+    "<|special_2130|>",
+    "<|special_2131|>",
+    "<|special_2132|>",
+    "<|special_2133|>",
+    "<|special_2134|>",
+    "<|special_2135|>",
+    "<|special_2136|>",
+    "<|special_2137|>",
+    "<|special_2138|>",
+    "<|special_2139|>",
+    "<|special_2140|>",
+    "<|special_2141|>",
+    "<|special_2142|>",
+    "<|special_2143|>",
+    "<|special_2144|>",
+    "<|special_2145|>",
+    "<|special_2146|>",
+    "<|special_2147|>",
+    "<|special_2148|>",
+    "<|special_2149|>",
+    "<|special_2150|>",
+    "<|special_2151|>",
+    "<|special_2152|>",
+    "<|special_2153|>",
+    "<|special_2154|>",
+    "<|special_2155|>",
+    "<|special_2156|>",
+    "<|special_2157|>",
+    "<|special_2158|>",
+    "<|special_2159|>",
+    "<|special_2160|>",
+    "<|special_2161|>",
+    "<|special_2162|>",
+    "<|special_2163|>",
+    "<|special_2164|>",
+    "<|special_2165|>",
+    "<|special_2166|>",
+    "<|special_2167|>",
+    "<|special_2168|>",
+    "<|special_2169|>",
+    "<|special_2170|>",
+    "<|special_2171|>",
+    "<|special_2172|>",
+    "<|special_2173|>",
+    "<|special_2174|>",
+    "<|special_2175|>",
+    "<|special_2176|>",
+    "<|special_2177|>",
+    "<|special_2178|>",
+    "<|special_2179|>",
+    "<|special_2180|>",
+    "<|special_2181|>",
+    "<|special_2182|>",
+    "<|special_2183|>",
+    "<|special_2184|>",
+    "<|special_2185|>",
+    "<|special_2186|>",
+    "<|special_2187|>",
+    "<|special_2188|>",
+    "<|special_2189|>",
+    "<|special_2190|>",
+    "<|special_2191|>",
+    "<|special_2192|>",
+    "<|special_2193|>",
+    "<|special_2194|>",
+    "<|special_2195|>",
+    "<|special_2196|>",
+    "<|special_2197|>",
+    "<|special_2198|>",
+    "<|special_2199|>",
+    "<|special_2200|>",
+    "<|special_2201|>",
+    "<|special_2202|>",
+    "<|special_2203|>",
+    "<|special_2204|>",
+    "<|special_2205|>",
+    "<|special_2206|>",
+    "<|special_2207|>",
+    "<|special_2208|>",
+    "<|special_2209|>",
+    "<|special_2210|>",
+    "<|special_2211|>",
+    "<|special_2212|>",
+    "<|special_2213|>",
+    "<|special_2214|>",
+    "<|special_2215|>",
+    "<|special_2216|>",
+    "<|special_2217|>",
+    "<|special_2218|>",
+    "<|special_2219|>",
+    "<|special_2220|>",
+    "<|special_2221|>",
+    "<|special_2222|>",
+    "<|special_2223|>",
+    "<|special_2224|>",
+    "<|special_2225|>",
+    "<|special_2226|>",
+    "<|special_2227|>",
+    "<|special_2228|>",
+    "<|special_2229|>",
+    "<|special_2230|>",
+    "<|special_2231|>",
+    "<|special_2232|>",
+    "<|special_2233|>",
+    "<|special_2234|>",
+    "<|special_2235|>",
+    "<|special_2236|>",
+    "<|special_2237|>",
+    "<|special_2238|>",
+    "<|special_2239|>",
+    "<|special_2240|>",
+    "<|special_2241|>",
+    "<|special_2242|>",
+    "<|special_2243|>",
+    "<|special_2244|>",
+    "<|special_2245|>",
+    "<|special_2246|>",
+    "<|special_2247|>",
+    "<|special_2248|>",
+    "<|special_2249|>",
+    "<|special_2250|>",
+    "<|special_2251|>",
+    "<|special_2252|>",
+    "<|special_2253|>",
+    "<|special_2254|>",
+    "<|special_2255|>",
+    "<|special_2256|>",
+    "<|special_2257|>",
+    "<|special_2258|>",
+    "<|special_2259|>",
+    "<|special_2260|>",
+    "<|special_2261|>",
+    "<|special_2262|>",
+    "<|special_2263|>",
+    "<|special_2264|>",
+    "<|special_2265|>",
+    "<|special_2266|>",
+    "<|special_2267|>",
+    "<|special_2268|>",
+    "<|special_2269|>",
+    "<|special_2270|>",
+    "<|special_2271|>",
+    "<|special_2272|>",
+    "<|special_2273|>",
+    "<|special_2274|>",
+    "<|special_2275|>",
+    "<|special_2276|>",
+    "<|special_2277|>",
+    "<|special_2278|>",
+    "<|special_2279|>",
+    "<|special_2280|>",
+    "<|special_2281|>",
+    "<|special_2282|>",
+    "<|special_2283|>",
+    "<|special_2284|>",
+    "<|special_2285|>",
+    "<|special_2286|>",
+    "<|special_2287|>",
+    "<|special_2288|>",
+    "<|special_2289|>",
+    "<|special_2290|>",
+    "<|special_2291|>",
+    "<|special_2292|>",
+    "<|special_2293|>",
+    "<|special_2294|>",
+    "<|special_2295|>",
+    "<|special_2296|>",
+    "<|special_2297|>",
+    "<|special_2298|>",
+    "<|special_2299|>",
+    "<|special_2300|>",
+    "<|special_2301|>",
+    "<|special_2302|>",
+    "<|special_2303|>",
+    "<|special_2304|>",
+    "<|special_2305|>",
+    "<|special_2306|>",
+    "<|special_2307|>",
+    "<|special_2308|>",
+    "<|special_2309|>",
+    "<|special_2310|>",
+    "<|special_2311|>",
+    "<|special_2312|>",
+    "<|special_2313|>",
+    "<|special_2314|>",
+    "<|special_2315|>",
+    "<|special_2316|>",
+    "<|special_2317|>",
+    "<|special_2318|>",
+    "<|special_2319|>",
+    "<|special_2320|>",
+    "<|special_2321|>",
+    "<|special_2322|>",
+    "<|special_2323|>",
+    "<|special_2324|>",
+    "<|special_2325|>",
+    "<|special_2326|>",
+    "<|special_2327|>",
+    "<|special_2328|>",
+    "<|special_2329|>",
+    "<|special_2330|>",
+    "<|special_2331|>",
+    "<|special_2332|>",
+    "<|special_2333|>",
+    "<|special_2334|>",
+    "<|special_2335|>",
+    "<|special_2336|>",
+    "<|special_2337|>",
+    "<|special_2338|>",
+    "<|special_2339|>",
+    "<|special_2340|>",
+    "<|special_2341|>",
+    "<|special_2342|>",
+    "<|special_2343|>",
+    "<|special_2344|>",
+    "<|special_2345|>",
+    "<|special_2346|>",
+    "<|special_2347|>",
+    "<|special_2348|>",
+    "<|special_2349|>",
+    "<|special_2350|>",
+    "<|special_2351|>",
+    "<|special_2352|>",
+    "<|special_2353|>",
+    "<|special_2354|>",
+    "<|special_2355|>",
+    "<|special_2356|>",
+    "<|special_2357|>",
+    "<|special_2358|>",
+    "<|special_2359|>",
+    "<|special_2360|>",
+    "<|special_2361|>",
+    "<|special_2362|>",
+    "<|special_2363|>",
+    "<|special_2364|>",
+    "<|special_2365|>",
+    "<|special_2366|>",
+    "<|special_2367|>",
+    "<|special_2368|>",
+    "<|special_2369|>",
+    "<|special_2370|>",
+    "<|special_2371|>",
+    "<|special_2372|>",
+    "<|special_2373|>",
+    "<|special_2374|>",
+    "<|special_2375|>",
+    "<|special_2376|>",
+    "<|special_2377|>",
+    "<|special_2378|>",
+    "<|special_2379|>",
+    "<|special_2380|>",
+    "<|special_2381|>",
+    "<|special_2382|>",
+    "<|special_2383|>",
+    "<|special_2384|>",
+    "<|special_2385|>",
+    "<|special_2386|>",
+    "<|special_2387|>",
+    "<|special_2388|>",
+    "<|special_2389|>",
+    "<|special_2390|>",
+    "<|special_2391|>",
+    "<|special_2392|>",
+    "<|special_2393|>",
+    "<|special_2394|>",
+    "<|special_2395|>",
+    "<|special_2396|>",
+    "<|special_2397|>",
+    "<|special_2398|>",
+    "<|special_2399|>",
+    "<|special_2400|>",
+    "<|special_2401|>",
+    "<|special_2402|>",
+    "<|special_2403|>",
+    "<|special_2404|>",
+    "<|special_2405|>",
+    "<|special_2406|>",
+    "<|special_2407|>",
+    "<|special_2408|>",
+    "<|special_2409|>",
+    "<|special_2410|>",
+    "<|special_2411|>",
+    "<|special_2412|>",
+    "<|special_2413|>",
+    "<|special_2414|>",
+    "<|special_2415|>",
+    "<|special_2416|>",
+    "<|special_2417|>",
+    "<|special_2418|>",
+    "<|special_2419|>",
+    "<|special_2420|>",
+    "<|special_2421|>",
+    "<|special_2422|>",
+    "<|special_2423|>",
+    "<|special_2424|>",
+    "<|special_2425|>",
+    "<|special_2426|>",
+    "<|special_2427|>",
+    "<|special_2428|>",
+    "<|special_2429|>",
+    "<|special_2430|>",
+    "<|special_2431|>",
+    "<|special_2432|>",
+    "<|special_2433|>",
+    "<|special_2434|>",
+    "<|special_2435|>",
+    "<|special_2436|>",
+    "<|special_2437|>",
+    "<|special_2438|>",
+    "<|special_2439|>",
+    "<|special_2440|>",
+    "<|special_2441|>",
+    "<|special_2442|>",
+    "<|special_2443|>",
+    "<|special_2444|>",
+    "<|special_2445|>",
+    "<|special_2446|>",
+    "<|special_2447|>",
+    "<|special_2448|>",
+    "<|special_2449|>",
+    "<|special_2450|>",
+    "<|special_2451|>",
+    "<|special_2452|>",
+    "<|special_2453|>",
+    "<|special_2454|>",
+    "<|special_2455|>",
+    "<|special_2456|>",
+    "<|special_2457|>",
+    "<|special_2458|>",
+    "<|special_2459|>",
+    "<|special_2460|>",
+    "<|special_2461|>",
+    "<|special_2462|>",
+    "<|special_2463|>",
+    "<|special_2464|>",
+    "<|special_2465|>",
+    "<|special_2466|>",
+    "<|special_2467|>",
+    "<|special_2468|>",
+    "<|special_2469|>",
+    "<|special_2470|>",
+    "<|special_2471|>",
+    "<|special_2472|>",
+    "<|special_2473|>",
+    "<|special_2474|>",
+    "<|special_2475|>",
+    "<|special_2476|>",
+    "<|special_2477|>",
+    "<|special_2478|>",
+    "<|special_2479|>",
+    "<|special_2480|>",
+    "<|special_2481|>",
+    "<|special_2482|>",
+    "<|special_2483|>",
+    "<|special_2484|>",
+    "<|special_2485|>",
+    "<|special_2486|>",
+    "<|special_2487|>",
+    "<|special_2488|>",
+    "<|special_2489|>",
+    "<|special_2490|>",
+    "<|special_2491|>",
+    "<|special_2492|>",
+    "<|special_2493|>",
+    "<|special_2494|>",
+    "<|special_2495|>",
+    "<|special_2496|>",
+    "<|special_2497|>",
+    "<|special_2498|>",
+    "<|special_2499|>",
+    "<|special_2500|>",
+    "<|special_2501|>",
+    "<|special_2502|>",
+    "<|special_2503|>",
+    "<|special_2504|>",
+    "<|special_2505|>",
+    "<|special_2506|>",
+    "<|special_2507|>",
+    "<|special_2508|>",
+    "<|special_2509|>",
+    "<|special_2510|>",
+    "<|special_2511|>",
+    "<|special_2512|>",
+    "<|special_2513|>",
+    "<|special_2514|>",
+    "<|special_2515|>",
+    "<|special_2516|>",
+    "<|special_2517|>",
+    "<|special_2518|>",
+    "<|special_2519|>",
+    "<|special_2520|>",
+    "<|special_2521|>",
+    "<|special_2522|>",
+    "<|special_2523|>",
+    "<|special_2524|>",
+    "<|special_2525|>",
+    "<|special_2526|>",
+    "<|special_2527|>",
+    "<|special_2528|>",
+    "<|special_2529|>",
+    "<|special_2530|>",
+    "<|special_2531|>",
+    "<|special_2532|>",
+    "<|special_2533|>",
+    "<|special_2534|>",
+    "<|special_2535|>",
+    "<|special_2536|>",
+    "<|special_2537|>",
+    "<|special_2538|>",
+    "<|special_2539|>",
+    "<|special_2540|>",
+    "<|special_2541|>",
+    "<|special_2542|>",
+    "<|special_2543|>",
+    "<|special_2544|>",
+    "<|special_2545|>",
+    "<|special_2546|>",
+    "<|special_2547|>",
+    "<|special_2548|>",
+    "<|special_2549|>",
+    "<|special_2550|>",
+    "<|special_2551|>",
+    "<|special_2552|>",
+    "<|special_2553|>",
+    "<|special_2554|>",
+    "<|special_2555|>",
+    "<|special_2556|>",
+    "<|special_2557|>",
+    "<|special_2558|>",
+    "<|special_2559|>",
+    "<|special_2560|>",
+    "<|special_2561|>",
+    "<|special_2562|>",
+    "<|special_2563|>",
+    "<|special_2564|>",
+    "<|special_2565|>",
+    "<|special_2566|>",
+    "<|special_2567|>",
+    "<|special_2568|>",
+    "<|special_2569|>",
+    "<|special_2570|>",
+    "<|special_2571|>",
+    "<|special_2572|>",
+    "<|special_2573|>",
+    "<|special_2574|>",
+    "<|special_2575|>",
+    "<|special_2576|>",
+    "<|special_2577|>",
+    "<|special_2578|>",
+    "<|special_2579|>",
+    "<|special_2580|>",
+    "<|special_2581|>",
+    "<|special_2582|>",
+    "<|special_2583|>",
+    "<|special_2584|>",
+    "<|special_2585|>",
+    "<|special_2586|>",
+    "<|special_2587|>",
+    "<|special_2588|>",
+    "<|special_2589|>",
+    "<|special_2590|>",
+    "<|special_2591|>",
+    "<|special_2592|>",
+    "<|special_2593|>",
+    "<|special_2594|>",
+    "<|special_2595|>",
+    "<|special_2596|>",
+    "<|special_2597|>",
+    "<|special_2598|>",
+    "<|special_2599|>",
+    "<|special_2600|>",
+    "<|special_2601|>",
+    "<|special_2602|>",
+    "<|special_2603|>",
+    "<|special_2604|>",
+    "<|special_2605|>",
+    "<|special_2606|>",
+    "<|special_2607|>",
+    "<|special_2608|>",
+    "<|special_2609|>",
+    "<|special_2610|>",
+    "<|special_2611|>",
+    "<|special_2612|>",
+    "<|special_2613|>",
+    "<|special_2614|>",
+    "<|special_2615|>",
+    "<|special_2616|>",
+    "<|special_2617|>",
+    "<|special_2618|>",
+    "<|special_2619|>",
+    "<|special_2620|>",
+    "<|special_2621|>",
+    "<|special_2622|>",
+    "<|special_2623|>",
+    "<|special_2624|>",
+    "<|special_2625|>",
+    "<|special_2626|>",
+    "<|special_2627|>",
+    "<|special_2628|>",
+    "<|special_2629|>",
+    "<|special_2630|>",
+    "<|special_2631|>",
+    "<|special_2632|>",
+    "<|special_2633|>",
+    "<|special_2634|>",
+    "<|special_2635|>",
+    "<|special_2636|>",
+    "<|special_2637|>",
+    "<|special_2638|>",
+    "<|special_2639|>",
+    "<|special_2640|>",
+    "<|special_2641|>",
+    "<|special_2642|>",
+    "<|special_2643|>",
+    "<|special_2644|>",
+    "<|special_2645|>",
+    "<|special_2646|>",
+    "<|special_2647|>",
+    "<|special_2648|>",
+    "<|special_2649|>",
+    "<|special_2650|>",
+    "<|special_2651|>",
+    "<|special_2652|>",
+    "<|special_2653|>",
+    "<|special_2654|>",
+    "<|special_2655|>",
+    "<|special_2656|>",
+    "<|special_2657|>",
+    "<|special_2658|>",
+    "<|special_2659|>",
+    "<|special_2660|>",
+    "<|special_2661|>",
+    "<|special_2662|>",
+    "<|special_2663|>",
+    "<|special_2664|>",
+    "<|special_2665|>",
+    "<|special_2666|>",
+    "<|special_2667|>",
+    "<|special_2668|>",
+    "<|special_2669|>",
+    "<|special_2670|>",
+    "<|special_2671|>",
+    "<|special_2672|>",
+    "<|special_2673|>",
+    "<|special_2674|>",
+    "<|special_2675|>",
+    "<|special_2676|>",
+    "<|special_2677|>",
+    "<|special_2678|>",
+    "<|special_2679|>",
+    "<|special_2680|>",
+    "<|special_2681|>",
+    "<|special_2682|>",
+    "<|special_2683|>",
+    "<|special_2684|>",
+    "<|special_2685|>",
+    "<|special_2686|>",
+    "<|special_2687|>",
+    "<|special_2688|>",
+    "<|special_2689|>",
+    "<|special_2690|>",
+    "<|special_2691|>",
+    "<|special_2692|>",
+    "<|special_2693|>",
+    "<|special_2694|>",
+    "<|special_2695|>",
+    "<|special_2696|>",
+    "<|special_2697|>",
+    "<|special_2698|>",
+    "<|special_2699|>",
+    "<|special_2700|>",
+    "<|special_2701|>",
+    "<|special_2702|>",
+    "<|special_2703|>",
+    "<|special_2704|>",
+    "<|special_2705|>",
+    "<|special_2706|>",
+    "<|special_2707|>",
+    "<|special_2708|>",
+    "<|special_2709|>",
+    "<|special_2710|>",
+    "<|special_2711|>",
+    "<|special_2712|>",
+    "<|special_2713|>",
+    "<|special_2714|>",
+    "<|special_2715|>",
+    "<|special_2716|>",
+    "<|special_2717|>",
+    "<|special_2718|>",
+    "<|special_2719|>",
+    "<|special_2720|>",
+    "<|special_2721|>",
+    "<|special_2722|>",
+    "<|special_2723|>",
+    "<|special_2724|>",
+    "<|special_2725|>",
+    "<|special_2726|>",
+    "<|special_2727|>",
+    "<|special_2728|>",
+    "<|special_2729|>",
+    "<|special_2730|>",
+    "<|special_2731|>",
+    "<|special_2732|>",
+    "<|special_2733|>",
+    "<|special_2734|>",
+    "<|special_2735|>",
+    "<|special_2736|>",
+    "<|special_2737|>",
+    "<|special_2738|>",
+    "<|special_2739|>",
+    "<|special_2740|>",
+    "<|special_2741|>",
+    "<|special_2742|>",
+    "<|special_2743|>",
+    "<|special_2744|>",
+    "<|special_2745|>",
+    "<|special_2746|>",
+    "<|special_2747|>",
+    "<|special_2748|>",
+    "<|special_2749|>",
+    "<|special_2750|>",
+    "<|special_2751|>",
+    "<|special_2752|>",
+    "<|special_2753|>",
+    "<|special_2754|>",
+    "<|special_2755|>",
+    "<|special_2756|>",
+    "<|special_2757|>",
+    "<|special_2758|>",
+    "<|special_2759|>",
+    "<|special_2760|>",
+    "<|special_2761|>",
+    "<|special_2762|>",
+    "<|special_2763|>",
+    "<|special_2764|>",
+    "<|special_2765|>",
+    "<|special_2766|>",
+    "<|special_2767|>",
+    "<|special_2768|>",
+    "<|special_2769|>",
+    "<|special_2770|>",
+    "<|special_2771|>",
+    "<|special_2772|>",
+    "<|special_2773|>",
+    "<|special_2774|>",
+    "<|special_2775|>",
+    "<|special_2776|>",
+    "<|special_2777|>",
+    "<|special_2778|>",
+    "<|special_2779|>",
+    "<|special_2780|>",
+    "<|special_2781|>",
+    "<|special_2782|>",
+    "<|special_2783|>",
+    "<|special_2784|>",
+    "<|special_2785|>",
+    "<|special_2786|>",
+    "<|special_2787|>",
+    "<|special_2788|>",
+    "<|special_2789|>",
+    "<|special_2790|>",
+    "<|special_2791|>",
+    "<|special_2792|>",
+    "<|special_2793|>",
+    "<|special_2794|>",
+    "<|special_2795|>",
+    "<|special_2796|>",
+    "<|special_2797|>",
+    "<|special_2798|>",
+    "<|special_2799|>",
+    "<|special_2800|>",
+    "<|special_2801|>",
+    "<|special_2802|>",
+    "<|special_2803|>",
+    "<|special_2804|>",
+    "<|special_2805|>",
+    "<|special_2806|>",
+    "<|special_2807|>",
+    "<|special_2808|>",
+    "<|special_2809|>",
+    "<|special_2810|>",
+    "<|special_2811|>",
+    "<|special_2812|>",
+    "<|special_2813|>",
+    "<|special_2814|>",
+    "<|special_2815|>",
+    "<|special_2816|>",
+    "<|special_2817|>",
+    "<|special_2818|>",
+    "<|special_2819|>",
+    "<|special_2820|>",
+    "<|special_2821|>",
+    "<|special_2822|>",
+    "<|special_2823|>",
+    "<|special_2824|>",
+    "<|special_2825|>",
+    "<|special_2826|>",
+    "<|special_2827|>",
+    "<|special_2828|>",
+    "<|special_2829|>",
+    "<|special_2830|>",
+    "<|special_2831|>",
+    "<|special_2832|>",
+    "<|special_2833|>",
+    "<|special_2834|>",
+    "<|special_2835|>",
+    "<|special_2836|>",
+    "<|special_2837|>",
+    "<|special_2838|>",
+    "<|special_2839|>",
+    "<|special_2840|>",
+    "<|special_2841|>",
+    "<|special_2842|>",
+    "<|special_2843|>",
+    "<|special_2844|>",
+    "<|special_2845|>",
+    "<|special_2846|>",
+    "<|special_2847|>",
+    "<|special_2848|>",
+    "<|special_2849|>",
+    "<|special_2850|>",
+    "<|special_2851|>",
+    "<|special_2852|>",
+    "<|special_2853|>",
+    "<|special_2854|>",
+    "<|special_2855|>",
+    "<|special_2856|>",
+    "<|special_2857|>",
+    "<|special_2858|>",
+    "<|special_2859|>",
+    "<|special_2860|>",
+    "<|special_2861|>",
+    "<|special_2862|>",
+    "<|special_2863|>",
+    "<|special_2864|>",
+    "<|special_2865|>",
+    "<|special_2866|>",
+    "<|special_2867|>",
+    "<|special_2868|>",
+    "<|special_2869|>",
+    "<|special_2870|>",
+    "<|special_2871|>",
+    "<|special_2872|>",
+    "<|special_2873|>",
+    "<|special_2874|>",
+    "<|special_2875|>",
+    "<|special_2876|>",
+    "<|special_2877|>",
+    "<|special_2878|>",
+    "<|special_2879|>",
+    "<|special_2880|>",
+    "<|special_2881|>",
+    "<|special_2882|>",
+    "<|special_2883|>",
+    "<|special_2884|>",
+    "<|special_2885|>",
+    "<|special_2886|>",
+    "<|special_2887|>",
+    "<|special_2888|>",
+    "<|special_2889|>",
+    "<|special_2890|>",
+    "<|special_2891|>",
+    "<|special_2892|>",
+    "<|special_2893|>",
+    "<|special_2894|>",
+    "<|special_2895|>",
+    "<|special_2896|>",
+    "<|special_2897|>",
+    "<|special_2898|>",
+    "<|special_2899|>",
+    "<|special_2900|>",
+    "<|special_2901|>",
+    "<|special_2902|>",
+    "<|special_2903|>",
+    "<|special_2904|>",
+    "<|special_2905|>",
+    "<|special_2906|>",
+    "<|special_2907|>",
+    "<|special_2908|>",
+    "<|special_2909|>",
+    "<|special_2910|>",
+    "<|special_2911|>",
+    "<|special_2912|>",
+    "<|special_2913|>",
+    "<|special_2914|>",
+    "<|special_2915|>",
+    "<|special_2916|>",
+    "<|special_2917|>",
+    "<|special_2918|>",
+    "<|special_2919|>",
+    "<|special_2920|>",
+    "<|special_2921|>",
+    "<|special_2922|>",
+    "<|special_2923|>",
+    "<|special_2924|>",
+    "<|special_2925|>",
+    "<|special_2926|>",
+    "<|special_2927|>",
+    "<|special_2928|>",
+    "<|special_2929|>",
+    "<|special_2930|>",
+    "<|special_2931|>",
+    "<|special_2932|>",
+    "<|special_2933|>",
+    "<|special_2934|>",
+    "<|special_2935|>",
+    "<|special_2936|>",
+    "<|special_2937|>",
+    "<|special_2938|>",
+    "<|special_2939|>",
+    "<|special_2940|>",
+    "<|special_2941|>",
+    "<|special_2942|>",
+    "<|special_2943|>",
+    "<|special_2944|>",
+    "<|special_2945|>",
+    "<|special_2946|>",
+    "<|special_2947|>",
+    "<|special_2948|>",
+    "<|special_2949|>",
+    "<|special_2950|>",
+    "<|special_2951|>",
+    "<|special_2952|>",
+    "<|special_2953|>",
+    "<|special_2954|>",
+    "<|special_2955|>",
+    "<|special_2956|>",
+    "<|special_2957|>",
+    "<|special_2958|>",
+    "<|special_2959|>",
+    "<|special_2960|>",
+    "<|special_2961|>",
+    "<|special_2962|>",
+    "<|special_2963|>",
+    "<|special_2964|>",
+    "<|special_2965|>",
+    "<|special_2966|>",
+    "<|special_2967|>",
+    "<|special_2968|>",
+    "<|special_2969|>",
+    "<|special_2970|>",
+    "<|special_2971|>",
+    "<|special_2972|>",
+    "<|special_2973|>",
+    "<|special_2974|>",
+    "<|special_2975|>",
+    "<|special_2976|>",
+    "<|special_2977|>",
+    "<|special_2978|>",
+    "<|special_2979|>",
+    "<|special_2980|>",
+    "<|special_2981|>",
+    "<|special_2982|>",
+    "<|special_2983|>",
+    "<|special_2984|>",
+    "<|special_2985|>",
+    "<|special_2986|>",
+    "<|special_2987|>",
+    "<|special_2988|>",
+    "<|special_2989|>",
+    "<|special_2990|>",
+    "<|special_2991|>",
+    "<|special_2992|>",
+    "<|special_2993|>",
+    "<|special_2994|>",
+    "<|special_2995|>",
+    "<|special_2996|>",
+    "<|special_2997|>",
+    "<|special_2998|>",
+    "<|special_2999|>",
+    "<|special_3000|>",
+    "<|special_3001|>",
+    "<|special_3002|>",
+    "<|special_3003|>",
+    "<|special_3004|>",
+    "<|special_3005|>",
+    "<|special_3006|>",
+    "<|special_3007|>",
+    "<|special_3008|>",
+    "<|special_3009|>",
+    "<|special_3010|>",
+    "<|special_3011|>",
+    "<|special_3012|>",
+    "<|special_3013|>",
+    "<|special_3014|>",
+    "<|special_3015|>",
+    "<|special_3016|>",
+    "<|special_3017|>",
+    "<|special_3018|>",
+    "<|special_3019|>",
+    "<|special_3020|>",
+    "<|special_3021|>",
+    "<|special_3022|>",
+    "<|special_3023|>",
+    "<|special_3024|>",
+    "<|special_3025|>",
+    "<|special_3026|>",
+    "<|special_3027|>",
+    "<|special_3028|>",
+    "<|special_3029|>",
+    "<|special_3030|>",
+    "<|special_3031|>",
+    "<|special_3032|>",
+    "<|special_3033|>",
+    "<|special_3034|>",
+    "<|special_3035|>",
+    "<|special_3036|>",
+    "<|special_3037|>",
+    "<|special_3038|>",
+    "<|special_3039|>",
+    "<|special_3040|>",
+    "<|special_3041|>",
+    "<|special_3042|>",
+    "<|special_3043|>",
+    "<|special_3044|>",
+    "<|special_3045|>",
+    "<|special_3046|>",
+    "<|special_3047|>",
+    "<|special_3048|>",
+    "<|special_3049|>",
+    "<|special_3050|>",
+    "<|special_3051|>",
+    "<|special_3052|>",
+    "<|special_3053|>",
+    "<|special_3054|>",
+    "<|special_3055|>",
+    "<|special_3056|>",
+    "<|special_3057|>",
+    "<|special_3058|>",
+    "<|special_3059|>",
+    "<|special_3060|>",
+    "<|special_3061|>",
+    "<|special_3062|>",
+    "<|special_3063|>",
+    "<|special_3064|>",
+    "<|special_3065|>",
+    "<|special_3066|>",
+    "<|special_3067|>",
+    "<|special_3068|>",
+    "<|special_3069|>",
+    "<|special_3070|>",
+    "<|special_3071|>",
+    "<|special_3072|>",
+    "<|special_3073|>",
+    "<|special_3074|>",
+    "<|special_3075|>",
+    "<|special_3076|>",
+    "<|special_3077|>",
+    "<|special_3078|>",
+    "<|special_3079|>",
+    "<|special_3080|>",
+    "<|special_3081|>",
+    "<|special_3082|>",
+    "<|special_3083|>",
+    "<|special_3084|>",
+    "<|special_3085|>",
+    "<|special_3086|>",
+    "<|special_3087|>",
+    "<|special_3088|>",
+    "<|special_3089|>",
+    "<|special_3090|>",
+    "<|special_3091|>",
+    "<|special_3092|>",
+    "<|special_3093|>",
+    "<|special_3094|>",
+    "<|special_3095|>",
+    "<|special_3096|>",
+    "<|special_3097|>",
+    "<|special_3098|>",
+    "<|special_3099|>",
+    "<|special_3100|>",
+    "<|special_3101|>",
+    "<|special_3102|>",
+    "<|special_3103|>",
+    "<|special_3104|>",
+    "<|special_3105|>",
+    "<|special_3106|>",
+    "<|special_3107|>",
+    "<|special_3108|>",
+    "<|special_3109|>",
+    "<|special_3110|>",
+    "<|special_3111|>",
+    "<|special_3112|>",
+    "<|special_3113|>",
+    "<|special_3114|>",
+    "<|special_3115|>",
+    "<|special_3116|>",
+    "<|special_3117|>",
+    "<|special_3118|>",
+    "<|special_3119|>",
+    "<|special_3120|>",
+    "<|special_3121|>",
+    "<|special_3122|>",
+    "<|special_3123|>",
+    "<|special_3124|>",
+    "<|special_3125|>",
+    "<|special_3126|>",
+    "<|special_3127|>",
+    "<|special_3128|>",
+    "<|special_3129|>",
+    "<|special_3130|>",
+    "<|special_3131|>",
+    "<|special_3132|>",
+    "<|special_3133|>",
+    "<|special_3134|>",
+    "<|special_3135|>",
+    "<|special_3136|>",
+    "<|special_3137|>",
+    "<|special_3138|>",
+    "<|special_3139|>",
+    "<|special_3140|>",
+    "<|special_3141|>",
+    "<|special_3142|>",
+    "<|special_3143|>",
+    "<|special_3144|>",
+    "<|special_3145|>",
+    "<|special_3146|>",
+    "<|special_3147|>",
+    "<|special_3148|>",
+    "<|special_3149|>",
+    "<|special_3150|>",
+    "<|special_3151|>",
+    "<|special_3152|>",
+    "<|special_3153|>",
+    "<|special_3154|>",
+    "<|special_3155|>",
+    "<|special_3156|>",
+    "<|special_3157|>",
+    "<|special_3158|>",
+    "<|special_3159|>",
+    "<|special_3160|>",
+    "<|special_3161|>",
+    "<|special_3162|>",
+    "<|special_3163|>",
+    "<|special_3164|>",
+    "<|special_3165|>",
+    "<|special_3166|>",
+    "<|special_3167|>",
+    "<|special_3168|>",
+    "<|special_3169|>",
+    "<|special_3170|>",
+    "<|special_3171|>",
+    "<|special_3172|>",
+    "<|special_3173|>",
+    "<|special_3174|>",
+    "<|special_3175|>",
+    "<|special_3176|>",
+    "<|special_3177|>",
+    "<|special_3178|>",
+    "<|special_3179|>",
+    "<|special_3180|>",
+    "<|special_3181|>",
+    "<|special_3182|>",
+    "<|special_3183|>",
+    "<|special_3184|>",
+    "<|special_3185|>",
+    "<|special_3186|>",
+    "<|special_3187|>",
+    "<|special_3188|>",
+    "<|special_3189|>",
+    "<|special_3190|>",
+    "<|special_3191|>",
+    "<|special_3192|>",
+    "<|special_3193|>",
+    "<|special_3194|>",
+    "<|special_3195|>",
+    "<|special_3196|>",
+    "<|special_3197|>",
+    "<|special_3198|>",
+    "<|special_3199|>",
+    "<|special_3200|>",
+    "<|special_3201|>",
+    "<|special_3202|>",
+    "<|special_3203|>",
+    "<|special_3204|>",
+    "<|special_3205|>",
+    "<|special_3206|>",
+    "<|special_3207|>",
+    "<|special_3208|>",
+    "<|special_3209|>",
+    "<|special_3210|>",
+    "<|special_3211|>",
+    "<|special_3212|>",
+    "<|special_3213|>",
+    "<|special_3214|>",
+    "<|special_3215|>",
+    "<|special_3216|>",
+    "<|special_3217|>",
+    "<|special_3218|>",
+    "<|special_3219|>",
+    "<|special_3220|>",
+    "<|special_3221|>",
+    "<|special_3222|>",
+    "<|special_3223|>",
+    "<|special_3224|>",
+    "<|special_3225|>",
+    "<|special_3226|>",
+    "<|special_3227|>",
+    "<|special_3228|>",
+    "<|special_3229|>",
+    "<|special_3230|>",
+    "<|special_3231|>",
+    "<|special_3232|>",
+    "<|special_3233|>",
+    "<|special_3234|>",
+    "<|special_3235|>",
+    "<|special_3236|>",
+    "<|special_3237|>",
+    "<|special_3238|>",
+    "<|special_3239|>",
+    "<|special_3240|>",
+    "<|special_3241|>",
+    "<|special_3242|>",
+    "<|special_3243|>",
+    "<|special_3244|>",
+    "<|special_3245|>",
+    "<|special_3246|>",
+    "<|special_3247|>",
+    "<|special_3248|>",
+    "<|special_3249|>",
+    "<|special_3250|>",
+    "<|special_3251|>",
+    "<|special_3252|>",
+    "<|special_3253|>",
+    "<|special_3254|>",
+    "<|special_3255|>",
+    "<|special_3256|>",
+    "<|special_3257|>",
+    "<|special_3258|>",
+    "<|special_3259|>",
+    "<|special_3260|>",
+    "<|special_3261|>",
+    "<|special_3262|>",
+    "<|special_3263|>",
+    "<|special_3264|>",
+    "<|special_3265|>",
+    "<|special_3266|>",
+    "<|special_3267|>",
+    "<|special_3268|>",
+    "<|special_3269|>",
+    "<|special_3270|>",
+    "<|special_3271|>",
+    "<|special_3272|>",
+    "<|special_3273|>",
+    "<|special_3274|>",
+    "<|special_3275|>",
+    "<|special_3276|>",
+    "<|special_3277|>",
+    "<|special_3278|>",
+    "<|special_3279|>",
+    "<|special_3280|>",
+    "<|special_3281|>",
+    "<|special_3282|>",
+    "<|special_3283|>",
+    "<|special_3284|>",
+    "<|special_3285|>",
+    "<|special_3286|>",
+    "<|special_3287|>",
+    "<|special_3288|>",
+    "<|special_3289|>",
+    "<|special_3290|>",
+    "<|special_3291|>",
+    "<|special_3292|>",
+    "<|special_3293|>",
+    "<|special_3294|>",
+    "<|special_3295|>",
+    "<|special_3296|>",
+    "<|special_3297|>",
+    "<|special_3298|>",
+    "<|special_3299|>",
+    "<|special_3300|>",
+    "<|special_3301|>",
+    "<|special_3302|>",
+    "<|special_3303|>",
+    "<|special_3304|>",
+    "<|special_3305|>",
+    "<|special_3306|>",
+    "<|special_3307|>",
+    "<|special_3308|>",
+    "<|special_3309|>",
+    "<|special_3310|>",
+    "<|special_3311|>",
+    "<|special_3312|>",
+    "<|special_3313|>",
+    "<|special_3314|>",
+    "<|special_3315|>",
+    "<|special_3316|>",
+    "<|special_3317|>",
+    "<|special_3318|>",
+    "<|special_3319|>",
+    "<|special_3320|>",
+    "<|special_3321|>",
+    "<|special_3322|>",
+    "<|special_3323|>",
+    "<|special_3324|>",
+    "<|special_3325|>",
+    "<|special_3326|>",
+    "<|special_3327|>",
+    "<|special_3328|>",
+    "<|special_3329|>",
+    "<|special_3330|>",
+    "<|special_3331|>",
+    "<|special_3332|>",
+    "<|special_3333|>",
+    "<|special_3334|>",
+    "<|special_3335|>",
+    "<|special_3336|>",
+    "<|special_3337|>",
+    "<|special_3338|>",
+    "<|special_3339|>",
+    "<|special_3340|>",
+    "<|special_3341|>",
+    "<|special_3342|>",
+    "<|special_3343|>",
+    "<|special_3344|>",
+    "<|special_3345|>",
+    "<|special_3346|>",
+    "<|special_3347|>",
+    "<|special_3348|>",
+    "<|special_3349|>",
+    "<|special_3350|>",
+    "<|special_3351|>",
+    "<|special_3352|>",
+    "<|special_3353|>",
+    "<|special_3354|>",
+    "<|special_3355|>",
+    "<|special_3356|>",
+    "<|special_3357|>",
+    "<|special_3358|>",
+    "<|special_3359|>",
+    "<|special_3360|>",
+    "<|special_3361|>",
+    "<|special_3362|>",
+    "<|special_3363|>",
+    "<|special_3364|>",
+    "<|special_3365|>",
+    "<|special_3366|>",
+    "<|special_3367|>",
+    "<|special_3368|>",
+    "<|special_3369|>",
+    "<|special_3370|>",
+    "<|special_3371|>",
+    "<|special_3372|>",
+    "<|special_3373|>",
+    "<|special_3374|>",
+    "<|special_3375|>",
+    "<|special_3376|>",
+    "<|special_3377|>",
+    "<|special_3378|>",
+    "<|special_3379|>",
+    "<|special_3380|>",
+    "<|special_3381|>",
+    "<|special_3382|>",
+    "<|special_3383|>",
+    "<|special_3384|>",
+    "<|special_3385|>",
+    "<|special_3386|>",
+    "<|special_3387|>",
+    "<|special_3388|>",
+    "<|special_3389|>",
+    "<|special_3390|>",
+    "<|special_3391|>",
+    "<|special_3392|>",
+    "<|special_3393|>",
+    "<|special_3394|>",
+    "<|special_3395|>",
+    "<|special_3396|>",
+    "<|special_3397|>",
+    "<|special_3398|>",
+    "<|special_3399|>",
+    "<|special_3400|>",
+    "<|special_3401|>",
+    "<|special_3402|>",
+    "<|special_3403|>",
+    "<|special_3404|>",
+    "<|special_3405|>",
+    "<|special_3406|>",
+    "<|special_3407|>",
+    "<|special_3408|>",
+    "<|special_3409|>",
+    "<|special_3410|>",
+    "<|special_3411|>",
+    "<|special_3412|>",
+    "<|special_3413|>",
+    "<|special_3414|>",
+    "<|special_3415|>",
+    "<|special_3416|>",
+    "<|special_3417|>",
+    "<|special_3418|>",
+    "<|special_3419|>",
+    "<|special_3420|>",
+    "<|special_3421|>",
+    "<|special_3422|>",
+    "<|special_3423|>",
+    "<|special_3424|>",
+    "<|special_3425|>",
+    "<|special_3426|>",
+    "<|special_3427|>",
+    "<|special_3428|>",
+    "<|special_3429|>",
+    "<|special_3430|>",
+    "<|special_3431|>",
+    "<|special_3432|>",
+    "<|special_3433|>",
+    "<|special_3434|>",
+    "<|special_3435|>",
+    "<|special_3436|>",
+    "<|special_3437|>",
+    "<|special_3438|>",
+    "<|special_3439|>",
+    "<|special_3440|>",
+    "<|special_3441|>",
+    "<|special_3442|>",
+    "<|special_3443|>",
+    "<|special_3444|>",
+    "<|special_3445|>",
+    "<|special_3446|>",
+    "<|special_3447|>",
+    "<|special_3448|>",
+    "<|special_3449|>",
+    "<|special_3450|>",
+    "<|special_3451|>",
+    "<|special_3452|>",
+    "<|special_3453|>",
+    "<|special_3454|>",
+    "<|special_3455|>",
+    "<|special_3456|>",
+    "<|special_3457|>",
+    "<|special_3458|>",
+    "<|special_3459|>",
+    "<|special_3460|>",
+    "<|special_3461|>",
+    "<|special_3462|>",
+    "<|special_3463|>",
+    "<|special_3464|>",
+    "<|special_3465|>",
+    "<|special_3466|>",
+    "<|special_3467|>",
+    "<|special_3468|>",
+    "<|special_3469|>",
+    "<|special_3470|>",
+    "<|special_3471|>",
+    "<|special_3472|>",
+    "<|special_3473|>",
+    "<|special_3474|>",
+    "<|special_3475|>",
+    "<|special_3476|>",
+    "<|special_3477|>",
+    "<|special_3478|>",
+    "<|special_3479|>",
+    "<|special_3480|>",
+    "<|special_3481|>",
+    "<|special_3482|>",
+    "<|special_3483|>",
+    "<|special_3484|>",
+    "<|special_3485|>",
+    "<|special_3486|>",
+    "<|special_3487|>",
+    "<|special_3488|>",
+    "<|special_3489|>",
+    "<|special_3490|>",
+    "<|special_3491|>",
+    "<|special_3492|>",
+    "<|special_3493|>",
+    "<|special_3494|>",
+    "<|special_3495|>",
+    "<|special_3496|>",
+    "<|special_3497|>",
+    "<|special_3498|>",
+    "<|special_3499|>",
+    "<|special_3500|>",
+    "<|special_3501|>",
+    "<|special_3502|>",
+    "<|special_3503|>",
+    "<|special_3504|>",
+    "<|special_3505|>",
+    "<|special_3506|>",
+    "<|special_3507|>",
+    "<|special_3508|>",
+    "<|special_3509|>",
+    "<|special_3510|>",
+    "<|special_3511|>",
+    "<|special_3512|>",
+    "<|special_3513|>",
+    "<|special_3514|>",
+    "<|special_3515|>",
+    "<|special_3516|>",
+    "<|special_3517|>",
+    "<|special_3518|>",
+    "<|special_3519|>",
+    "<|special_3520|>",
+    "<|special_3521|>",
+    "<|special_3522|>",
+    "<|special_3523|>",
+    "<|special_3524|>",
+    "<|special_3525|>",
+    "<|special_3526|>",
+    "<|special_3527|>",
+    "<|special_3528|>",
+    "<|special_3529|>",
+    "<|special_3530|>",
+    "<|special_3531|>",
+    "<|special_3532|>",
+    "<|special_3533|>",
+    "<|special_3534|>",
+    "<|special_3535|>",
+    "<|special_3536|>",
+    "<|special_3537|>",
+    "<|special_3538|>",
+    "<|special_3539|>",
+    "<|special_3540|>",
+    "<|special_3541|>",
+    "<|special_3542|>",
+    "<|special_3543|>",
+    "<|special_3544|>",
+    "<|special_3545|>",
+    "<|special_3546|>",
+    "<|special_3547|>",
+    "<|special_3548|>",
+    "<|special_3549|>",
+    "<|special_3550|>",
+    "<|special_3551|>",
+    "<|special_3552|>",
+    "<|special_3553|>",
+    "<|special_3554|>",
+    "<|special_3555|>",
+    "<|special_3556|>",
+    "<|special_3557|>",
+    "<|special_3558|>",
+    "<|special_3559|>",
+    "<|special_3560|>",
+    "<|special_3561|>",
+    "<|special_3562|>",
+    "<|special_3563|>",
+    "<|special_3564|>",
+    "<|special_3565|>",
+    "<|special_3566|>",
+    "<|special_3567|>",
+    "<|special_3568|>",
+    "<|special_3569|>",
+    "<|special_3570|>",
+    "<|special_3571|>",
+    "<|special_3572|>",
+    "<|special_3573|>",
+    "<|special_3574|>",
+    "<|special_3575|>",
+    "<|special_3576|>",
+    "<|special_3577|>",
+    "<|special_3578|>",
+    "<|special_3579|>",
+    "<|special_3580|>",
+    "<|special_3581|>",
+    "<|special_3582|>",
+    "<|special_3583|>",
+    "<|special_3584|>",
+    "<|special_3585|>",
+    "<|special_3586|>",
+    "<|special_3587|>",
+    "<|special_3588|>",
+    "<|special_3589|>",
+    "<|special_3590|>",
+    "<|special_3591|>",
+    "<|special_3592|>",
+    "<|special_3593|>",
+    "<|special_3594|>",
+    "<|special_3595|>",
+    "<|special_3596|>",
+    "<|special_3597|>",
+    "<|special_3598|>",
+    "<|special_3599|>",
+    "<|special_3600|>",
+    "<|special_3601|>",
+    "<|special_3602|>",
+    "<|special_3603|>",
+    "<|special_3604|>",
+    "<|special_3605|>",
+    "<|special_3606|>",
+    "<|special_3607|>",
+    "<|special_3608|>",
+    "<|special_3609|>",
+    "<|special_3610|>",
+    "<|special_3611|>",
+    "<|special_3612|>",
+    "<|special_3613|>",
+    "<|special_3614|>",
+    "<|special_3615|>",
+    "<|special_3616|>",
+    "<|special_3617|>",
+    "<|special_3618|>",
+    "<|special_3619|>",
+    "<|special_3620|>",
+    "<|special_3621|>",
+    "<|special_3622|>",
+    "<|special_3623|>",
+    "<|special_3624|>",
+    "<|special_3625|>",
+    "<|special_3626|>",
+    "<|special_3627|>",
+    "<|special_3628|>",
+    "<|special_3629|>",
+    "<|special_3630|>",
+    "<|special_3631|>",
+    "<|special_3632|>",
+    "<|special_3633|>",
+    "<|special_3634|>",
+    "<|special_3635|>",
+    "<|special_3636|>",
+    "<|special_3637|>",
+    "<|special_3638|>",
+    "<|special_3639|>",
+    "<|special_3640|>",
+    "<|special_3641|>",
+    "<|special_3642|>",
+    "<|special_3643|>",
+    "<|special_3644|>",
+    "<|special_3645|>",
+    "<|special_3646|>",
+    "<|special_3647|>",
+    "<|special_3648|>",
+    "<|special_3649|>",
+    "<|special_3650|>",
+    "<|special_3651|>",
+    "<|special_3652|>",
+    "<|special_3653|>",
+    "<|special_3654|>",
+    "<|special_3655|>",
+    "<|special_3656|>",
+    "<|special_3657|>",
+    "<|special_3658|>",
+    "<|special_3659|>",
+    "<|special_3660|>",
+    "<|special_3661|>",
+    "<|special_3662|>",
+    "<|special_3663|>",
+    "<|special_3664|>",
+    "<|special_3665|>",
+    "<|special_3666|>",
+    "<|special_3667|>",
+    "<|special_3668|>",
+    "<|special_3669|>",
+    "<|special_3670|>",
+    "<|special_3671|>",
+    "<|special_3672|>",
+    "<|special_3673|>",
+    "<|special_3674|>",
+    "<|special_3675|>",
+    "<|special_3676|>",
+    "<|special_3677|>",
+    "<|special_3678|>",
+    "<|special_3679|>",
+    "<|special_3680|>",
+    "<|special_3681|>",
+    "<|special_3682|>",
+    "<|special_3683|>",
+    "<|special_3684|>",
+    "<|special_3685|>",
+    "<|special_3686|>",
+    "<|special_3687|>",
+    "<|special_3688|>",
+    "<|special_3689|>",
+    "<|special_3690|>",
+    "<|special_3691|>",
+    "<|special_3692|>",
+    "<|special_3693|>",
+    "<|special_3694|>",
+    "<|special_3695|>",
+    "<|special_3696|>",
+    "<|special_3697|>",
+    "<|special_3698|>",
+    "<|special_3699|>",
+    "<|special_3700|>",
+    "<|special_3701|>",
+    "<|special_3702|>",
+    "<|special_3703|>",
+    "<|special_3704|>",
+    "<|special_3705|>",
+    "<|special_3706|>",
+    "<|special_3707|>",
+    "<|special_3708|>",
+    "<|special_3709|>",
+    "<|special_3710|>",
+    "<|special_3711|>",
+    "<|special_3712|>",
+    "<|special_3713|>",
+    "<|special_3714|>",
+    "<|special_3715|>",
+    "<|special_3716|>",
+    "<|special_3717|>",
+    "<|special_3718|>",
+    "<|special_3719|>",
+    "<|special_3720|>",
+    "<|special_3721|>",
+    "<|special_3722|>",
+    "<|special_3723|>",
+    "<|special_3724|>",
+    "<|special_3725|>",
+    "<|special_3726|>",
+    "<|special_3727|>",
+    "<|special_3728|>",
+    "<|special_3729|>",
+    "<|special_3730|>",
+    "<|special_3731|>",
+    "<|special_3732|>",
+    "<|special_3733|>",
+    "<|special_3734|>",
+    "<|special_3735|>",
+    "<|special_3736|>",
+    "<|special_3737|>",
+    "<|special_3738|>",
+    "<|special_3739|>",
+    "<|special_3740|>",
+    "<|special_3741|>",
+    "<|special_3742|>",
+    "<|special_3743|>",
+    "<|special_3744|>",
+    "<|special_3745|>",
+    "<|special_3746|>",
+    "<|special_3747|>",
+    "<|special_3748|>",
+    "<|special_3749|>",
+    "<|special_3750|>",
+    "<|special_3751|>",
+    "<|special_3752|>",
+    "<|special_3753|>",
+    "<|special_3754|>",
+    "<|special_3755|>",
+    "<|special_3756|>",
+    "<|special_3757|>",
+    "<|special_3758|>",
+    "<|special_3759|>",
+    "<|special_3760|>",
+    "<|special_3761|>",
+    "<|special_3762|>",
+    "<|special_3763|>",
+    "<|special_3764|>",
+    "<|special_3765|>",
+    "<|special_3766|>",
+    "<|special_3767|>",
+    "<|special_3768|>",
+    "<|special_3769|>",
+    "<|special_3770|>",
+    "<|special_3771|>",
+    "<|special_3772|>",
+    "<|special_3773|>",
+    "<|special_3774|>",
+    "<|special_3775|>",
+    "<|special_3776|>",
+    "<|special_3777|>",
+    "<|special_3778|>",
+    "<|special_3779|>",
+    "<|special_3780|>",
+    "<|special_3781|>",
+    "<|special_3782|>",
+    "<|special_3783|>",
+    "<|special_3784|>",
+    "<|special_3785|>",
+    "<|special_3786|>",
+    "<|special_3787|>",
+    "<|special_3788|>",
+    "<|special_3789|>",
+    "<|special_3790|>",
+    "<|special_3791|>",
+    "<|special_3792|>",
+    "<|special_3793|>",
+    "<|special_3794|>",
+    "<|special_3795|>",
+    "<|special_3796|>",
+    "<|special_3797|>",
+    "<|special_3798|>",
+    "<|special_3799|>",
+    "<|special_3800|>",
+    "<|special_3801|>",
+    "<|special_3802|>",
+    "<|special_3803|>",
+    "<|special_3804|>",
+    "<|special_3805|>",
+    "<|special_3806|>",
+    "<|special_3807|>",
+    "<|special_3808|>",
+    "<|special_3809|>",
+    "<|special_3810|>",
+    "<|special_3811|>",
+    "<|special_3812|>",
+    "<|special_3813|>",
+    "<|special_3814|>",
+    "<|special_3815|>",
+    "<|special_3816|>",
+    "<|special_3817|>",
+    "<|special_3818|>",
+    "<|special_3819|>",
+    "<|special_3820|>",
+    "<|special_3821|>",
+    "<|special_3822|>",
+    "<|special_3823|>",
+    "<|special_3824|>",
+    "<|special_3825|>",
+    "<|special_3826|>",
+    "<|special_3827|>",
+    "<|special_3828|>",
+    "<|special_3829|>",
+    "<|special_3830|>",
+    "<|special_3831|>",
+    "<|special_3832|>",
+    "<|special_3833|>",
+    "<|special_3834|>",
+    "<|special_3835|>",
+    "<|special_3836|>",
+    "<|special_3837|>",
+    "<|special_3838|>",
+    "<|special_3839|>",
+    "<|special_3840|>",
+    "<|special_3841|>",
+    "<|special_3842|>",
+    "<|special_3843|>",
+    "<|special_3844|>",
+    "<|special_3845|>",
+    "<|special_3846|>",
+    "<|special_3847|>",
+    "<|special_3848|>",
+    "<|special_3849|>",
+    "<|special_3850|>",
+    "<|special_3851|>",
+    "<|special_3852|>",
+    "<|special_3853|>",
+    "<|special_3854|>",
+    "<|special_3855|>",
+    "<|special_3856|>",
+    "<|special_3857|>",
+    "<|special_3858|>",
+    "<|special_3859|>",
+    "<|special_3860|>",
+    "<|special_3861|>",
+    "<|special_3862|>",
+    "<|special_3863|>",
+    "<|special_3864|>",
+    "<|special_3865|>",
+    "<|special_3866|>",
+    "<|special_3867|>",
+    "<|special_3868|>",
+    "<|special_3869|>",
+    "<|special_3870|>",
+    "<|special_3871|>",
+    "<|special_3872|>",
+    "<|special_3873|>",
+    "<|special_3874|>",
+    "<|special_3875|>",
+    "<|special_3876|>",
+    "<|special_3877|>",
+    "<|special_3878|>",
+    "<|special_3879|>",
+    "<|special_3880|>",
+    "<|special_3881|>",
+    "<|special_3882|>",
+    "<|special_3883|>",
+    "<|special_3884|>",
+    "<|special_3885|>",
+    "<|special_3886|>",
+    "<|special_3887|>",
+    "<|special_3888|>",
+    "<|special_3889|>",
+    "<|special_3890|>",
+    "<|special_3891|>",
+    "<|special_3892|>",
+    "<|special_3893|>",
+    "<|special_3894|>",
+    "<|special_3895|>",
+    "<|special_3896|>",
+    "<|special_3897|>",
+    "<|special_3898|>",
+    "<|special_3899|>",
+    "<|special_3900|>",
+    "<|special_3901|>",
+    "<|special_3902|>",
+    "<|special_3903|>",
+    "<|special_3904|>",
+    "<|special_3905|>",
+    "<|special_3906|>",
+    "<|special_3907|>",
+    "<|special_3908|>",
+    "<|special_3909|>",
+    "<|special_3910|>",
+    "<|special_3911|>",
+    "<|special_3912|>",
+    "<|special_3913|>",
+    "<|special_3914|>",
+    "<|special_3915|>",
+    "<|special_3916|>",
+    "<|special_3917|>",
+    "<|special_3918|>",
+    "<|special_3919|>",
+    "<|special_3920|>",
+    "<|special_3921|>",
+    "<|special_3922|>",
+    "<|special_3923|>",
+    "<|special_3924|>",
+    "<|special_3925|>",
+    "<|special_3926|>",
+    "<|special_3927|>",
+    "<|special_3928|>",
+    "<|special_3929|>",
+    "<|special_3930|>",
+    "<|special_3931|>",
+    "<|special_3932|>",
+    "<|special_3933|>",
+    "<|special_3934|>",
+    "<|special_3935|>",
+    "<|special_3936|>",
+    "<|special_3937|>",
+    "<|special_3938|>",
+    "<|special_3939|>",
+    "<|special_3940|>",
+    "<|special_3941|>",
+    "<|special_3942|>",
+    "<|special_3943|>",
+    "<|special_3944|>",
+    "<|special_3945|>",
+    "<|special_3946|>",
+    "<|special_3947|>",
+    "<|special_3948|>",
+    "<|special_3949|>",
+    "<|special_3950|>",
+    "<|special_3951|>",
+    "<|special_3952|>",
+    "<|special_3953|>",
+    "<|special_3954|>",
+    "<|special_3955|>",
+    "<|special_3956|>",
+    "<|special_3957|>",
+    "<|special_3958|>",
+    "<|special_3959|>",
+    "<|special_3960|>",
+    "<|special_3961|>",
+    "<|special_3962|>",
+    "<|special_3963|>",
+    "<|special_3964|>",
+    "<|special_3965|>",
+    "<|special_3966|>",
+    "<|special_3967|>",
+    "<|special_3968|>",
+    "<|special_3969|>",
+    "<|special_3970|>",
+    "<|special_3971|>",
+    "<|special_3972|>",
+    "<|special_3973|>",
+    "<|special_3974|>",
+    "<|special_3975|>",
+    "<|special_3976|>",
+    "<|special_3977|>",
+    "<|special_3978|>",
+    "<|special_3979|>",
+    "<|special_3980|>",
+    "<|special_3981|>",
+    "<|special_3982|>",
+    "<|special_3983|>",
+    "<|special_3984|>",
+    "<|special_3985|>",
+    "<|special_3986|>",
+    "<|special_3987|>",
+    "<|special_3988|>",
+    "<|special_3989|>",
+    "<|special_3990|>",
+    "<|special_3991|>",
+    "<|special_3992|>",
+    "<|special_3993|>",
+    "<|special_3994|>",
+    "<|special_3995|>",
+    "<|special_3996|>",
+    "<|special_3997|>",
+    "<|special_3998|>",
+    "<|special_3999|>",
+    "<|special_4000|>",
+    "<|special_4001|>",
+    "<|special_4002|>",
+    "<|special_4003|>",
+    "<|special_4004|>",
+    "<|special_4005|>",
+    "<|special_4006|>",
+    "<|special_4007|>",
+    "<|special_4008|>",
+    "<|special_4009|>",
+    "<|special_4010|>",
+    "<|special_4011|>",
+    "<|special_4012|>",
+    "<|special_4013|>",
+    "<|special_4014|>",
+    "<|special_4015|>",
+    "<|special_4016|>",
+    "<|special_4017|>",
+    "<|special_4018|>",
+    "<|special_4019|>",
+    "<|special_4020|>",
+    "<|special_4021|>",
+    "<|special_4022|>",
+    "<|special_4023|>",
+    "<|special_4024|>",
+    "<|special_4025|>",
+    "<|special_4026|>",
+    "<|special_4027|>",
+    "<|special_4028|>",
+    "<|special_4029|>",
+    "<|special_4030|>",
+    "<|special_4031|>",
+    "<|special_4032|>",
+    "<|special_4033|>",
+    "<|special_4034|>",
+    "<|special_4035|>",
+    "<|special_4036|>",
+    "<|special_4037|>",
+    "<|special_4038|>",
+    "<|special_4039|>",
+    "<|special_4040|>",
+    "<|special_4041|>",
+    "<|special_4042|>",
+    "<|special_4043|>",
+    "<|special_4044|>",
+    "<|special_4045|>",
+    "<|special_4046|>",
+    "<|special_4047|>",
+    "<|special_4048|>",
+    "<|special_4049|>",
+    "<|special_4050|>",
+    "<|special_4051|>",
+    "<|special_4052|>",
+    "<|special_4053|>",
+    "<|special_4054|>",
+    "<|special_4055|>",
+    "<|special_4056|>",
+    "<|special_4057|>",
+    "<|special_4058|>",
+    "<|special_4059|>",
+    "<|special_4060|>",
+    "<|special_4061|>",
+    "<|special_4062|>",
+    "<|special_4063|>",
+    "<|special_4064|>",
+    "<|special_4065|>",
+    "<|special_4066|>",
+    "<|special_4067|>",
+    "<|special_4068|>",
+    "<|special_4069|>",
+    "<|special_4070|>",
+    "<|special_4071|>",
+    "<|special_4072|>",
+    "<|special_4073|>",
+    "<|special_4074|>",
+    "<|special_4075|>",
+    "<|special_4076|>",
+    "<|special_4077|>",
+    "<|special_4078|>",
+    "<|special_4079|>",
+    "<|special_4080|>",
+    "<|special_4081|>",
+    "<|special_4082|>",
+    "<|special_4083|>",
+    "<|special_4084|>",
+    "<|special_4085|>",
+    "<|special_4086|>",
+    "<|special_4087|>",
+    "<|special_4088|>",
+    "<|special_4089|>",
+    "<|special_4090|>",
+    "<|special_4091|>",
+    "<|special_4092|>",
+    "<|special_4093|>",
+    "<|special_4094|>",
+    "<|special_4095|>"
+  ],
+  "bos_token": {
+    "content": "<|startoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<|endoftext|>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b7a8c81670e4651cc290fbe0ab397e0357abfa933964b9b6f56f508557c04c19
+size 16473485