Hugging Face's logo

jkim96
/
EXAONE-4.5-33B-DASHQ-INT2-g32

Image-Text-to-Text
Transformers
Safetensors
exaone4_5
text-generation
dashq
quantized
post-training-quantization
int2
conversational
custom_code
Model card Files
xet
 Community
jkim96 commited on
Commit
d9ff734
·
verified ·
1 Parent(s): 72c406f

Upload dashq quantized checkpoint (INT2, g32, scale_zero_dtype=float16, zero-shot avg=67.7)

Browse files
Files changed (13) hide show
  1. .gitattributes +1 -0
  2. README.md +63 -0
  3. chat_template.jinja +161 -0
  4. config.json +143 -0
  5. dashq_config.json +0 -0
  6. generation_config.json +13 -0
  7. model-00001-of-00004.safetensors +3 -0
  8. model-00002-of-00004.safetensors +3 -0
  9. model-00003-of-00004.safetensors +3 -0
  10. model-00004-of-00004.safetensors +3 -0
  11. model.safetensors.index.json +0 -0
  12. tokenizer.json +3 -0
  13. tokenizer_config.json +21 -0
.gitattributes CHANGED
@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
 
 
33
  *.zip filter=lfs diff=lfs merge=lfs -text
34
  *.zst filter=lfs diff=lfs merge=lfs -text
35
  *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ tokenizer.json filter=lfs diff=lfs merge=lfs -text
README.md ADDED
@@ -0,0 +1,63 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ license: apache-2.0
3
+ base_model: LGAI-EXAONE/EXAONE-4.5-33B
4
+ library_name: transformers
5
+ tags:
6
+ - dashq
7
+ - quantized
8
+ - post-training-quantization
9
+ ---
10
+
11
+ # EXAONE-4.5-33B-DASHQ-INT2-g32
12
+
13
+ This repository contains a DASH-Q packed quantized checkpoint for `LGAI-EXAONE/EXAONE-4.5-33B`.
14
+
15
+ DASH-Q checkpoints require the lightweight DASH-Q runtime package for loading. They are not plain Transformers checkpoints because linear layers are stored as `PackedQuantizedLinear` modules.
16
+
17
+ ## Install
18
+
19
+ ```bash
20
+ pip install git+https://github.com/JaeminK/dashq.git
21
+ ```
22
+
23
+ ## Load
24
+
25
+ ```python
26
+ from dashq import load_quantized
27
+
28
+ model, tokenizer = load_quantized(
29
+ "jkim96/EXAONE-4.5-33B-DASHQ-INT2-g32",
30
+ device_map="auto",
31
+ )
32
+ ```
33
+
34
+ ## Quantization
35
+
36
+ | Field | Value |
37
+ | --- | --- |
38
+ | Base model | `LGAI-EXAONE/EXAONE-4.5-33B` |
39
+ | Bits | `2` |
40
+ | Group size | `32` |
41
+ | Scale/zero dtype | `float16` |
42
+ | Calibration dataset | `wikitext2` |
43
+ | Calibration samples | `128` |
44
+ | Sequence length | `2048` |
45
+ | Original size | `68.7003 GB` |
46
+ | Quantized size | `17.3280 GB` |
47
+
48
+ ## Evaluation
49
+
50
+ | Metric | Value |
51
+ | --- | ---: |
52
+ | `wikitext2_ppl` | 10.4470 |
53
+ | `zero-shot accuracy avg` | 67.7376 |
54
+ | `arc_challenge` | 54.4369 |
55
+ | `arc_easy` | 83.0808 |
56
+ | `commonsense_qa` | 70.9255 |
57
+ | `hellaswag` | 71.0118 |
58
+ | `lambada_openai` | not run |
59
+ | `openbookqa` | not run |
60
+ | `piqa` | 78.1828 |
61
+ | `truthfulqa_mc2` | 45.0182 |
62
+ | `winogrande` | 71.5075 |
63
+
chat_template.jinja ADDED
@@ -0,0 +1,161 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {% set image_count = namespace(value=0) %}
2
+ {% set video_count = namespace(value=0) %}
3
+
4
+ {%- set role_indicators = {
5
+ 'user': '<|user|>\n',
6
+ 'assistant': '<|assistant|>\n',
7
+ 'system': '<|system|>\n',
8
+ 'tool': '<|tool|>\n',
9
+ 'tool_declare': '<|tool_declare|>\n'
10
+ } %}
11
+ {%- set end_of_turn = '<|endofturn|>\n' %}
12
+
13
+
14
+ {%- macro declare_available_tools(tools) %}
15
+ {{- "# Tools" }}
16
+ {{- "\n" }}
17
+ {%- for tool in tools %}
18
+ {{- "<tool>" }}
19
+ {{- tool | tojson(ensure_ascii=False) | safe }}
20
+ {{- "</tool>\n" }}
21
+ {%- endfor %}
22
+ {%- endmacro %}
23
+
24
+
25
+ {%- set ns = namespace(last_query_index = messages|length - 1, last_query_index_not_yet_determined = true) %}
26
+ {%- for message in messages[::-1] %}
27
+ {%- set index = (messages|length - 1) - loop.index0 %}
28
+ {%- if ns.last_query_index_not_yet_determined and message.role == "user" and message.content is string %}
29
+ {%- set ns.last_query_index = index -%}
30
+ {%- set ns.last_query_index_not_yet_determined = false -%}
31
+ {%- endif %}
32
+ {%- endfor %}
33
+
34
+ {%- if tools is defined and tools %}
35
+ {{- role_indicators['tool_declare'] }}
36
+ {{- declare_available_tools(tools) }}
37
+ {{- end_of_turn -}}
38
+ {%- endif %}
39
+
40
+ {%- for i in range(messages | length) %}
41
+ {%- set msg = messages[i] %}
42
+ {%- set role = msg.role %}
43
+ {%- if role not in role_indicators %}
44
+ {{- raise_exception('Unknown role: ' ~ role) }}
45
+ {%- endif %}
46
+
47
+ {%- if i == 0 %}
48
+ {%- if role == 'system' %}
49
+ {{- role_indicators['system'] }}
50
+ {{- msg.content }}
51
+ {{- end_of_turn -}}
52
+ {%- continue %}
53
+ {%- endif %}
54
+ {%- endif %}
55
+
56
+ {%- if role == 'assistant' %}
57
+ {{- role_indicators['assistant'] }}
58
+
59
+ {%- set content = (msg.content if (msg.content is defined and msg.content) else "") -%}
60
+ {%- set reasoning = none -%}
61
+
62
+ {%- if msg.reasoning_content is defined and msg.reasoning_content%}
63
+ {%- set reasoning = msg.reasoning_content.strip() -%}
64
+ {%- elif content and "</think>" in content %}
65
+ {%- set _parts = content.split('</think>') -%}
66
+ {%- set reasoning = _parts[0].lstrip('<think>').strip() -%}
67
+ {%- set content = _parts[-1].strip() -%}
68
+ {%- endif %}
69
+
70
+ {%- if not (reasoning and i > ns.last_query_index) or (skip_think is defined and skip_think) %}
71
+ {%- set reasoning = none %}
72
+ {%- endif %}
73
+
74
+ {%- set content = content.strip() -%}
75
+
76
+ {{- "<think>\n" }}
77
+ {{- (reasoning if reasoning is not none else "") }}
78
+ {{- "\n</think>\n\n" }}
79
+
80
+ {{- content }}
81
+
82
+ {%- if msg.tool_calls %}
83
+ {%- if content is defined and content %}
84
+ {{- "\n" }}
85
+ {%- endif %}
86
+ {%- for tool_call in msg.tool_calls %}
87
+ {%- if tool_call.function is defined %}
88
+ {%- set tool_call = tool_call.function %}
89
+ {%- endif %}
90
+
91
+ {%- if tool_call.arguments is defined %}
92
+ {%- set arguments = tool_call.arguments %}
93
+ {%- elif tool_call.parameters is defined %}
94
+ {%- set arguments = tool_call.parameters %}
95
+ {%- else %}
96
+ {{- raise_exception('arguments or parameters are mandatory: ' ~ tool_call) }}
97
+ {%- endif %}
98
+ {%- if arguments is string %}
99
+ {{- "<tool_call>" }}{"name": "{{- tool_call.name }}", "arguments": {{ arguments }}}{{- "</tool_call>" }}
100
+ {%- else %}
101
+ {{- "<tool_call>" }}{"name": "{{- tool_call.name }}", "arguments": {{ arguments | tojson(ensure_ascii=False) | safe }}}{{- "</tool_call>" }}
102
+ {%- endif %}
103
+ {%- if not loop.last %}
104
+ {{- "\n" }}
105
+ {%- endif %}
106
+
107
+ {%- endfor %}
108
+ {%- endif %}
109
+ {{- end_of_turn -}}
110
+
111
+ {%- elif role == "tool" %}
112
+ {%- if i == 0 or messages[i - 1].role != "tool" %}
113
+ {{- role_indicators['tool'] }}
114
+ {%- endif %}
115
+ {%- if msg.content is defined %}
116
+ {%- if msg.content is string %}
117
+ {{- "<tool_result>" }}{{ msg.content }}{{- "</tool_result>" }}
118
+ {%- else %}
119
+ {{- "<tool_result>" }}{{ msg.content | tojson(ensure_ascii=False) | safe }}{{- "</tool_result>" }}
120
+ {%- endif %}
121
+ {%- endif %}
122
+ {%- if loop.last or messages[i + 1].role != "tool" %}
123
+ {{- end_of_turn -}}
124
+ {%- else %}
125
+ {{- "\n" }}
126
+ {%- endif %}
127
+
128
+ {%- else %}
129
+ {{- role_indicators[role] }}
130
+ {%- if msg.content is string %}
131
+ {{- msg.content }}
132
+ {%- else %}
133
+ {%- for content in msg.content %}
134
+ {%- if content.type == 'image' %}
135
+ {%- set image_count.value = image_count.value + 1 %}
136
+ {%- if add_vision_id %}Picture {{ image_count.value }}: {% endif %}
137
+ {{- "<vision><|image_pad|></vision>\n" }}
138
+ {%- elif content.type == 'video' %}
139
+ {%- set video_count.value = video_count.value + 1 %}
140
+ {%- if add_vision_id %}Video {{ video_count.value }}: {% endif %}
141
+ {{- "<vision><|video_pad|></vision>\n" }}
142
+ {%- elif content.type == 'text' %}
143
+ {{- content.text }}
144
+ {%- else %}
145
+ {{- content.text }}
146
+ {%- endif %}
147
+ {%- endfor %}
148
+ {%- endif %}
149
+ {{- end_of_turn -}}
150
+ {%- endif %}
151
+ {% endfor %}
152
+
153
+
154
+ {%- if add_generation_prompt %}
155
+ {{- role_indicators['assistant'] }}
156
+ {%- if enable_thinking is not defined or enable_thinking is true %}
157
+ {{- "<think>\n" }}
158
+ {%- else %}
159
+ {{- "<think>\n\n</think>\n\n" }}
160
+ {%- endif %}
161
+ {%- endif %}
config.json ADDED
@@ -0,0 +1,143 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "Exaone4_5_ForConditionalGeneration"
4
+ ],
5
+ "dtype": "bfloat16",
6
+ "image_token_id": 67,
7
+ "model_type": "exaone4_5",
8
+ "text_config": {
9
+ "_num_mtp_layers": 1,
10
+ "architectures": [
11
+ "Exaone4ForCausalLM"
12
+ ],
13
+ "attention_dropout": 0.0,
14
+ "bos_token_id": 1,
15
+ "dtype": "bfloat16",
16
+ "eos_token_id": 53,
17
+ "hidden_act": "silu",
18
+ "hidden_size": 5120,
19
+ "initializer_range": 0.02,
20
+ "intermediate_size": 27392,
21
+ "layer_types": [
22
+ "sliding_attention",
23
+ "sliding_attention",
24
+ "sliding_attention",
25
+ "full_attention",
26
+ "sliding_attention",
27
+ "sliding_attention",
28
+ "sliding_attention",
29
+ "full_attention",
30
+ "sliding_attention",
31
+ "sliding_attention",
32
+ "sliding_attention",
33
+ "full_attention",
34
+ "sliding_attention",
35
+ "sliding_attention",
36
+ "sliding_attention",
37
+ "full_attention",
38
+ "sliding_attention",
39
+ "sliding_attention",
40
+ "sliding_attention",
41
+ "full_attention",
42
+ "sliding_attention",
43
+ "sliding_attention",
44
+ "sliding_attention",
45
+ "full_attention",
46
+ "sliding_attention",
47
+ "sliding_attention",
48
+ "sliding_attention",
49
+ "full_attention",
50
+ "sliding_attention",
51
+ "sliding_attention",
52
+ "sliding_attention",
53
+ "full_attention",
54
+ "sliding_attention",
55
+ "sliding_attention",
56
+ "sliding_attention",
57
+ "full_attention",
58
+ "sliding_attention",
59
+ "sliding_attention",
60
+ "sliding_attention",
61
+ "full_attention",
62
+ "sliding_attention",
63
+ "sliding_attention",
64
+ "sliding_attention",
65
+ "full_attention",
66
+ "sliding_attention",
67
+ "sliding_attention",
68
+ "sliding_attention",
69
+ "full_attention",
70
+ "sliding_attention",
71
+ "sliding_attention",
72
+ "sliding_attention",
73
+ "full_attention",
74
+ "sliding_attention",
75
+ "sliding_attention",
76
+ "sliding_attention",
77
+ "full_attention",
78
+ "sliding_attention",
79
+ "sliding_attention",
80
+ "sliding_attention",
81
+ "full_attention",
82
+ "sliding_attention",
83
+ "sliding_attention",
84
+ "sliding_attention",
85
+ "full_attention"
86
+ ],
87
+ "max_position_embeddings": 262144,
88
+ "model_type": "exaone4",
89
+ "mtp_loss_scaling_factor": 0.05,
90
+ "mtp_share_layers": true,
91
+ "num_attention_heads": 40,
92
+ "num_hidden_layers": 64,
93
+ "num_key_value_heads": 8,
94
+ "num_nextn_predict_layers": 1,
95
+ "pad_token_id": 0,
96
+ "rms_norm_eps": 1e-05,
97
+ "rope_parameters": {
98
+ "factor": 16.0,
99
+ "high_freq_factor": 4.0,
100
+ "low_freq_factor": 1.0,
101
+ "original_max_position_embeddings": 8192,
102
+ "rope_theta": 1000000.0,
103
+ "rope_type": "llama3"
104
+ },
105
+ "sliding_window": 4096,
106
+ "sliding_window_pattern": "LLLG",
107
+ "tie_word_embeddings": false,
108
+ "use_cache": true,
109
+ "vocab_size": 153600
110
+ },
111
+ "tie_word_embeddings": false,
112
+ "transformers_version": "5.8.0",
113
+ "use_cache": false,
114
+ "video_token_id": 68,
115
+ "vision_config": {
116
+ "depth": 28,
117
+ "dtype": "bfloat16",
118
+ "fullatt_block_indexes": [
119
+ 6,
120
+ 13,
121
+ 20,
122
+ 27
123
+ ],
124
+ "hidden_act": "silu",
125
+ "hidden_size": 2048,
126
+ "in_channels": 3,
127
+ "initializer_range": 0.02,
128
+ "intermediate_size": 5120,
129
+ "model_type": "exaone4_5_vision",
130
+ "num_heads": 32,
131
+ "num_key_value_heads": 8,
132
+ "out_hidden_size": 5120,
133
+ "patch_size": 14,
134
+ "spatial_merge_size": 2,
135
+ "temporal_patch_size": 2,
136
+ "tokens_per_second": 2,
137
+ "window_size": 112
138
+ },
139
+ "vision_end_token_id": 74,
140
+ "vision_start_token_id": 73,
141
+ "vision_token_id": 67,
142
+ "vocab_size": 153600
143
+ }
dashq_config.json ADDED
The diff for this file is too large to render. See raw diff
 
generation_config.json ADDED
@@ -0,0 +1,13 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "_from_model_config": true,
3
+ "bos_token_id": 1,
4
+ "do_sample": true,
5
+ "eos_token_id": 53,
6
+ "output_attentions": false,
7
+ "output_hidden_states": false,
8
+ "pad_token_id": 0,
9
+ "temperature": 1.0,
10
+ "top_p": 0.95,
11
+ "transformers_version": "5.8.0",
12
+ "use_cache": false
13
+ }
model-00001-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:1901416fa057758d2076eadfdff2f742c77251d1b5131ea2c58762682924dae2
3
+ size 4991465320
model-00002-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:80ca5829378450a3f1826fc2d8841b90094a6785670edb5e750d0f522bf61325
3
+ size 4982642720
model-00003-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:cf68b7a7e007ffd75efc31ef576ca9abf10e4e96c49c59afde2481c4362a6320
3
+ size 4994088696
model-00004-of-00004.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:5b3590e6e79ea1546931f989f72bc722202fd3c2740df1e6bb3ab8cd9cba3458
3
+ size 2359834248
model.safetensors.index.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer.json ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:0bd798efa30739e209d51f36cfc2f0a636711e37ad9d69b77e4e5c8ca5f09fab
3
+ size 12160205
tokenizer_config.json ADDED
@@ -0,0 +1,21 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "backend": "tokenizers",
3
+ "bos_token": "[BOS]",
4
+ "clean_up_tokenization_spaces": false,
5
+ "eos_token": "<|endofturn|>",
6
+ "errors": "replace",
7
+ "is_local": false,
8
+ "local_files_only": false,
9
+ "model_input_names": [
10
+ "input_ids",
11
+ "attention_mask"
12
+ ],
13
+ "model_max_length": 1000000000000000019884624838656,
14
+ "model_specific_special_tokens": {},
15
+ "pad_token": "[PAD]",
16
+ "padding_side": "right",
17
+ "processor_class": "Exaone4_5_Processor",
18
+ "split_special_tokens": false,
19
+ "tokenizer_class": "TokenizersBackend",
20
+ "unk_token": "[UNK]"
21
+ }