Commit ·
0eeffab
1
Parent(s): 76716fa
Add pipeline tag, arxiv metadata and fix usage snippets (#1)
Browse files- Add pipeline tag, arxiv metadata and fix usage snippets (edc44d3c98e87f7a1b7bba86acab72f8656f409a)
Co-authored-by: Niels Rogge <nielsr@users.noreply.huggingface.co>
README.md
CHANGED
|
@@ -1,4 +1,5 @@
|
|
| 1 |
---
|
|
|
|
| 2 |
language:
|
| 3 |
- en
|
| 4 |
license: apache-2.0
|
|
@@ -11,8 +12,9 @@ tags:
|
|
| 11 |
- qwen
|
| 12 |
- llm
|
| 13 |
- compression
|
| 14 |
-
base_model: Qwen/Qwen3-32B
|
| 15 |
base_model_relation: quantized
|
|
|
|
|
|
|
| 16 |
---
|
| 17 |
|
| 18 |
<p align="center">
|
|
@@ -25,6 +27,9 @@ base_model_relation: quantized
|
|
| 25 |
# A-SINQ 4-bit Quantized Qwen3-32B model
|
| 26 |
|
| 27 |
This repository contains the official **4-bit quantized** version of the [`Qwen3-32B`](https://huggingface.co/Qwen/Qwen3-32B) model using the *calibrated* version of **SINQ (Sinkhorn-Normalized Quantization)** method.
|
|
|
|
|
|
|
|
|
|
| 28 |
SINQ is a novel, fast and high-quality quantization method designed to make any Large Language Models smaller while keeping their accuracy almost intact.
|
| 29 |
|
| 30 |
To support the project please put a star ⭐ in the official [SINQ](https://github.com/huawei-csl/SINQ) github repository.
|
|
@@ -48,7 +53,7 @@ To support the project please put a star ⭐ in the official [SINQ](https://gith
|
|
| 48 |
|
| 49 |
---
|
| 50 |
|
| 51 |
-
# 🚀 Usage
|
| 52 |
|
| 53 |
## Prerequisite
|
| 54 |
Before running the quantization script, make sure the **SINQ** library is installed.
|
|
@@ -58,6 +63,7 @@ Installation instructions and setup details are available in the [SINQ official
|
|
| 58 |
You can load and use the model with our wrapper based on the 🤗 Transformers library:
|
| 59 |
|
| 60 |
```python
|
|
|
|
| 61 |
from transformers import AutoTokenizer
|
| 62 |
from sinq.patch_model import AutoSINQHFModel
|
| 63 |
|
|
@@ -83,6 +89,7 @@ print(tokenizer.decode(out_ids[0], skip_special_tokens=True))
|
|
| 83 |
The quantized model was obtained using the **SINQ** quantization library, following the steps below:
|
| 84 |
|
| 85 |
```python
|
|
|
|
| 86 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 87 |
from sinq.patch_model import AutoSINQHFModel
|
| 88 |
from sinq.sinqlinear import BaseQuantizeConfig
|
|
|
|
| 1 |
---
|
| 2 |
+
base_model: Qwen/Qwen3-32B
|
| 3 |
language:
|
| 4 |
- en
|
| 5 |
license: apache-2.0
|
|
|
|
| 12 |
- qwen
|
| 13 |
- llm
|
| 14 |
- compression
|
|
|
|
| 15 |
base_model_relation: quantized
|
| 16 |
+
pipeline_tag: text-generation
|
| 17 |
+
arxiv: 2509.22944
|
| 18 |
---
|
| 19 |
|
| 20 |
<p align="center">
|
|
|
|
| 27 |
# A-SINQ 4-bit Quantized Qwen3-32B model
|
| 28 |
|
| 29 |
This repository contains the official **4-bit quantized** version of the [`Qwen3-32B`](https://huggingface.co/Qwen/Qwen3-32B) model using the *calibrated* version of **SINQ (Sinkhorn-Normalized Quantization)** method.
|
| 30 |
+
|
| 31 |
+
The method was introduced in the paper [SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights](https://huggingface.co/papers/2509.22944) by Lorenz K. Müller, Philippe Bich, Jiawei Zhuang, Ahmet Çelik, Luca Benfenati, and Lukas Cavigelli.
|
| 32 |
+
|
| 33 |
SINQ is a novel, fast and high-quality quantization method designed to make any Large Language Models smaller while keeping their accuracy almost intact.
|
| 34 |
|
| 35 |
To support the project please put a star ⭐ in the official [SINQ](https://github.com/huawei-csl/SINQ) github repository.
|
|
|
|
| 53 |
|
| 54 |
---
|
| 55 |
|
| 56 |
+
# 🚀 Usage
|
| 57 |
|
| 58 |
## Prerequisite
|
| 59 |
Before running the quantization script, make sure the **SINQ** library is installed.
|
|
|
|
| 63 |
You can load and use the model with our wrapper based on the 🤗 Transformers library:
|
| 64 |
|
| 65 |
```python
|
| 66 |
+
import torch
|
| 67 |
from transformers import AutoTokenizer
|
| 68 |
from sinq.patch_model import AutoSINQHFModel
|
| 69 |
|
|
|
|
| 89 |
The quantized model was obtained using the **SINQ** quantization library, following the steps below:
|
| 90 |
|
| 91 |
```python
|
| 92 |
+
import torch
|
| 93 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
| 94 |
from sinq.patch_model import AutoSINQHFModel
|
| 95 |
from sinq.sinqlinear import BaseQuantizeConfig
|