---
base_model: meta-llama/Meta-Llama-3.1-8B-Instruct
library_name: transformers
pipeline_tag: text-generation
tags:
- omniquant
- quantization
- weight-only-quantization
- fake-quantized
datasets:
- Salesforce/wikitext
---

# Meta-Llama-3.1-8B-Instruct-OmniQuant-4bit

This repository contains a **W4A16 OmniQuant** checkpoint derived from
[`meta-llama/Meta-Llama-3.1-8B-Instruct`](https://huggingface.co/meta-llama/Meta-Llama-3.1-8B-Instruct).

## Quantization configuration

- Method: OmniQuant
- Weight precision: 4-bit
- Activation precision: 16-bit
- Group size: 128
- Optimization epochs: 20
- Learnable weight clipping (LWC): enabled
- Learnable equivalent transformation (LET): disabled
- Calibration dataset: C4 English validation split
- Calibration samples: 128
- Calibration sequence length: 512
- Calibration seed: 42

## Evaluation

| Dataset | Split | Sequence length | Perplexity |
|---|---|---:|---:|
| WikiText2 | test | 2048 | 7.5728 |

The evaluation used the full tokenized WikiText2 test corpus with non-overlapping
2048-token windows.

## Checkpoint format

This follows OmniQuant's fake-quantized Hugging Face save path. It is not a
packed low-bit runtime checkpoint, so its storage and loading memory can remain
close to FP16 despite representing W4A16 quantized weights.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "yw223/Meta-Llama-3.1-8B-Instruct-OmniQuant-4bit"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)
```

Transformers may report unused `weight_quantizer.scales` and
`weight_quantizer.zeros` entries when loading. The fake-quantized model weights
still load through the standard Transformers path used for the reported PPL.

## License

Use of this checkpoint is subject to the license and terms of the base model.