---
license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: tencent/Hy3
base_model_relation: quantized
tags:
- hunyuan
- hy3
- moe
- text-generation
- nvfp4
- compressed-tensors
- quantized
- vllm
---
# Hy3 — NVFP4 (routed experts, MSE scales)
> **A 4-bit NVFP4 quantization of [`tencent/Hy3`](https://huggingface.co/tencent/Hy3).**
> The original model card is preserved in full [below](#model-introduction).
Weight-only NVFP4 quant produced with [**qstream**](https://github.com/olka/qstream)
using MSE-optimal group-scale selection: the routed MoE experts (≈95% of the weights)
are quantized to 4-bit; everything quality-sensitive stays BF16.
> ## ⚠️ MARLIN-only (this is a **W4A16** build)
>
> Because the activations stay BF16 (**weight-only, W4A16**), vLLM serves this build
> **exclusively on the MARLIN NvFp4 kernel**. The faster FlashInfer fp4 tensor-core
> backends (`flashinfer_trtllm` / `cutlass` / `cutedsl`) are **W4A4** and *reject a
> weight-only scheme* — `--moe-backend flashinfer_trtllm` fails with
> `kernel does not support QuantKey(u8, scale(f8e4m3))`, so vLLM auto-selects MARLIN.
>
> **Consequence:** measured **~26% slower** than the MXFP4 sibling with MTP
> (**674 vs 849 tok/s**, aggregate at concurrency 16 on one B300). If throughput
> matters, prefer [**`olka-fi/Hy3-MXFP4`**](https://huggingface.co/olka-fi/Hy3-MXFP4),
> which runs on the standard MXFP4 path. A FlashInfer-capable W4A4 variant needs
> activation calibration (planned separately — see [below](#serving-with-vllm)).
| | |
|---|---|
| **Size** | **169 GB** (down from ~590 GB BF16 source, ~29%) |
| **Format** | compressed-tensors `nvfp4-pack-quantized`, **weight-only (W4A16)** — E2M1 4-bit weights + FP8-E4M3 group-16 scales + FP32 per-tensor global scale |
| **Base** | [tencent/Hy3](https://huggingface.co/tencent/Hy3) — 295B MoE (21B active, 3.8B MTP), 80 layers + 1 MTP layer, 192 experts top-8 + 1 shared, 256K context |
### What is quantized to what
| Component | Precision | Why |
|---|---|---|
| Routed experts (`…mlp.experts.{0..191}.{gate,up,down}_proj`), incl. the MTP layer | **NVFP4** (4-bit) | the bulk of the weights — the only place worth the size win |
| Shared expert (`…mlp.shared_mlp.*`) | **BF16** | always-on path — quantizing it amplifies error across every token |
| Attention, MoE router, dense-layer MLPs | **BF16** | sensitive / small — kept native |
| Embeddings, `lm_head`, all norms | **BF16 / F32** | unchanged |
### Quality & faithfulness
Functional evals run **end-to-end on a single NVIDIA B300 (275 GB)** with the
**MARLIN** NvFp4 MoE kernel (see [Serving](#serving-with-vllm) — this build is MARLIN-only).
| Metric | Result | What it shows |
|---|---|---|
| **WikiText-2 perplexity** (raw test, 290,674 tokens, 2048-ctx) | **5.356** | language modeling intact |
| **GSM8K** (full 1319-task, CoT, greedy) | **1263 / 1319 (95.8%)** | math/reasoning preserved |
| **MTP acceptance** (`num_speculative_tokens=1`, 104.6k draft tokens on GSM8K) | **83.4%** | the MTP draft layer speculates well — free decode speedup, lossless |
| **Routed-expert SQNR** (sampled, NVFP4 vs BF16) | **≈ 21.3 dB** (‖W−dequant‖/‖W‖ ≈ 8.65%) | ~2 dB better weight reconstruction than the MXFP4 sibling (FP8 group scale vs E8M0) |
### Fidelity, footprint & provenance
- **MTP preserved:** the MTP draft layer (layer 80) is served with
`--speculative-config '{"method":"mtp","num_speculative_tokens":1}'` and reaches **83.4%**
draft-token acceptance — lossless (GSM8K unchanged with it on).
- **Footprint:** ~169 GB — fits a single large-VRAM GPU (verified on one **B300, 275 GB**)
or multi-GPU tensor parallelism. `fastsafetensors` loads it in ~90 s.
- **Provenance:** built with [qstream](https://github.com/olka/qstream), experts-only,
NVFP4 with MSE-optimal FP8-E4M3 group-scale selection (group_size 16).
### Serving with vLLM
`HYV3ForCausalLM` is natively supported. **This is a weight-only (W4A16) build, so vLLM
serves it on the MARLIN NvFp4 kernel** — the FlashInfer fp4 tensor-core backends
(`flashinfer_trtllm`/`cutlass`/`cutedsl`) are **W4A4** and reject a weight-only scheme
(`kernel does not support QuantKey(u8, scale(f8e4m3))`), so vLLM auto-selects MARLIN.
```bash
vllm serve olka-fi/Hy3-nvfp4-mse \
--served-model-name hy3 --tensor-parallel-size 1 \
--max-model-len 4096 --gpu-memory-utilization 0.90 \
--load-format fastsafetensors \
--speculative-config '{"method":"mtp","num_speculative_tokens":1}'
```
> **`config.json` note.** The `ignore` list uses regex (`re:.*mlp\.down_proj$`), which
> already covers vLLM's shared-expert load path — no per-layer literal fixes needed.
### vs the MXFP4 sibling ([`olka-fi/Hy3-MXFP4`](https://huggingface.co/olka-fi/Hy3-MXFP4))
Despite ~2 dB **better weight reconstruction**, this NVFP4-mse build measured **slightly
worse downstream** than MXFP4 — higher weight SQNR did not translate to better quality:
| | MXFP4 | **NVFP4-mse** |
|---|---|---|
| GSM8K (1319) | 96.3% | 95.8% |
| WikiText-2 PPL | **5.12** | 5.356 |
| MTP acceptance | 82.8% | 83.4% |
| Throughput +MTP (conc. 16, 1×B300) | **849 tok/s** | 674 tok/s |
| Size | 160 GB | 169 GB |
| vLLM kernel | standard MXFP4 path | **MARLIN-only (W4A16)** |
**Recommendation: prefer the MXFP4 build** — better perplexity, smaller, broader kernel
support. Part of the NVFP4 PPL gap may be the MARLIN kernel rather than the quantization.
> **Future work — W4A4 for FlashInfer.** This build is weight-only. A W4A4 variant
> (adding FP4 activation quantization via per-expert input scales) would run on the fast
> FlashInfer fp4 tensor-core kernels; producing it needs activation calibration and is
> planned separately.
---
> ⬇️ **The original `tencent/Hy3` model card follows, unmodified.** ⬇️
中文 ｜ English
[](#license)
[](https://huggingface.co/tencent/Hy3)
[](https://modelscope.cn/models/Tencent-Hunyuan/Hy3)
[](https://cnb.cool/ai-models/tencent/Hy3)
[](https://ai.gitcode.com/tencent_hunyuan/Hy3)
🖥️ Official Website |
💬 GitHub
---
## Table of Contents
- [Model Introduction](#model-introduction)
- [Stronger Agent Performance](#stronger-agent-performance)
- [Product Experience: More Reliable, More Cost-Effective](#product-experience-more-reliable-more-cost-effective)
- [Benchmark Appendix](#benchmark-appendix)
- [News](#news)
- [Model Links](#model-links)
- [Quickstart](#quickstart)
- [Deployment](#deployment)
- [vLLM](#vllm)
- [SGLang](#sglang)
- [Finetuning](#finetuning)
- [Quantization](#quantization)
- [License](#license)
- [Contact Us](#contact-us)
---
## Model Introduction
**Hy3** is a 295B-parameter Mixture-of-Experts (MoE) model with 21B active parameters and 3.8B MTP layer parameters, developed by the Tencent Hy Team. Following the Hy3 Preview launch in late April, we gathered feedback from 50+ product teams. We fixed various issues in task execution and interaction, and improved both the quality and scale of our post-training pipeline. Today, we are launching Hy3. It significantly outperforms similar-size models and rivals flagship open-source models with 2-5x the parameters. It also shows solid gains in utility across productivity tasks and real-world applications.
| Property | Value |
|:---|:---|
| Architecture | Mixture-of-Experts (MoE) |
| Total Parameters | 295B |
| Activated Parameters | 21B |
| MTP Layer Parameters | 3.8B |
| Number of Layers (excluding MTP layer) | 80 |
| Number of MTP Layers | 1 |
| Attention Heads | 64 (GQA, 8 KV heads, head dim 128) |
| Hidden Size | 4096 |
| Intermediate Size | 13312 |
| Context Length | 256K |
| Vocabulary Size | 120832 |
| Number of Experts | 192 experts, top-8 activated |
| Supported Precisions | BF16 |
## Stronger Agent Performance
Building on Hy3 Preview, we improved post-training data quality and diversity while scaling up RL training. Hy3 shows solid gains across reasoning, agentic workflows, and long-context tasks. Its performance is close to leading flagship models, both domestic and international.
In productivity scenarios such as coding, document processing, financial analysis, game development, and frontend design, Hy3 has made solid gains, positioning it as a reliable, cost-effective option.
We don't think public benchmark scores tell the full story. So we ran a blind test with 270 experts from various disciplines, working on real-world workflows, and collected 312 valid comparisons. Hy3 scored 2.67/4, outperforming GLM-5.1 at 2.51/4. The advantage was clearest in frontend development, CI/CD, and data & storage.
## Product Experience: More Reliable, More Cost-Effective
Utility in production is not fully captured by benchmarks. Based on extensive user feedback and product telemetry, we identified real-world behavior issues that break product experience and improved the model's capabilities in those areas, earning uniformly positive feedback from product teams.
**Output Formatting and Tool Calling Stability**: We fixed multiple baseline reliability issues, bringing the model to production-grade standards across tool configurations and output constraints. Tool-call success rates and error recovery improved, and invalid calls that trigger infinite loops dropped. Hy3 also generalizes across different agent scaffoldings. On SWE-Bench Verified, accuracy variance across scaffoldings like CodeBuddy, Cline, and KiloCode remains within 4%.
**World Knowledge and Anti-Hallucination**: Internal knowledge and external hallucination are interconnected and critical to real-world product experience. Guided by the ideal behavior pattern: "answer when grounded, state when evidence is missing, do not conflate sources, do not fabricate data," we implemented fine-grained data cleaning and specific training constraints. In internal evaluations on real-world scenarios, Hy3's hallucination rate dropped from 12.5% to 5.4%, and commonsense error rates fell from 25.4% to 12.7%. These improvements materially reduce fact conflation, fabrication, and logical contradiction.
**Complex Context Retention and Multi-turn Intent Tracking**: Through joint optimization of SFT and RL, Hy3 improved on operational pain points like coreference resolution, ellipsis recovery, and multi-turn constraint inheritance. On internal comprehensive multi-turn tests, the issue rate dropped from 17.4% to 7.9%. It also posted significant gains on open-source long-dialogue benchmarks like MRCR, from 42.9% to 75.1%. Overall outputs are more concise while ensuring complex intents do not decay or drift over long-horizon interactions.
## Benchmark Appendix
## News
* 🔥 We open-source **Hy3** and **Hy3-FP8** model weights on [Hugging Face](https://huggingface.co/tencent/Hy3), [ModelScope](https://modelscope.cn/models/Tencent-Hunyuan/Hy3), [GitCode](https://ai.gitcode.com/tencent_hunyuan/Hy3), and [CNB](https://cnb.cool/ai-models/tencent/Hy3).
## Model Links
| Model Name | Description | Hugging Face | ModelScope | GitCode | CNB |
|:---|:---|:---:|:---:|:---:|:---:|
| Hy3 | Instruct model | 🤗 [Model](https://huggingface.co/tencent/Hy3) | [Model](https://modelscope.cn/models/Tencent-Hunyuan/Hy3) | [Model](https://ai.gitcode.com/tencent_hunyuan/Hy3) | [Model](https://cnb.cool/ai-models/tencent/Hy3) |
| Hy3-FP8 | FP8 quantized instruct model | 🤗 [Model](https://huggingface.co/tencent/Hy3-FP8) | [Model](https://modelscope.cn/models/Tencent-Hunyuan/Hy3-FP8) | [Model](https://ai.gitcode.com/tencent_hunyuan/Hy3-FP8) | [Model](https://cnb.cool/ai-models/tencent/Hy3-FP8) |
## Quickstart
Deploy Hy3 with [vLLM](#vllm) or [SGLang](#sglang) first, then call the OpenAI-compatible API:
```python
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="hy3",
messages=[
{"role": "user", "content": "Hello! Can you briefly introduce yourself?"},
],
temperature=0.9,
top_p=1.0,
# reasoning_effort: "no_think" (default, direct response), "low", "high" (deep chain-of-thought)
extra_body={"chat_template_kwargs": {"reasoning_effort": "no_think"}},
)
print(response.choices[0].message.content)
```
> **Recommended parameters**: `temperature=0.9`, `top_p=1.0`.
>
> **Reasoning mode**: Set `reasoning_effort` to `"high"` for complex tasks (math, coding, reasoning) or `"no_think"` for direct responses.
See the [Deployment](#deployment) section below for how to start the API server.
## Deployment
Hy3 has 295B parameters in total. To serve it on 8 GPUs, we recommend using H20-3e or other GPUs with larger memory capacity.
### vLLM
Build vLLM from source:
```bash
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
git clone https://github.com/vllm-project/vllm.git
cd vllm
uv pip install --editable . --torch-backend=auto
```
Start the vLLM server with MTP enabled:
```bash
# Switch to trtllm backend to work-around mnnvl workspace size issue.
export VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm
vllm serve tencent/Hy3 \
--tensor-parallel-size 8 \
--speculative-config.method mtp \
--speculative-config.num_speculative_tokens 2 \
--tool-call-parser hy_v3 \
--reasoning-parser hy_v3 \
--enable-auto-tool-choice \
--port 8000 \
--served-model-name hy3
```
### SGLang
Build SGLang from source:
```bash
git clone https://github.com/sgl-project/sglang
cd sglang
pip3 install pip --upgrade
pip3 install "transformers>=5.6.0"
pip3 install -e "python"
```
Launch SGLang server with MTP enabled:
```bash
python3 -m sglang.launch_server \
--model tencent/Hy3 \
--tp-size 8 \
--tool-call-parser hunyuan \
--reasoning-parser hunyuan \
--speculative-num-steps 2 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 3 \
--speculative-algorithm EAGLE \
--port 8000 \
--served-model-name hy3
```
## Finetuning
Hy3 provides a complete model finetuning pipeline. For detailed documentation, please refer to: [Finetuning Guide](https://huggingface.co/tencent/Hy3/blob/main/finetune/README.md)
## Quantization
We provide [AngelSlim](https://github.com/tencent/AngelSlim), a more accessible, comprehensive, and efficient toolkit for large model compression. AngelSlim supports a comprehensive suite of compression tools for large-scale multimodal models, including common quantization algorithms, low-bit quantization, and speculative sampling.
## License
Hy3 is released under the **Apache License 2.0**. See [LICENSE](https://huggingface.co/tencent/Hy3/blob/main/LICENSE) for details.
## Contact Us
If you would like to leave a message for our R&D and product teams, welcome to contact us. You can also reach us via email:
📧 **hunyuan_opensource@tencent.com**
---
Hy3 is developed by the Tencent Hy Team.