kernelpool/Kimi-K3-2bit-UVMAX
Mixed-precision (UVMAX) quantization of moonshotai/Kimi-K3.
What is UVMAX?
UVMAX is a mixed-precision scheme: bit widths are assigned per tensor class from measured round-trip quantization error, rather than uniformly.
| Tensor class | Bits | Parameters | Size | Share |
|---|---|---|---|---|
| Expert FFNs (routed, latent space) | 2 (gs 128) | 2.72 T | 713.2 GiB | 93.8% |
| Shared experts, MoE latent projections, dense MLP | 8 (gs 64) | 17.5 B | 17.4 GiB | 2.3% |
| Attention (KDA + MLA, all projections) | 6 (gs 64) | 36 B | 27.4 GiB | 3.6% |
Embeddings, lm_head |
4 (gs 64) | 2.4 B | 1.2 GiB | 0.2% |
| MoE routers | 8 (gs 64) | 0.6 B | 0.6 GiB | 0.1% |
| Vision tower + projector (unquantized bf16) | — | 0.4 B | 0.8 GiB | 0.1% |
| Norms, AttnRes projections, gate params (unquantized) | — | — | 0.1 GiB | <0.1% |
Use with mlx
This model requires Kimi K3 support from mlx-lm PR #1626, which has not yet been merged. Until it is included in an mlx-lm release, install mlx-lm from the PR branch:
pip install git+https://github.com/ml-explore/mlx-lm.git@refs/pull/1626/head
pip install tiktoken
from mlx_lm import load, generate
model, tokenizer = load(
"kernelpool/Kimi-K3-2bit-UVMAX",
tokenizer_config={"trust_remote_code": True},
trust_remote_code=True,
)
prompt = "hello"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
response = generate(model, tokenizer, prompt=prompt, verbose=True)
- Downloads last month
- 4,793
Model size
2.8T params
Tensor type
BF16
·
U32 ·
F32 ·
Hardware compatibility
Log In to add your hardware
4-bit
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 2 Ask for provider support
Model tree for kernelpool/Kimi-K3-2bit-UVMAX
Base model
moonshotai/Kimi-K3
# Make sure mlx-vlm is installed # pip install --upgrade mlx-vlm from mlx_vlm import load, generate from mlx_vlm.prompt_utils import apply_chat_template from mlx_vlm.utils import load_config # Load the model model, processor = load("kernelpool/Kimi-K3-2bit-UVMAX") config = load_config("kernelpool/Kimi-K3-2bit-UVMAX") # Prepare input image = ["http://images.cocodataset.org/val2017/000000039769.jpg"] prompt = "Describe this image." # Apply chat template formatted_prompt = apply_chat_template( processor, config, prompt, num_images=1 ) # Generate output output = generate(model, processor, formatted_prompt, image) print(output)