EVA-Qwen2.5-32B-v0.2-mlx-6bit

MLX 6-bit quant of EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2. Disk: ~25 GB. Active memory: ~27 GB plus KV cache.

Quantization

mlx_lm.convert --hf-path EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2 \
  --mlx-path ./EVA-Qwen2.5-32B-v0.2-mlx-6bit -q --q-bits 6

REQUIRED: EOS patch before serving

Same as the 4-bit. Source ships incomplete eos_token_id. Patch needed for ChatML turn boundaries:

python3 -c "
import json
p = './EVA-Qwen2.5-32B-v0.2-mlx-6bit/generation_config.json'
g = json.load(open(p))
g['eos_token_id'] = [151643, 151645]
json.dump(g, open(p, 'w'), indent=2)
"

This quant has it applied.

Recommended sampling

Same as 4-bit:

temperature: 1.0
min_p: 0.05
repetition_penalty: 1.05
repetition_context_size: 2048

Known mlx_lm.server bugs

Same as the 4-bit (XTC tokenizer bug, seed ignored, prompt-cache size cap recommended).

License

Apache-2.0 from base.