EVA-Qwen2.5-32B-v0.2-mlx-4bit

MLX 4-bit quant of EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2. Qwen2.5-32B creative-writing finetune. Disk: ~17 GB. Active memory: ~19 GB plus KV cache.

Quantization

mlx_lm.convert --hf-path EVA-UNIT-01/EVA-Qwen2.5-32B-v0.2 \
  --mlx-path ./EVA-Qwen2.5-32B-v0.2-mlx-4bit -q --q-bits 4

REQUIRED: EOS patch before serving

The published EVA finetune ships generation_config.json with eos_token_id: 151643 (single-int, <|endoftext|>) but emits ChatML turn boundaries that use <|im_end|> (151645). Without patching, the server doesn't recognize turn ends and <|im_end|> leaks into responses as raw text.

After conversion, fix it:

python3 -c "
import json
p = './EVA-Qwen2.5-32B-v0.2-mlx-4bit/generation_config.json'
g = json.load(open(p))
g['eos_token_id'] = [151643, 151645]
json.dump(g, open(p, 'w'), indent=2)
"

This quant has the patch applied.

Recommended sampling (RP)

temperature: 1.0
min_p: 0.05
repetition_penalty: 1.05
repetition_context_size: 2048

EVA's natural temperature is higher than typical (training data is creative-writing biased). Drop to 0.9 if outputs ramble or get incoherent; raise to 1.1 if too rigid.

Speculative decoding

No public Qwen2.5-3B (or smaller) draft model with matching tokenizer is widely deployed in MLX format. Spec-dec disabled by default. If you find one, it should drop in cleanly.

Known mlx_lm.server bugs

• xtc_special_tokens flatten bug crashes XTC sampling on Qwen2.5 tokenizers. The server passes tokenizer.encode("\n") (a list) inside a list with tokenizer.eos_token_id (an int), producing a mixed-rank index that fails MLX. Patch in your launcher (or wait for upstream fix). Until then, do not enable XTC with this model.
• seed parameter silently ignored in mlx-lm 0.31.3.
• Prompt-cache KV grows unbounded; set --prompt-cache-size 12 or lower.

EVA's known wallowing tendency

EVA-Qwen2.5 has a documented tendency to "wallow" in emotional beats and not progress narrative. Counter via Author's Note pinned at depth 2:

[{{char}} advances the scene each turn with concrete decisions or actions. Avoid looping back to feelings or topics already established.]

License

Apache-2.0 from base.