---
license: apache-2.0
base_model: yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1
library_name: mlx
pipeline_tag: text-generation
tags:
- gemma4
- coding
- code
- reasoning
- thinking
- safetensors
- transformers
- mlx
---
# Update

Added a Jinja chat template so the model can format conversations correctly and work smoothly with `mlx-lm` chat-style inference.

# MLX 8-Bit Quantized: Gemma-4-12B-Coder

This repository contains an 8-bit MLX-converted version of
[yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1](https://huggingface.co/yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1).

The model has been quantized to 8-bit to dramatically reduce memory requirements while retaining near-lossless reasoning and coding capabilities. It is optimized for local inference on Apple Silicon Macs using the `mlx-lm` library.

## How to Use with MLX

Install the required dependency:

```bash
pip install --upgrade mlx-lm
```

Run inference from Python:

```python
from mlx_lm import load, generate

# Load the 8-bit quantized MLX model.
model, tokenizer = load("mlx-community/gemma-4-12b-coder-fable5-composer2.5-8bit")

prompt = "Write a Python script to sort a dictionary by its values."
messages = [{"role": "user", "content": prompt}]

formatted_prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

response = generate(
    model,
    tokenizer,
    prompt=formatted_prompt,
    verbose=True,
    max_tokens=1024,
)
```

```python
response = generate(
    model,
    tokenizer,
    prompt=formatted_prompt,
    verbose=True,
    max_tokens=1024,
    temp=0.0,
)
```

## Base and License

- **Base model:** `google/gemma-4-12B-it`
- **Original fine-tune:** [`yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1`](https://huggingface.co/yuxinlu1/gemma-4-12B-coder-fable5-composer2.5-v1)
- **License:** Apache 2.0

Free to use, modify, and redistribute under the Apache 2.0 license.