granite-guardian-4.1-8b-mlx-4bit

MLX 4-bit quant of ibm-granite/granite-guardian-4.1-8b. IBM's safety classifier; flags harmful content across categories (harm, social bias, jailbreak, violence, profanity, sexual content, unethical behavior).

Disk: ~4.4 GB. Active memory while serving: ~5 GB plus KV cache.

Quantization

mlx_lm.convert --hf-path ibm-granite/granite-guardian-4.1-8b \
  --mlx-path ./granite-guardian-4.1-8b-mlx-4bit -q --q-bits 4

Usage

Granite Guardian uses a structured prompt to classify a piece of input/output text. The model returns "Yes" or "No" plus a confidence score. See the base model card for the prompt template (it's risk-category-specific).

For a quick check via mlx_lm.server's chat completions endpoint, send the structured prompt as the user message and read the assistant token.

Recommended sampling

Classification, not creative output:

temperature: 0.0
max_tokens: 16

Greedy decoding is appropriate. Bail early since the response is short.

License

Apache-2.0 from base.