Instructions to use ibm-granite/granite-4.2-30b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-4.2-30b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-4.2-30b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-4.2-30b")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-4.2-30b", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-4.2-30b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-4.2-30b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-4.2-30b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-4.2-30b

SGLang

How to use ibm-granite/granite-4.2-30b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-4.2-30b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-4.2-30b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-4.2-30b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-4.2-30b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ibm-granite/granite-4.2-30b with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-4.2-30b
```

Update Serving with SGLang section

by JustinTong - opened about 22 hours ago

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

+10

-11

Update Serving with SGLang sectionee753aa1

JustinTong

IBM Granite org about 22 hours ago

This PR uncomments and updates the Serving with SGLang section.

Changes vs. the commented-out draft:

Removed the "SGLang does not currently support the custom granite_thinking_parser" note. SGLang's built-in nemotron_3 reasoning parser already covers what granite_thinking_parser adds for vLLM, including the enable_thinking=False case (no </think> tag -> content lands in content, reasoning_content is null) and low_effort=True.
The API example also prints reasoning_content.
Added a link to the SGLang Granite 4.2 cookbook with launch recipes and benchmark data.

Verified on SGLang main (d10a656ad8) with all three Granite 4.2 checkpoints on H200 and B200: thinking / non-thinking / low-effort, tool calling, and streaming reasoning + tool-call responses all parse correctly with --reasoning-parser nemotron_3 --tool-call-parser qwen3_coder.

Use auto parser resolution and v0.5.18+ in the SGLang sectionb5cbf0c6

kswanand1

IBM Granite org about 22 hours ago

Looks good to me, thanks!

kswanand1 changed pull request status to merged about 20 hours ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment