Instructions to use ibm-granite/granite-4.0-h-1b-base with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-4.0-h-1b-base with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-4.0-h-1b-base")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-4.0-h-1b-base")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-4.0-h-1b-base")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-4.0-h-1b-base with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-4.0-h-1b-base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-4.0-h-1b-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-4.0-h-1b-base

SGLang

How to use ibm-granite/granite-4.0-h-1b-base with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-4.0-h-1b-base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-4.0-h-1b-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-4.0-h-1b-base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-4.0-h-1b-base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ibm-granite/granite-4.0-h-1b-base with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-4.0-h-1b-base
```

Remove "model.safetensors.index.json" as it references invalid weights files that do not exist

by mrutkows - opened Apr 3

base: refs/heads/main

←

from: refs/pr/1

Discussion Files changed

-1

mrutkows

IBM Granite org Apr 3

The file have a extraneous file named "model.safetensors.index.json" is invalid. It that confuses llama.cpp python script used for GGUF conversion as it references "part" files that do not exist since the weights are all in a single file in the repo. (i.e., "model.safetensors"). This leads to an os.read() error in HF Transformers "file not found". This may have accidentally worked for older HF Transformers versions at the time this model was published; however, on 4.57.1 and newer the logic has changed in Transformers where this read() error is occurring.

Please note that this file does not appear (as it is invalid) in other Granite 4 repos. which use a single weights file; it only appear for:

"granite-4.0-1b-base" or
"granite-4.0-h-1b-base"

I have confirmed that once the file "model.safetensors.index.json" is deleted before calling the python GGUF conversion script in llama.cpp (which uses HF Transformers) will work without error.

Remove "model.safetensors.index.json" as it references invalid weights files that do not exist217b2b48

mrutkows

IBM Granite org Apr 10

@gabegoodhart This is the issue I described to you yesterday and would like to tag who can act on this request...

yousafshah

IBM Granite org 21 days ago

Thanks @mrutkows The file has been removed.

yousafshah changed pull request status to closed 21 days ago

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment