Instructions to use ibm-granite/granite-3.0-8b-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ibm-granite/granite-3.0-8b-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ibm-granite/granite-3.0-8b-instruct")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ibm-granite/granite-3.0-8b-instruct")
model = AutoModelForCausalLM.from_pretrained("ibm-granite/granite-3.0-8b-instruct", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ibm-granite/granite-3.0-8b-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ibm-granite/granite-3.0-8b-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-8b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ibm-granite/granite-3.0-8b-instruct

SGLang

How to use ibm-granite/granite-3.0-8b-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ibm-granite/granite-3.0-8b-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-8b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ibm-granite/granite-3.0-8b-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ibm-granite/granite-3.0-8b-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ibm-granite/granite-3.0-8b-instruct with Docker Model Runner:
```
docker model run hf.co/ibm-granite/granite-3.0-8b-instruct
```

Appreciate the model drop!

by Nitral-AI - opened Oct 21, 2024

Discussion

Nitral-AI

Oct 21, 2024

But why is it only 4k? Its 2024 man, those are rookie numbers.

gabegoodhart

IBM Granite org Oct 21, 2024

Hi @Nitral-AI , thanks for your interest in these new Granite models. You're right that 4k is a short context window by today's standards. Long context is coming soon (expected to be ready by the end of the year).

Training longer context into the models is sequential, so in order to train longer context lengths, we first needed to train for shorter lengths. These 3.0 short-context models are useful for a number of use cases. We opted to release them as-is with 4k context so that users can start with them now if they fit the use case. For long context (128k), keep an eye out for the upcoming 3.1 drop!

deleted

Oct 21, 2024

@gabegoodhart

Code version too, i hope ?

keep an eye out for the upcoming 3.1 drop!

gabegoodhart

IBM Granite org Oct 21, 2024

@Nurb4000 Yes! These models are already trained with code-centric capabilities, but due to their short context windows we don't recommend moving off of granite-code (yet!)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment