Instructions to use google/gemma-2-2b-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/gemma-2-2b-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="google/gemma-2-2b-it")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b-it")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use google/gemma-2-2b-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/gemma-2-2b-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-2-2b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/google/gemma-2-2b-it

SGLang

How to use google/gemma-2-2b-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/gemma-2-2b-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-2-2b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/gemma-2-2b-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-2-2b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use google/gemma-2-2b-it with Docker Model Runner:
```
docker model run hf.co/google/gemma-2-2b-it
```

unable to access model despite clicking on prompted button to gain access.

#54

by Khyatikhandelwal20 - opened Dec 22, 2024

Discussion

Khyatikhandelwal20

Dec 22, 2024

now it simply says that I am awaiting access from owners. Please help.

LazerLambda

Dec 22, 2024

Same problem for one week now

Water-Ty

Dec 23, 2024

Same here :(

LazerLambda

Dec 24, 2024

Found the answer here: https://huggingface.co/google/gemma-2-9b-it/discussions/53

You have to request access through kaggle

fredwang1972

Dec 26, 2024

me too.

lkv

Google org Dec 26, 2024

Hi @fredwang1972 and @Khyatikhandelwal20 ,

For now to get instant approval could you please visit the Kaggle page for Gemma and request access to the model weights through Hugging Face. Kindly find this kaggle page reference link.

Thank you.

lkv

Google org Jan 3, 2025

•

edited Jan 3, 2025

Hi @fredwang1972 and @Khyatikhandelwal20 ,

Follow the link provided below and complete the steps to gain access to the “Google Gemma models on HuggingFace”.
https://www.kaggle.com/models/google/gemma-2/license/consent

Click on “Verify via Hugging Face” and also click “continue with Hugging Face”.

By following above step then will get as

Click on "Authorize," which will redirect you to the Kaggle Gemma Consent Form page. Fill out the required fields, accept the terms, and you will receive access to the Google Gemma models on HuggingFace in less than a minute.

Kindly try and let me know if you have any concerns. Thank you.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment