Instructions to use google/recurrentgemma-2b-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/recurrentgemma-2b-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="google/recurrentgemma-2b-it")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("google/recurrentgemma-2b-it")
model = AutoModelForMultimodalLM.from_pretrained("google/recurrentgemma-2b-it")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use google/recurrentgemma-2b-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/recurrentgemma-2b-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/recurrentgemma-2b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/google/recurrentgemma-2b-it

SGLang

How to use google/recurrentgemma-2b-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/recurrentgemma-2b-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/recurrentgemma-2b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/recurrentgemma-2b-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/recurrentgemma-2b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use google/recurrentgemma-2b-it with Docker Model Runner:
```
docker model run hf.co/google/recurrentgemma-2b-it
```

Are you going to release your Hawk architecture models as well as your larger Griffin models (e.g: Griffin 14B) from your paper?

by Joseph717171 - opened Apr 11, 2024

Discussion

Joseph717171

Apr 11, 2024

•

edited Apr 11, 2024

In your paper, you detail another architecture: Hawk, and you list larger Griffin models (Griffin 14B). Are you going to release these other models as well? 🙏😁

Joseph717171 changed discussion title from Are you going to release your hawk models as well as the larger Griffin models from your paper? to Are you going to release your Hawk architecture models as well as your larger Griffin models (e.g: Griffin 14B) from your paper? Apr 11, 2024

slsmith

Google org Apr 11, 2024

Unfortunately we cannot release trained models from the Griffin paper, as Google can't release weights from models trained on MassiveText (since it contains the Books dataset).

It should be straightforward however to create the config for Hawk from the code we released on GitHub. Eg we instantiate an example model config in this example:
https://github.com/google-deepmind/recurrentgemma/blob/0f5ca57442f17c7309c70b0228fd8e5505cbdaa1/examples/simple_run_jax.py#L43

block_types is a tuple which lists the temporal-mixing blocks used (ie the length of block_types is the model depth). To get a Hawk model, simply repeat "recurrentgemma.TemporalBlockType.RECURRENT" for the desired depth.

This is a good suggestion though, and we will look into adding a clearer description of how to create the different models from the paper. I don't actually know how the HuggingFace code is structured, but I imagine it is also straightforward to create Hawk models from there as well!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment