Instructions to use BSC-LT/salamandra7b_rag_prompt_ca-en-es with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BSC-LT/salamandra7b_rag_prompt_ca-en-es with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="BSC-LT/salamandra7b_rag_prompt_ca-en-es")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("BSC-LT/salamandra7b_rag_prompt_ca-en-es")
model = AutoModelForCausalLM.from_pretrained("BSC-LT/salamandra7b_rag_prompt_ca-en-es", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use BSC-LT/salamandra7b_rag_prompt_ca-en-es with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BSC-LT/salamandra7b_rag_prompt_ca-en-es"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BSC-LT/salamandra7b_rag_prompt_ca-en-es",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/BSC-LT/salamandra7b_rag_prompt_ca-en-es

SGLang

How to use BSC-LT/salamandra7b_rag_prompt_ca-en-es with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "BSC-LT/salamandra7b_rag_prompt_ca-en-es" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BSC-LT/salamandra7b_rag_prompt_ca-en-es",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "BSC-LT/salamandra7b_rag_prompt_ca-en-es" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BSC-LT/salamandra7b_rag_prompt_ca-en-es",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use BSC-LT/salamandra7b_rag_prompt_ca-en-es with Docker Model Runner:
```
docker model run hf.co/BSC-LT/salamandra7b_rag_prompt_ca-en-es
```

How to use

This instructed model uses a chat template that must be adhered to the input for conversational use. The easiest way to apply it is using the tokenizer's built-in chat template, as shown in the following snippet.

from transformers import AutoTokenizer, AutoModelForCausalLM
import transformers
import torch

model_id = "BSC-LT/salamandra7b_rag_prompt_ca-en-es"

prompt = "Here is a question that you should answer based on the given context. Write a response that answers the question using only information provided in the context. Provide the answer in Spanish."

context = """Water boils at 100°C (212°F) at standard atmospheric pressure, which is at sea level.
However, this boiling point can vary depending on altitude and atmospheric pressure.
At higher altitudes, where atmospheric pressure is lower, water boils at a lower temperature.
For example, at 2,000 meters (about 6,600 feet) above sea level, water boils at around 93°C (199°F).
"""
instruction = "At what temperature does water boil?"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="cuda",
    torch_dtype=torch.bfloat16
  )

content = f"{prompt}\n\nContext:\n{context}\n\nQuestion:\n{instruction}"
chat = [ { "role": "user", "content": content } ]

prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)

eos_tokens = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|im_end|>"),
  ]

inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
outputs = model.generate(input_ids=inputs.to(model.device), eos_token_id=eos_tokens, max_new_tokens=200)

Downloads last month: 43

Safetensors

Model size

8B params

Tensor type

BF16