Instructions to use davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores")
model = AutoModelForCausalLM.from_pretrained("davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores

SGLang

How to use davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores with Docker Model Runner:
```
docker model run hf.co/davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores
```

Neuronx for mistralai/Mistral-7B-Instruct-v0.2 - Updated Mistral 7B Model on AWS Inferentia2

This model has been exported to the neuron format using specific input_shapes and compiler parameters detailed in the paragraphs below.

Please refer to the 🤗 optimum-neuron documentation for an explanation of these parameters.

Note: To compile the mistralai/Mistral-7B-Instruct-v0.2 on Inf2, you need to update the model config sliding_window (either file or model variable) from null to default 4096.

Usage with 🤗 `optimum-neuron`

>>> from optimum.neuron import pipeline

>>> p = pipeline('text-generation', 'davidshtian/Mistral-7B-Instruct-v0.2-neuron-1x2048-2-cores')
>>> p("My favorite place on earth is", max_new_tokens=64, do_sample=True, top_k=50)
[{'generated_text': 'My favorite place on earth is Hawaii,” she said, her voice bright and clear despite her quietness.
“That place, and the ocean. But it’s hard to ever live there permanently. The ocean is there and it calls to me, but
it’s big and vast and doesn’t allow me a lot of freedom.”'}]

This repository contains tags specific to versions of neuronx. When using with 🤗 optimum-neuron, use the repo revision specific to the version of neuronx you are using, to load the right serialized checkpoints.

Arguments passed during export

input_shapes

{
  "batch_size": 1,
  "sequence_length": 2048,
}

compiler_args

{
  "auto_cast_type": "bf16",
  "num_cores": 2,
}

Downloads last month: 3

Neuronx for mistralai/Mistral-7B-Instruct-v0.2 - Updated Mistral 7B Model on AWS Inferentia2

Usage with 🤗 optimum-neuron

Arguments passed during export

Usage with 🤗 `optimum-neuron`