Instructions to use Vortex5/Maroon-Sunset-12B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Vortex5/Maroon-Sunset-12B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Vortex5/Maroon-Sunset-12B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("Vortex5/Maroon-Sunset-12B")
model = AutoModelForMultimodalLM.from_pretrained("Vortex5/Maroon-Sunset-12B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Vortex5/Maroon-Sunset-12B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Vortex5/Maroon-Sunset-12B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vortex5/Maroon-Sunset-12B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Vortex5/Maroon-Sunset-12B

SGLang

How to use Vortex5/Maroon-Sunset-12B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Vortex5/Maroon-Sunset-12B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vortex5/Maroon-Sunset-12B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Vortex5/Maroon-Sunset-12B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Vortex5/Maroon-Sunset-12B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Vortex5/Maroon-Sunset-12B with Docker Model Runner:
```
docker model run hf.co/Vortex5/Maroon-Sunset-12B
```

Vortex5 commited on Jan 20

Commit

36b5694

verified ·

1 Parent(s): 6ea2d77

Update README.md

Browse files

Files changed (1) hide show

README.md +24 -13

README.md CHANGED Viewed

@@ -265,8 +265,12 @@ tokenizer:
   display: block;
   margin-top: 18px;
   border-radius: 14px;
-  border: 1px solid rgba(255,255,255,.12);
-  background: rgba(0,0,0,.25);
   overflow: hidden;
 }
@@ -283,9 +287,13 @@ tokenizer:
   letter-spacing: .14em;
   text-transform: uppercase;
-  color: rgba(255,244,236,.90);
-  background: linear-gradient(180deg, rgba(255,255,255,.06), rgba(255,255,255,.03));
-  border-bottom: 1px solid rgba(255,255,255,.10);
 }
 .card-details summary::-webkit-details-marker{
@@ -296,8 +304,8 @@ tokenizer:
   content: "";
   width: 10px;
   height: 10px;
-  border-right: 2px solid rgba(255, 207, 109, 0.85);
-  border-bottom: 2px solid rgba(255, 207, 109, 0.85);
   transform: rotate(-45deg);
   justify-self: end;
   margin-left: 10px;
@@ -307,8 +315,8 @@ tokenizer:
 .card-details[open] summary::after{
   transform: rotate(45deg);
-  border-right-color: rgba(255, 208, 181, 0.95);
-  border-bottom-color: rgba(255, 208, 181, 0.95);
   opacity: 1;
 }
@@ -316,14 +324,17 @@ tokenizer:
   margin: 0;
   padding: 14px 14px 16px;
-  font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "JetBrains Mono", "Liberation Mono", monospace;
   font-size: 0.86rem;
   line-height: 1.6;
-  color: rgba(255, 244, 236, 0.86);
-  text-align: left;
-  background: rgba(0,0,0,.18);
   overflow-x: auto;
 }

   display: block;
   margin-top: 18px;
   border-radius: 14px;
+  border: 1px solid rgba(240,240,255,0.12);
+  background:
+    radial-gradient(700px 200px at 20% 0%, rgba(202, 66, 92, 0.10), transparent 55%),
+    rgba(0,0,0,0.28);
   overflow: hidden;
 }
   letter-spacing: .14em;
   text-transform: uppercase;
+  color: rgba(255,244,236,.92);
+  background:
+    radial-gradient(600px 180px at 20% 0%, rgba(202, 66, 92, 0.12), transparent 55%),
+    rgba(255,255,255,0.04);
+  border-bottom: 1px solid rgba(240,240,255,0.10);
 }
 .card-details summary::-webkit-details-marker{
   content: "";
   width: 10px;
   height: 10px;
+  border-right: 2px solid rgba(202, 66, 92, 0.85);
+  border-bottom: 2px solid rgba(202, 66, 92, 0.85);
   transform: rotate(-45deg);
   justify-self: end;
   margin-left: 10px;
 .card-details[open] summary::after{
   transform: rotate(45deg);
+  border-right-color: rgba(202, 66, 92, 0.95);
+  border-bottom-color: rgba(202, 66, 92, 0.95);
   opacity: 1;
 }
   margin: 0;
   padding: 14px 14px 16px;
+  font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, "JetBrains Mono", monospace;
   font-size: 0.86rem;
   line-height: 1.6;
+  color: rgba(255, 244, 236, 0.88);
+  background:
+    radial-gradient(600px 200px at 20% 0%, rgba(166, 28, 59, 0.14), transparent 55%),
+    rgba(0,0,0,0.22);
+  border-top: 1px solid rgba(240,240,255,0.08);
   overflow-x: auto;
 }