Instructions to use EldritchLabs/KrakenSakura-Maelstrom-12B-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use EldritchLabs/KrakenSakura-Maelstrom-12B-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="EldritchLabs/KrakenSakura-Maelstrom-12B-v1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("EldritchLabs/KrakenSakura-Maelstrom-12B-v1")
model = AutoModelForMultimodalLM.from_pretrained("EldritchLabs/KrakenSakura-Maelstrom-12B-v1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

NeMo
How to use EldritchLabs/KrakenSakura-Maelstrom-12B-v1 with NeMo:
```
# tag did not correspond to a valid NeMo domain.
```
Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use EldritchLabs/KrakenSakura-Maelstrom-12B-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "EldritchLabs/KrakenSakura-Maelstrom-12B-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EldritchLabs/KrakenSakura-Maelstrom-12B-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/EldritchLabs/KrakenSakura-Maelstrom-12B-v1

SGLang

How to use EldritchLabs/KrakenSakura-Maelstrom-12B-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "EldritchLabs/KrakenSakura-Maelstrom-12B-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EldritchLabs/KrakenSakura-Maelstrom-12B-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "EldritchLabs/KrakenSakura-Maelstrom-12B-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "EldritchLabs/KrakenSakura-Maelstrom-12B-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use EldritchLabs/KrakenSakura-Maelstrom-12B-v1 with Docker Model Runner:
```
docker model run hf.co/EldritchLabs/KrakenSakura-Maelstrom-12B-v1
```

How exactly is this used?

by Iskar-kholin - opened May 12

Discussion

Iskar-kholin

May 12

I am not familiar with this type of models and how exactly you are meant to engage with them. Just drop it in LM studio and chat with it like a person or does it require a specialized chat program?

Naphula

EldritchLabs org May 12

It depends on what system you are using. Since i have windows and 8GB VRAM I use kobold.cpp with GGUF extension (usually Q6_K for 12B, or IQ4_XS for 24B).

But LM Studio should work fine as well. This does not require any special program and it should have a distinct "chat style" with few to no refusals.

Just make sure to use ChatML or Mistral Tekken chat template for best results. Kobold.cpp has these as options in the settings menu.

yano2mch

May 13

I recommend KoboldCPP, no install, fairly lightweight, gives you a basic interface without needing anything else (though you can plus SillyTavern or the like out).

Lot of options, but just select your max context size and stay within that and it should do fairlywell. Anything that doesn't fit in you VRam will run on your CPU and RAM, while a lot slower 16B and under aren't all that bad, though i consider 5t/s a usable speed while 1t/s and slower is an unusable speed.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment