Instructions to use Ateron/Sketch-Cydonia-24B-V1.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Ateron/Sketch-Cydonia-24B-V1.2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Ateron/Sketch-Cydonia-24B-V1.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("Ateron/Sketch-Cydonia-24B-V1.2")
model = AutoModelForMultimodalLM.from_pretrained("Ateron/Sketch-Cydonia-24B-V1.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Ateron/Sketch-Cydonia-24B-V1.2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Ateron/Sketch-Cydonia-24B-V1.2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ateron/Sketch-Cydonia-24B-V1.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Ateron/Sketch-Cydonia-24B-V1.2

SGLang

How to use Ateron/Sketch-Cydonia-24B-V1.2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Ateron/Sketch-Cydonia-24B-V1.2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ateron/Sketch-Cydonia-24B-V1.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Ateron/Sketch-Cydonia-24B-V1.2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Ateron/Sketch-Cydonia-24B-V1.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Ateron/Sketch-Cydonia-24B-V1.2 with Docker Model Runner:
```
docker model run hf.co/Ateron/Sketch-Cydonia-24B-V1.2
```

This is an amazing model! 👏

by NeuR0mancR - opened Mar 20

Discussion

NeuR0mancR

Mar 20

I don't know how carefully you tested/weighted your choices for this merge, if it's a fluke or genius. I just wanted to shout in the aether how much I appreciated your model.

Also, I'd like to know if you ran your sampling with Top NSigma, DRY or mirostat. It's quite resilient, but I'd like to test it further from the merger's perspective, if possible.

Ateron

Owner Mar 21

I don't know how carefully you tested/weighted your choices for this merge, if it's a fluke or genius. I just wanted to shout in the aether how much I appreciated your model.

Also, I'd like to know if you ran your sampling with Top NSigma, DRY or mirostat. It's quite resilient, but I'd like to test it further from the merger's perspective, if possible.

Thanks! Yeah, I did the weight intentional, and tested for couple of days. Had like 4 other merge settings that failed. My samplings are pretty casual, just temperature 1, min_p around 0.05-0.1, top_k adjust 85-95.

NeuR0mancR

Mar 21

Had like 4 other merge settings that failed.

Hopefully a few gave you good insights or ideas for future ones. I had one turn full Naked Lunch on me with very tame samplings setting, failures can be interesting sometimes.

My samplings are pretty casual, just temperature 1, min_p around 0.05-0.1, top_k adjust 85-95.

Thank you! I'll give it a try. If it matters to you, I tested it with Temp 0.85, Min-P 0.1, Rep-Pen 1.05, Top-P 1, Top-K 0, DRY 0.8.

NeuR0mancR changed discussion status to closed Mar 23

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment