Instructions to use knifeayumu/Lite-Cydonia-22B-v1.1-50-50 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use knifeayumu/Lite-Cydonia-22B-v1.1-50-50 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="knifeayumu/Lite-Cydonia-22B-v1.1-50-50")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("knifeayumu/Lite-Cydonia-22B-v1.1-50-50")
model = AutoModelForMultimodalLM.from_pretrained("knifeayumu/Lite-Cydonia-22B-v1.1-50-50")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use knifeayumu/Lite-Cydonia-22B-v1.1-50-50 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "knifeayumu/Lite-Cydonia-22B-v1.1-50-50"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "knifeayumu/Lite-Cydonia-22B-v1.1-50-50",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/knifeayumu/Lite-Cydonia-22B-v1.1-50-50

SGLang

How to use knifeayumu/Lite-Cydonia-22B-v1.1-50-50 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "knifeayumu/Lite-Cydonia-22B-v1.1-50-50" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "knifeayumu/Lite-Cydonia-22B-v1.1-50-50",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "knifeayumu/Lite-Cydonia-22B-v1.1-50-50" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "knifeayumu/Lite-Cydonia-22B-v1.1-50-50",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use knifeayumu/Lite-Cydonia-22B-v1.1-50-50 with Docker Model Runner:
```
docker model run hf.co/knifeayumu/Lite-Cydonia-22B-v1.1-50-50
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

"A balancing act between smart do-gooders and creative evil-doers."

The Drummer turns into a Joshi Youchien

This is a merge of pre-trained language models created using mergekit.

GGUF quants : knifeayumu/Lite-Cydonia-22B-v1.1-50-50-GGUF

Inspiration

I thought both BeaverAI/Cydonia-22B-v1f-GGUF and BeaverAI/Cydonia-22B-v1e-GGUF versions were a bit too evil. The sense of morality was too screwed up, and it was quite deterministic (swipes didn't offer much variety) compared to the base model. Then an idea popped into my mind — why not merge it back with the base model? That way, we could give it a sense of "good" again, at least a little. Maybe this would also fix some of the deterministic generations.

Quick testing shows... it works? Zero-shot evil Q&A no longer works, but with a bit of persuasion, it did answer. Unlike knifeayumu/Lite-Cydonia-22B-v1.1-75-25, this merge is more censored but also smarter.

Credits to TheDrummer and BeaverAI who makes such finetunes. "Lightly decensored" is a heavy understatement in this case.

Merge Details

Merge Method

This model was merged using the task arithmetic merge method using mistralai/Mistral-Small-Instruct-2409 as a base.

Models Merged

The following models were included in the merge:

TheDrummer/Cydonia-22B-v1.1

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: mistralai/Mistral-Small-Instruct-2409
    parameters:
      weight: 0.5
  - model: TheDrummer/Cydonia-22B-v1.1
    parameters:
      weight: 0.5
merge_method: task_arithmetic
base_model: mistralai/Mistral-Small-Instruct-2409
dtype: float16