Instructions to use son-of-man/HoloViolet-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use son-of-man/HoloViolet-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="son-of-man/HoloViolet-7B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("son-of-man/HoloViolet-7B")
model = AutoModelForCausalLM.from_pretrained("son-of-man/HoloViolet-7B", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use son-of-man/HoloViolet-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "son-of-man/HoloViolet-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "son-of-man/HoloViolet-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/son-of-man/HoloViolet-7B

SGLang

How to use son-of-man/HoloViolet-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "son-of-man/HoloViolet-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "son-of-man/HoloViolet-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "son-of-man/HoloViolet-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "son-of-man/HoloViolet-7B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use son-of-man/HoloViolet-7B with Docker Model Runner:
```
docker model run hf.co/son-of-man/HoloViolet-7B
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

HoloViolet-7B-test5

The best version of HoloViolet. At this point it seems outclassed by twizzler, but I still love it for its proactive writing and sometimes unexpected outputs.

Update: quants available over here, kudos to mradermacher.

A very discriptive model, harnessing the literary benefits of KoboldAI's Mistral Holodeck, but less schizo. Manages to get an understanding of the situation, doesn't ignore context nearly as much, while expanding on it creatively. It's not very subtle about telling you a character's intentions, as it is still a 7B, but it writes well imo. GreenNode V1olet is a great model for supplying smarts since it doesn't gravitate towards GPT'isms nearly as much as the other smart mistral tunes. Use Roleplay prompt preset on sillytavern, I find simple prompts work better with these smaller models.

HoloViolet-7B-test5 is a merge of the following models using LazyMergekit:

🧩 Configuration

slices:
  - sources:
      - model: GreenNode/GreenNode-mini-7B-multilingual-v1olet
        layer_range: [0, 32]
      - model: KoboldAI/Mistral-7B-Holodeck-1
        layer_range: [0, 32]
merge_method: slerp
base_model: GreenNode/GreenNode-mini-7B-multilingual-v1olet
parameters:
  t:
    - value: 0.32
dtype: bfloat16

💻 Usage

!pip install -qU transformers accelerate

from transformers import AutoTokenizer
import transformers
import torch

model = "son-of-man/HoloViolet-7B-test5"
messages = [{"role": "user", "content": "What is a large language model?"}]

tokenizer = AutoTokenizer.from_pretrained(model)
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
pipeline = transformers.pipeline(
    "text-generation",
    model=model,
    torch_dtype=torch.float16,
    device_map="auto",
)

outputs = pipeline(prompt, max_new_tokens=256, do_sample=True, temperature=0.7, top_k=50, top_p=0.95)
print(outputs[0]["generated_text"])

Downloads last month: 5

Safetensors

Model size

7B params

Tensor type

BF16

Model tree for son-of-man/HoloViolet-7B

GreenNode/GreenNode-mini-7B-multilingual-v1olet

KoboldAI/Mistral-7B-Holodeck-1

Merge model

this model

Quantizations

1 model