Instructions to use Nohobby/MS-Schisandra-22B-v0.2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nohobby/MS-Schisandra-22B-v0.2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Nohobby/MS-Schisandra-22B-v0.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("Nohobby/MS-Schisandra-22B-v0.2")
model = AutoModelForMultimodalLM.from_pretrained("Nohobby/MS-Schisandra-22B-v0.2")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nohobby/MS-Schisandra-22B-v0.2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nohobby/MS-Schisandra-22B-v0.2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nohobby/MS-Schisandra-22B-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Nohobby/MS-Schisandra-22B-v0.2

SGLang

How to use Nohobby/MS-Schisandra-22B-v0.2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nohobby/MS-Schisandra-22B-v0.2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nohobby/MS-Schisandra-22B-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nohobby/MS-Schisandra-22B-v0.2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nohobby/MS-Schisandra-22B-v0.2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Nohobby/MS-Schisandra-22B-v0.2 with Docker Model Runner:
```
docker model run hf.co/Nohobby/MS-Schisandra-22B-v0.2
```

Nohobby commited on Nov 6, 2024

Commit

671d190

verified ·

1 Parent(s): 589d444

Upload ST-formatting-Schisandra.json

Browse files

Files changed (1) hide show

ST-formatting-Schisandra.json +137 -0

ST-formatting-Schisandra.json ADDED Viewed

	@@ -0,0 +1,137 @@

+{
+    "instruct": {
+        "input_sequence": "\n[INST] ",
+        "output_sequence": "",
+        "last_output_sequence": "",
+        "system_sequence": "\n[INST] Narrative Instructions: ",
+        "stop_sequence": "",
+        "wrap": false,
+        "macro": true,
+        "activation_regex": "",
+        "system_sequence_prefix": "",
+        "system_sequence_suffix": "",
+        "first_output_sequence": "",
+        "skip_examples": true,
+        "output_suffix": "</s>",
+        "input_suffix": "[/INST]",
+        "system_suffix": "[/INST]",
+        "user_alignment_message": "",
+        "system_same_as_user": false,
+        "last_system_sequence": "",
+        "first_input_sequence": "",
+        "last_input_sequence": "",
+        "names_behavior": "always",
+        "names_force_groups": true,
+        "name": "MS-Instruct"
+    },
+    "context": {
+        "story_string": "\n<s>[INST] {{#if system}}{{system}}\n\n{{/if}}{{#if wiBefore}}## World Info:\n{{wiBefore}}\n{{/if}}{{#if description}}## {{char}}'s Description:\n{{description}}\n{{/if}}{{#if personality}}## {{char}}'s Personality:\n{{personality}}\n{{/if}}{{#if persona}}## {{user}}'s Persona:\n{{persona}}\n{{/if}}{{#if scenario}}## Scenario:\n{{scenario}}\n{{/if}}{{#if wiAfter}}## World Info:\n{{wiAfter}}\n{{/if}}{{#if mesExamples}}## {{char}}'s Example Response:\n{{mesExamples}}\n{{/if}}\n[/INST]",
+        "example_separator": "",
+        "chat_start": "## Exchange:",
+        "use_stop_strings": false,
+        "allow_jailbreak": false,
+        "names_as_stop_strings": false,
+        "always_force_name2": false,
+        "trim_sentences": true,
+        "single_line": false,
+        "name": "MS-Context"
+    },
+    "preset": {
+        "temp": 0.9,
+        "temperature_last": true,
+        "top_p": 0.88,
+        "top_k": 100,
+        "top_a": 0,
+        "tfs": 1,
+        "epsilon_cutoff": 0,
+        "eta_cutoff": 0,
+        "typical_p": 1,
+        "min_p": 0.003,
+        "rep_pen": 1.04,
+        "rep_pen_range": 0,
+        "rep_pen_decay": 0,
+        "rep_pen_slope": 0.7,
+        "no_repeat_ngram_size": 0,
+        "penalty_alpha": 0,
+        "num_beams": 1,
+        "length_penalty": 1,
+        "min_length": 0,
+        "encoder_rep_pen": 1,
+        "freq_pen": 0,
+        "presence_pen": 0.03,
+        "skew": 0,
+        "do_sample": true,
+        "early_stopping": true,
+        "dynatemp": false,
+        "min_temp": 0,
+        "max_temp": 2,
+        "dynatemp_exponent": 1,
+        "smoothing_factor": 0,
+        "smoothing_curve": 1,
+        "dry_allowed_length": 2,
+        "dry_multiplier": 0.8,
+        "dry_base": 1.75,
+        "dry_sequence_breakers": "[\"\\n\", \":\", \"\\\"\", \"*\"]",
+        "dry_penalty_last_n": 28672,
+        "add_bos_token": true,
+        "ban_eos_token": false,
+        "skip_special_tokens": true,
+        "mirostat_mode": 0,
+        "mirostat_tau": 5,
+        "mirostat_eta": 0.1,
+        "guidance_scale": 1,
+        "negative_prompt": "",
+        "grammar_string": "",
+        "json_schema": {},
+        "banned_tokens": "",
+        "sampler_priority": [
+            "repetition_penalty",
+            "presence_penalty",
+            "frequency_penalty",
+            "dry",
+            "temperature",
+            "dynamic_temperature",
+            "quadratic_sampling",
+            "top_k",
+            "top_p",
+            "typical_p",
+            "epsilon_cutoff",
+            "eta_cutoff",
+            "tfs",
+            "top_a",
+            "min_p",
+            "mirostat",
+            "xtc",
+            "encoder_repetition_penalty",
+            "no_repeat_ngram"
+        ],
+        "samplers": [
+            "top_k",
+            "tfs_z",
+            "typical_p",
+            "top_p",
+            "min_p",
+            "xtc",
+            "temperature"
+        ],
+        "ignore_eos_token": false,
+        "spaces_between_special_tokens": true,
+        "speculative_ngram": false,
+        "sampler_order": [
+            6,
+            0,
+            1,
+            3,
+            4,
+            2,
+            5
+        ],
+        "logit_bias": [],
+        "xtc_threshold": 0.1,
+        "xtc_probability": 0.23,
+        "rep_pen_size": 0,
+        "genamt": 350,
+        "max_length": 28672,
+        "name": "vKW"
+    }
+}