Instructions to use pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g")
model = AutoModelForCausalLM.from_pretrained("pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g

SGLang

How to use pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g with Docker Model Runner:
```
docker model run hf.co/pszemraj/stablelm-7b-sft-v7e3-autogptq-4bit-128g
```

stablelm-7b-sft-v7e3-autogptq-4bit-128g / README.md

pszemraj

Super-squash branch 'main' using huggingface_hub

246c1fa verified 5 months ago

preview code

raw

history blame contribute delete

2.12 kB

metadata

license: cc-by-sa-4.0
inference: false
pipeline_tag: text-generation
tags:
  - gptq
  - auto-gptq
  - quantization
  - sft
  - openassistant
language:
  - en

Open-Assistant StableLM-7B SFT-7 Model: GPTQ 4-bit

A GPTQ version of the Open-Assistant StableLM-7B SFT-7 Model quantized via AutoGPTQ

Will update this more later - take a look at the colab for now to see an inference example, or a stripped-down version of the notebook here

Note: this model uses special tokens, structured as: <|prompter|>What's the Earth total population<|endoftext|><|assistant|>

Query Example

I want you to act as an expert linguist. I'm trying to uncover the true meaning of the corporate buzzword 'value'. For example: "I've been furiously delivering value today". What could it really mean?

The corporate buzzword "value" can have a few different meanings, but one possible explanation is that it refers to the perceived worth or significance that is attached to something or someone. In this sense, it could be used to describe the perceived benefits that are associated with a particular product, service, or experience.

For example, if someone were to say "I've been furiously delivering value today," they might be emphasizing the importance of the product or service they are using to make that statement. This could be a way of signaling to the person or company that the value they are delivering is high quality and relevant.

It's worth noting that the meaning of "value" can be subjective and context-dependent, and it can be difficult to pin down. However, in many cases, it can be used to communicate the value or significance that is being attached to something.<|endoftext|>