Instructions to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF")

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF", dtype="auto")

llama-cpp-python

How to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF",
	filename="KafkaLM-8x7b-German-V0.1-DPO-Q2_K.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M

Use Docker

docker model run hf.co/LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M

SGLang

How to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with Ollama:
```
ollama run hf.co/LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M
```

Unsloth Studio

How to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF to start chatting

Atomic Chat new
Docker Model Runner
How to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with Docker Model Runner:
```
docker model run hf.co/LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M
```

Lemonade

How to use LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull LoneStriker/KafkaLM-8x7b-German-V0.1-DPO-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.KafkaLM-8x7b-German-V0.1-DPO-GGUF-Q4_K_M

List all available models

lemonade list

KafkaLM-8x7b-German-V0.1-DPO-GGUF / huggingface-metadata.txt

LoneStriker

Upload folder using huggingface_hub

8dec3d1 verified over 2 years ago

Raw

History Blame Contribute Delete

4.1 kB

	url: https://huggingface.co/seedboxai/KafkaLM-8x7b-German-V0.1-DPO
	branch: main
	download date: 2024-02-15 10:26:37
	sha256sum:
	5c990c6ea3bdfcde5d6fa97a20f7bb8489e8c7338644e2b22f8cf25818ce9e11 model-00001-of-00039.safetensors
	fe7646c12dd730078c98f2d44297b44458307dd12767b467b4f86a2984f1de47 model-00002-of-00039.safetensors
	13f19a49293abe99a576b0ce3929cc07ef41f27d0c9baaa3f52f68970b6720d4 model-00003-of-00039.safetensors
	d518bfaaaed4ab6745318fd3547c4a30ad8f5dc9ed3880e23597976d34c68554 model-00004-of-00039.safetensors
	7cadcd62f958377f8c6ecd2a8d91789b39d947608ef5966dcbe1c4a1f47049d0 model-00005-of-00039.safetensors
	0db04d581cc8843ee905224a263c533aea2bc040df93ebd9e21f7485ea753132 model-00006-of-00039.safetensors
	ff5119411700efc0b0d80b0e7b364eddbfb9847f91c3296e0b344acd981fcbd2 model-00007-of-00039.safetensors
	f2d6314b97e43abad0e93d8fb3fdf2a1ff8dc7dd1e18d56abf07334ca1b56096 model-00008-of-00039.safetensors
	b83d7e18dfbcca682465e23367fcb93655e2d5fe60ff00371b49844cdeb98b8b model-00009-of-00039.safetensors
	040485aa8f9d4d146e0b248ddcf16d2593ab47a156f026543ba659783bee51d8 model-00010-of-00039.safetensors
	f9fda815c4e401d63794a75664e4454086808cada111685d56ccf9590e6fe54b model-00011-of-00039.safetensors
	e7ac62fa7b4945beaa9aa2541df84b1b3df5cdce7d1af1accd308384ce7ddc9c model-00012-of-00039.safetensors
	41f87398271754733a0667682eb8ef4d211d1aab970ab86cff5bdca322f8a8df model-00013-of-00039.safetensors
	b1c4e598453fdcc597b6291e8b4788b21cab23be4f1b806e0871ab4140680603 model-00014-of-00039.safetensors
	eb037b28ce4cd25ae3f235c22a40735520ec542c14da4c335de353b6155e1835 model-00015-of-00039.safetensors
	0e2e178b8bedb7d402569aa1a1dbfa1fe3161944d2854b255eee94c73162015f model-00016-of-00039.safetensors
	76943d899f6924da8272f4cc50d02e8864de99158d355bc35f5ba0e8915a858e model-00017-of-00039.safetensors
	434c588897cde14211182cfd7d2eade857b0addd8c9c409655f72b1d005b910e model-00018-of-00039.safetensors
	c5bdbc41a8f2386b2ecfb6ff9844e26deec4023ada34b3f2dd555506f5ef9d73 model-00019-of-00039.safetensors
	d58562d1f02dd718c7ed1da1fd0ca6f73c9e356cb3bffcc5d93106b61a9359a3 model-00020-of-00039.safetensors
	26fab439fad7a239ae79a27dad2dbc7ca9ff1c35ffaad6c28922f7a1f91f81b2 model-00021-of-00039.safetensors
	f809057c08a390638024321e4c22d68c25dc5de15033bf875ccb4e3f807d1dd9 model-00022-of-00039.safetensors
	24141d66049ad3723104e8f55a99b90afb2e1dfe74cf42919ae4ef4c8a117ee0 model-00023-of-00039.safetensors
	25d5c910e8270ede8c48a189549fc8818e26913f1a7421ba02b5b95e8e4e22ee model-00024-of-00039.safetensors
	64dec72f25781d1370290a1a67c908fe326e6e088b544e5c3598810d25fa4160 model-00025-of-00039.safetensors
	f5c3a336a71bee7c2eb841c8a4a26b05ad034d691124108c7773b4b42809f0be model-00026-of-00039.safetensors
	4878787645006ae7b52fd7c53bbb26ff450d0ebdd179292d0bc532b6586492d2 model-00027-of-00039.safetensors
	0b7b39c7adc025284ad8e514487ecd4939bafc7386ae9deea94c528970b145d8 model-00028-of-00039.safetensors
	1e588bb3ca105a72e84a20e4230eb799e86539bd5e5e27009446af45ce139dff model-00029-of-00039.safetensors
	58294165ab3abb9ef2a73e7d93aaab5b638bdb419aae60249c50bd7deb088188 model-00030-of-00039.safetensors
	c14e641970ea2f713977d63036547814ccd243f2681352ec941570d9df17312a model-00031-of-00039.safetensors
	2b2c94d4d9fd8996474fde5ff1b568fbbdafeb130c7a7027dd48acfbf2413274 model-00032-of-00039.safetensors
	03f06fe39f9c0fa4dd8dd18894a9c54ffc0267a5b29c677c8fc65ce7d4542010 model-00033-of-00039.safetensors
	f241bf09bf2c482c01267eaca0d609ca76897ebb095f5811d17885bf376584b6 model-00034-of-00039.safetensors
	7dc056901ca382f1411315fdb058f5e81d94c84891eccd84f52f7305b55ce2af model-00035-of-00039.safetensors
	906664c2b302d64f59ac8b77f1574f96b4a8e050fd30a127d57ef3af119a68a1 model-00036-of-00039.safetensors
	57525a3d3d422788d35b394624fea656a5d8f7d1cc987d8377b7963bf006d34d model-00037-of-00039.safetensors
	1f6c9be67ffd7c39216759c94e4eb835ae455b9a1f15d3d2facd28a90774a0ce model-00038-of-00039.safetensors
	cf35d1c660a0ff1e3715fafc0b3575c26e9ed2bc1a9ba618db23aaab1bd44cb2 model-00039-of-00039.safetensors