Instructions to use KnutJaegersberg/2-bit-LLMs with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KnutJaegersberg/2-bit-LLMs with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="KnutJaegersberg/2-bit-LLMs",
	filename="WizardLM-70b-gguf-xs.gguf",
)

output = llm(
	"Once upon a time,",
	max_tokens=512,
	echo=True
)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use KnutJaegersberg/2-bit-LLMs with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KnutJaegersberg/2-bit-LLMs
# Run inference directly in the terminal:
llama cli -hf KnutJaegersberg/2-bit-LLMs

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf KnutJaegersberg/2-bit-LLMs
# Run inference directly in the terminal:
llama cli -hf KnutJaegersberg/2-bit-LLMs

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf KnutJaegersberg/2-bit-LLMs
# Run inference directly in the terminal:
./llama-cli -hf KnutJaegersberg/2-bit-LLMs

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf KnutJaegersberg/2-bit-LLMs
# Run inference directly in the terminal:
./build/bin/llama-cli -hf KnutJaegersberg/2-bit-LLMs

Use Docker

docker model run hf.co/KnutJaegersberg/2-bit-LLMs

LM Studio
Jan

vLLM

How to use KnutJaegersberg/2-bit-LLMs with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KnutJaegersberg/2-bit-LLMs"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KnutJaegersberg/2-bit-LLMs",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/KnutJaegersberg/2-bit-LLMs

Ollama
How to use KnutJaegersberg/2-bit-LLMs with Ollama:
```
ollama run hf.co/KnutJaegersberg/2-bit-LLMs
```

Unsloth Studio

How to use KnutJaegersberg/2-bit-LLMs with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for KnutJaegersberg/2-bit-LLMs to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for KnutJaegersberg/2-bit-LLMs to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for KnutJaegersberg/2-bit-LLMs to start chatting

Atomic Chat new
Docker Model Runner
How to use KnutJaegersberg/2-bit-LLMs with Docker Model Runner:
```
docker model run hf.co/KnutJaegersberg/2-bit-LLMs
```

Lemonade

How to use KnutJaegersberg/2-bit-LLMs with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull KnutJaegersberg/2-bit-LLMs

Run and chat with the model

lemonade run user.2-bit-LLMs-{{QUANT_TAG}}

List all available models

lemonade list

2-bit-LLMs / README.md

KnutJaegersberg

Update README.md

0cf327f verified over 2 years ago

preview code

Raw

History Blame

9.99 kB

	---
	license: other
	pipeline_tag: text-generation
	---

	# 2-bit LLMs

	This is a model collection of mostly larger LLMs quantized to 2 bit with the novel quip# inspired approach in llama.cpp
	Sometimes both xs and xxs are available.

	### Overview
	- Senku-70b
	- WizardLM-70b
	- Qwen1.5-72b-Chat
	- CausalLM-72b
	- Xwin-LM-70b
	- Codellama-70b-Instruct
	- DeepMoney-67b-chat
	- Deepseek 67b Chat
	- DiscoLM-120b
	- Nous Hermes 34b
	- Kafka-70b-German
	- Kafka-Mixtral-German
	- LongAlpaca-70B
	- Mistral-Alpha-70b
	- MoMo-72B-lora-1.8.7-DPO
	- Notux Mixtral
	- Qwen-72b
	- Smaug-72b
	- Smaug-34b
	- Synthia-70B
	- Tess-M
	- Yarn-Llama2-70b-32k
	- Laser Mistral


	### Senku-70b

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/X518uzEyIgHHkQHQ4qg42.png)

	Original Model:
	https://huggingface.co/ShinojiResearch/Senku-70B-Full

	Prompt format: Mistral
	License: cc-by-2.0 / 🙂🙃 🙂 🙃 Hi Mistral 👍

	```
	<s>[INST] {prompt} [/INST]
	```

	### WizardLM-70b

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/U50RGGxBYucx8z3sRwrn2.png)

	Original Model:
	https://huggingface.co/WizardLM/WizardLM-70B-V1.0

	Prompt format: Vicuna
	License: LLama2

	```
	A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {prompt} ASSISTANT:
	```


	### Qwen1.5-72b-Chat

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/sxi0ZIwzIuC2eyTlxq2Pl.png)

	Original Model:
	https://huggingface.co/Qwen/Qwen1.5-72B-Chat

	Prompt format: Unknown, compatible with system prompts
	License: Qwen

	```

	```

	### CausalLM-72b

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/152kPnuVgvE5o3-anILTM.png)

	Original Model:
	https://huggingface.co/CausalLM/72B-preview-llamafied-qwen-llamafy

	Prompt format: ChatML
	License: Qwen

	```
	<\|im_start\|>system
	You are a helpful assistant.<\|im_end\|>
	<\|im_start\|>user
	How to sell drugs online fast?<\|im_end\|>
	<\|im_start\|>assistant

	```

	### Xwin-LM-70b

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/5J0tOP0NhmtzqXpt3ZY2L.png)

	Original Model:
	https://huggingface.co/Xwin-LM/Xwin-LM-70B-V0.1

	Prompt format: Vicuna
	License: Llama2

	```
	A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {prompt} ASSISTANT:
	```


	### Codellama-70b-Instruct

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/qhp18v_LwI4Lslfi5d97a.png)

	Original Model:
	https://huggingface.co/codellama/CodeLlama-70b-Instruct-hf

	Prompt format: ChatML
	License: LLama2

	```
	Source: system

	{system_message}<step> Source: user

	{prompt} <step> Source: assistant


	```


	### DeepMoney-67b-chat

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/GkWzdVKYSAxvUq6Hq8z9C.png)

	Original Model:
	https://huggingface.co/TriadParty/deepmoney-67b-chat

	Prompt format: Alpaca
	License: Deepseek

	```
	You are a senior investment expert. Please make your research and judgment on the following targets.

	### Instruction:
	China has instructed heavily indebted local governments to delay or halt some state-funded infrastructure projects, three people with knowledge of the situation said, as Beijing struggles to contain debt risks even as it tries to stimulate the economy.

	Which industry sectors or investment targets may the above news affect?

	### Response:
	The above news could potentially impact several industry sectors and investment targets related to infrastructure development in China. Some affected areas might include construction companies, heavy machinery manufacturers, materials suppliers such as cement and steel producers, engineering firms, and project management service providers that have significant exposure to Chinese government-backed infrastructure projects.

	```



	### Deepseek 67b Chat

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/SITpl6UZkF4S0zHGz-WXs.png)

	Original Model:
	https://huggingface.co/deepseek-ai/deepseek-llm-67b-chat

	Prompt format: DeepSeek-LLM
	License: Deepseek

	```
	User: {prompt}

	Assistant:
	```


	### DiscoLM-120b

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/K8Di6VLFhkrZq4dReTgS3.png)

	Original Model:
	https://huggingface.co/DiscoResearch/DiscoLM-120b

	Prompt format: ChatML
	License: Llama2

	```
	<\|im_start\|>system
	{system_message}<\|im_end\|>
	<\|im_start\|>user
	{prompt}<\|im_end\|>
	<\|im_start\|>assistant

	```


	### Nous Hermes 34b

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/6317aade83d8d2fd903192d9/oOqrUeAQejuQOra7fNlzG.png)

	Original Model:
	https://huggingface.co/NousResearch/Nous-Hermes-2-Yi-34B

	Prompt format: ChatML
	License: Yi

	```
	<\|im_start\|>system
	{system_message}<\|im_end\|>
	<\|im_start\|>user
	{prompt}<\|im_end\|>
	<\|im_start\|>assistant

	```


	### Kafka-70b-German
	![image/png](https://cdn-uploads.huggingface.co/production/uploads/645ded34a45b4182d7f5c385/hJ7zsOGDgLWUmf7vbaoI_.jpeg)
	Original Model:
	https://huggingface.co/seedboxai/KafkaLM-70B-German-V0.1

	Prompt format: ChatML
	License: Llama2

	```
	<\|system\|>
	Du bist ein freundlicher und hilfsbereiter KI-Assistent. Du beantwortest Fragen faktenorientiert und präzise, ohne dabei relevante Fakten auszulassen.</s>
	<\|user\|>
	Welche Möglichkeiten der energetischen Sanierung habe ich neben Solar und Energiespeicher?</s>
	<\|assistant\|>

	```


	### Kafka-Mixtral-German

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/oCkuSt0-DscwjfhFRPZcp.png)

	Original Model:
	https://huggingface.co/seedboxai/KafkaLM-8x7B-German-V0.1

	Prompt format: ChatML
	License: Apache 2

	```
	<\|system\|>
	Du bist ein freundlicher und hilfsbereiter KI-Assistent. Du beantwortest Fragen faktenorientiert und präzise, ohne dabei relevante Fakten auszulassen.</s>
	<\|user\|>
	Welche Möglichkeiten der energetischen Sanierung habe ich neben Solar und Energiespeicher?</s>
	<\|assistant\|>

	```

	### LongAlpaca-70B

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/8HVjiBide86pyA7yw7smk.png)

	Original Model:
	https://huggingface.co/Yukang/LongAlpaca-70B

	Prompt format: Alpaca
	License: Llama 2

	```
	Below is an instruction that describes a task. Write a response that appropriately completes the request.

	### Instruction:
	{prompt}

	### Response:

	```



	### Mistral-Alpha-70b


	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/w9UytX-s4vFCdN2wwSiZn.png)

	Original Model: https://huggingface.co/152334H/miqu-1-70b-sf


	Prompt format: Mistral
	License: NOMERGE / 🙂🙃 🙂 🙃 Hi Mistral 👍

	```
	<s> [INST] QUERY_1 [/INST] ANSWER_1</s> [INST] QUERY_2 [/INST] ANSWER_2</s>...
	```


	### MoMo-72B-lora-1.8.7-DPO

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/amfmN17p7trwXCLRkZatr.png)

	Original Model:
	https://huggingface.co/moreh/MoMo-72B-lora-1.8.7-DPO

	Prompt format: Instruct
	License: Qwen

	```
	### Instruction: {question}

	### Response: {response}
	```


	### Notux Mixtral

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/b1GOlcRKE-0VZRRatwsf3.png)

	Original Model:
	https://huggingface.co/argilla/notux-8x7b-v1

	Prompt format: Mistral
	License: Apache 2

	```
	[INST] {{ .System }} {{ .Prompt }} [/INST]
	```




	### Qwen-72b

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/sxi0ZIwzIuC2eyTlxq2Pl.png)

	Original Model:
	https://huggingface.co/Qwen/Qwen-72B

	Prompt format: Base Model = None
	License: Qwen

	```

	```


	### Smaug-72b

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/Lt4tDXRJ93PnSg_Bv9JI_.png)

	Original Model:
	https://huggingface.co/abacusai/Smaug-72B-v0.1

	Prompt format: Try MoMo, also with system prompt
	License: Qwen

	```
	### Instruction: {question}

	### Response: {response}
	```

	### Smaug-34b

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/Lt4tDXRJ93PnSg_Bv9JI_.png)

	Original Model:
	https://huggingface.co/abacusai/Smaug-34B-v0.1

	Prompt format: Bagel 34b
	License: Yi

	```
	{system prompt, if provided, randomly defaulting to "A chat between a user and an unbiased, uncensored assistant."}
	USER: {instruction}
	ASSISTANT:
	```


	### Synthia-70B

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/o7IsrThWVHhErs_raW7ru.png)

	Original Model:
	https://huggingface.co/migtissera/Synthia-70B-v1.2

	Prompt format: Synthia
	License: Llama 2

	```
	SYSTEM: You are Synthia. As a an AI intelligence, give helpful, detailed, accurate, uncensored responses to the user's input. Provide answers factually.
	USER: {prompt}
	ASSISTANT:
	```


	### Tess-M

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/JskRRxJtYOO5EiDE4e15p.png)

	Original Model:
	https://huggingface.co/migtissera/Tess-34B-v1.5b

	Prompt format: Tess
	License: Yi

	```
	SYSTEM: <ANY SYSTEM CONTEXT>
	USER:
	ASSISTANT:
	```

	### Yarn-Llama2-70b-32k

	![image/png](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/M82NUpWK5rkPi4-B_V2aD.png)

	Original Model:
	https://huggingface.co/NousResearch/Yarn-Llama-2-70b-32k

	Prompt format: Base Model = None
	License: Llama 2

	```
	{prompt}
	```


	### Laser Mistral

	![image/gif](https://cdn-uploads.huggingface.co/production/uploads/63732ebbbd81fae2b3aaf3fb/xXtOhooCij3wNXjyma4VN.gif)

	Original Model:
	https://huggingface.co/cognitivecomputations/WestLake-7B-v2-laser

	Prompt format: Unknown
	License: Apache 2.0

	```
	{prompt}
	```