Instructions to use anthracite-org/magnum-v2-123b-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use anthracite-org/magnum-v2-123b-gguf with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="anthracite-org/magnum-v2-123b-gguf",
	filename="magnum-v2-123b-iq1_s.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use anthracite-org/magnum-v2-123b-gguf with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf anthracite-org/magnum-v2-123b-gguf:IQ1_S
# Run inference directly in the terminal:
llama cli -hf anthracite-org/magnum-v2-123b-gguf:IQ1_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf anthracite-org/magnum-v2-123b-gguf:IQ1_S
# Run inference directly in the terminal:
llama cli -hf anthracite-org/magnum-v2-123b-gguf:IQ1_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf anthracite-org/magnum-v2-123b-gguf:IQ1_S
# Run inference directly in the terminal:
./llama-cli -hf anthracite-org/magnum-v2-123b-gguf:IQ1_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf anthracite-org/magnum-v2-123b-gguf:IQ1_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf anthracite-org/magnum-v2-123b-gguf:IQ1_S

Use Docker

docker model run hf.co/anthracite-org/magnum-v2-123b-gguf:IQ1_S

LM Studio
Jan

vLLM

How to use anthracite-org/magnum-v2-123b-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "anthracite-org/magnum-v2-123b-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anthracite-org/magnum-v2-123b-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/anthracite-org/magnum-v2-123b-gguf:IQ1_S

Ollama
How to use anthracite-org/magnum-v2-123b-gguf with Ollama:
```
ollama run hf.co/anthracite-org/magnum-v2-123b-gguf:IQ1_S
```

Unsloth Studio

How to use anthracite-org/magnum-v2-123b-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for anthracite-org/magnum-v2-123b-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for anthracite-org/magnum-v2-123b-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for anthracite-org/magnum-v2-123b-gguf to start chatting

Atomic Chat new
Docker Model Runner
How to use anthracite-org/magnum-v2-123b-gguf with Docker Model Runner:
```
docker model run hf.co/anthracite-org/magnum-v2-123b-gguf:IQ1_S
```

Lemonade

How to use anthracite-org/magnum-v2-123b-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull anthracite-org/magnum-v2-123b-gguf:IQ1_S

Run and chat with the model

lemonade run user.magnum-v2-123b-gguf-IQ1_S

List all available models

lemonade list

This repo contains GGUF quants of the model. If you need the original weights, please find them here.

This is the sixth in a series of models designed to replicate the prose quality of the Claude 3 models, specifically Sonnet and Opus. This model is fine-tuned on top of Mistral-Large-Instruct-2407.

Prompting

Model has been Instruct tuned with the Mistral formatting. A typical input would look like this:

<s>[INST] SYSTEM MESSAGE\nUSER MESSAGE[/INST] ASSISTANT MESSAGE</s>[INST] USER MESSAGE[/INST]

We also provide SillyTavern presets for Context and Instruct respectively.

The Mistral preset included in SillyTavern seems to be misconfigured by default, so we recommend using these as a replacement.

Credits

This model has been a team effort, and the credits goes to all members of Anthracite.

Training

The training was done for 1.5 epochs. We used 8x AMD Instinct™ MI300X Accelerators for the full-parameter fine-tuning of the model.

In addition to this, we noticed that Mistral Large models seemed much more sensitive to learning rate adjustments than other models:

We hypothesize this is primarily due to the particularly narrow and low variance weight distributions typical of Mistral derived models regardless of their scale.

In the end, due to the costs that would be involved in training another full 2 epochs run ($600) on an even lower rate, we settled on our third attempt: 2e-6 with an effective batch size of 64, stopped earlier than the target 2 epochs.

We notice a correlation between the significance of the 2nd epoch loss drop and the strength of the learning rate, implying 4e-6 leads to more catastrophic forgetting.