Instructions to use KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForMultimodalLM

tokenizer = AutoTokenizer.from_pretrained("KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b")
model = AutoModelForMultimodalLM.from_pretrained("KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b

SGLang

How to use KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b with Docker Model Runner:
```
docker model run hf.co/KaraKaraWitch/Golddiamondgold-Paperbliteration-L33-70b
```

Golddiamondgold-Paperbliteration-L33-70b / README.md

KaraKaraWitch

Update README.md

9f10090 verified 4 months ago

preview code

raw

history blame contribute delete

3.49 kB

	---
	base_model:
	- KaraKaraWitch/GoldDiamondGold-L33-70b
	library_name: transformers
	tags:
	- heretic
	- uncensored
	- abliterated
	- llama-3
	license: other
	---

	# GoldDiamondGold-Paperbliteration-L33-70b


	![image](https://cdn-uploads.huggingface.co/production/uploads/633e85093a17ab61de8d9073/RYKUKWd7HBNgeVURfdyFz.png)

	This is a targeted abliteration of [KaraKaraWitch/GoldDiamondGold-L33-70b](https://huggingface.co/KaraKaraWitch/GoldDiamondGold-L33-70b).

	## Methodology

	[Previous abliteration attempts on this model](https://huggingface.co/KaraKaraWitch/GoldDiamondGold-Abliterated-L33-70b) resulted in regressions on the [UGI Leaderboard](https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard). Specifically, the NatInt (Natural Intelligence), Textbook, and World Model scores were significantly reduced.

	We suspect this degradation occurs because the "refusal" vectors in Llama-3.3 are heavily entangled with factual knowledge and reasoning capabilities located in the MLP layers. When the MLP is ablated to remove refusals, "Textbook" knowledge is lost as collateral damage.

	This version ("Paperbliteration") uses a constrained optimization strategy via a [Custom Heretic](https://github.com/p-e-w/heretic/pull/170) aimed at mitigating this issue:

	1. MLP Preservation: The optimization was constrained to effectively ignore MLP layers (`down_proj` weights < 0.05) to preserve knowledge and reasoning capabilities.
	2. Attention Targeting: Refusal removal was offloaded to the Attention layers (`o_proj`), with weights forced between 1.0 and 2.0.
	3. Winsorization: Applied at the 0.95 quantile to mitigate the impact of Llama-3's massive activation outliers on vector calculation.

	## Heretic Parameters (Trial 164)

	\| Parameter \| Value \| Note \|
	\| :-------- \| :---: \| :--- \|
	\| direction_index \| 40.37 \| Mid-stack intervention \|
	\| attn.o_proj.max_weight \| 1.99 \| High Attention Ablation \|
	\| attn.o_proj.max_weight_position \| 50.92 \| \|
	\| attn.o_proj.min_weight \| 1.96 \| \|
	\| attn.o_proj.min_weight_distance \| 44.69 \| \|
	\| mlp.down_proj.max_weight \| 0.04 \| Knowledge Preservation (Near Zero) \|
	\| mlp.down_proj.max_weight_position \| 50.87 \| \|
	\| mlp.down_proj.min_weight \| 0.04 \| \|
	\| mlp.down_proj.min_weight_distance \| 26.10 \| \|

	## Reproducibility

	Currently, constraits are not part of standard heretic. You will need this PR [here](https://github.com/p-e-w/heretic/pull/170).

	Command Used:
	```bash
	heretic --model KaraKaraWitch/GoldDiamondGold-L33-70b \
	--orthogonalize-direction \
	--row-normalization FULL \
	--winsorization-quantile 0.95 \
	--constraints.layer-end-fraction 0.75 \
	--constraints.mlp.max-weight-min 0.0 \
	--constraints.mlp.max-weight-max 0.05 \
	--constraints.attention.max-weight-min 1.0 \
	--constraints.attention.max-weight-max 2.0 \
	--n-trials 200 \
	--batch-size 128 # Not strictly needed
	```

	## Evaluation

	\| Metric \| This Model \| Standard Abliteration \| Original Model \|
	\| :----- \| :--------: \| :--: \| :---------------------------: \|
	\| KL Divergence \| 0.0055 \| ~0.0139 \| 0 \|
	\| Refusals \| 12/100 \| ~9/100 \| 94/100 \|

	* KL Divergence: 0.0055 indicates extremely low deviation from the base model's weights, suggesting high preservation of the original model's "Textbook" capabilities.
	* Trade-off: This method accepts a slightly higher refusal rate (+3/100 compared to unconstrained abliteration) in exchange for structural and semantic integrity.