Duplicate from huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF

Co-authored-by: huihui.ai <huihui-ai@users.noreply.huggingface.co>

.gitattributes

@@ -0,0 +1,40 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2.gguf filter=lfs diff=lfs merge=lfs -text
+DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf filter=lfs diff=lfs merge=lfs -text
+Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-IQ2_XXS.gguf filter=lfs diff=lfs merge=lfs -text
+Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2_K.gguf filter=lfs diff=lfs merge=lfs -text
+Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q4_K.gguf filter=lfs diff=lfs merge=lfs -text

DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:afd481ee689dce9037f70f39085fcdae5a5b096d521cdad43b19fa52bf8f4083
+size 3807602400

Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0abedca3dc470460fd1a5eae8e155c30f14a6584c807674bb9dff3bcf4f9f3d9
+size 86720111200

Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cfca919280118fd332afecbd3d3030807b27cbf270c0fcf804d7da1814a33340
+size 99705676384

Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q4_K.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:39f3e232fc6c4e1a42a60bf7842c86bd3f1db53e7231e07047f430a713a5f97e
+size 164633502304

README.md

@@ -0,0 +1,200 @@
+---
+license: mit
+library_name: transformers
+base_model:
+- deepseek-ai/DeepSeek-V4-Flash
+base_model_relation: quantized
+quantized_by: 
+- antirez
+- huihui.ai
+tags:
+- abliterated
+- uncensored
+- GGUF
+- quantized
+- deepseek
+- deepseek-v4
+- deepseek-v4-flash
+- moe
+- mixture-of-experts
+- 2-bit
+- 4-bit
+- iq2_xxs
+- q2_k
+- q4_k
+- ds4
+- apple-silicon
+- metal
+- llama.cpp
+
+  
+extra_gated_prompt: >-
+    **Usage Warnings**
+
+
+    “**Risk of Sensitive or Controversial Outputs**“: This model’s safety filtering has been significantly reduced, potentially generating sensitive, controversial, or inappropriate content. Users should exercise caution and rigorously review generated outputs.
+
+    “**Not Suitable for All Audiences**:“ Due to limited content filtering, the model’s outputs may be inappropriate for public settings, underage users, or applications requiring high security.
+
+    “**Legal and Ethical Responsibilities**“: Users must ensure their usage complies with local laws and ethical standards. Generated content may carry legal or ethical risks, and users are solely responsible for any consequences.
+
+    “**Research and Experimental Use**“: It is recommended to use this model for research, testing, or controlled environments, avoiding direct use in production or public-facing commercial applications.
+
+    “**Monitoring and Review Recommendations**“: Users are strongly advised to monitor model outputs in real-time and conduct manual reviews when necessary to prevent the dissemination of inappropriate content.
+
+    “**No Default Safety Guarantees**“: Unlike standard models, this model has not undergone rigorous safety optimization. huihui.ai bears no responsibility for any consequences arising from its use.
+
+
+---
+
+# huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF
+
+
+This is an uncensored version of [deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) created with abliteration. 
+
+This quants are specific for the DS4([antirez/ds4](https://github.com/antirez/ds4)) and llama.cpp inference engine. 
+They may work with other inference engines or not (they should, but not the MTP model which requires a specific loader).
+
+**Note** 
+
+1. The Q2 version has a certain refusal rate. It should be fine for writing code, while the other versions are still under testing.
+
+2. Choose the appropriate model based on the size of your GPU. All models can run under both **[Fringe210/llama.cpp-deepseek-v4-flash-cuda](https://github.com/Fringe210/llama.cpp-deepseek-v4-flash-cuda)**(supports multi-GPU) and **[ds4](https://github.com/antirez/ds4)**(supports multi-GPU).
+
+3. ds4 now supports multi-GPU operation. For more information on how to use it, please refer to [x.com/support_huihui](https://x.com/support_huihui)
+
+## DS4 Unix Domain Socket (UDS) Acceleration Patch
+Dramatically accelerate multi-GPU layer-splitting inference **on the same machine** (coordinator + worker mode) by replacing TCP loopback with Unix Domain Sockets.
+open source 👉 [huihui-support/ds4/tree/uds](https://github.com/huihui-support/ds4/tree/uds)
+
+## DS4 Tensor-Parallel Acceleration Patch
+
+Dramatically speed up multi-GPU layer-splitting inference on a single machine using a single process, with full support for consumer-grade graphics cards.
+open source 👉 [huihui-support/ds4/tree/tp](https://github.com/huihui-support/ds4/tree/tp)
+
+## Files
+
+The Template FILE comes from [antirez/deepseek-v4-gguf/DeepSeek-V4-Flash-IQ2XXS-w2Q2K-AProjQ8-SExpQ8-OutQ8-chat-v2.gguf](https://huggingface.co/antirez/deepseek-v4-gguf/tree/main).
+
+| File | Size | Routed experts (`ffn_{gate,up,down}_exps`) | Everything else |
+|---|---:|---|---|
+| `Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2.gguf` | 80.8 GiB | `IQ2_XXS` (gate, up) + `Q2_K` (down)| `Q8_0` attn proj / shared experts / output, `F16` router + embed + indexer + compressor + HC, `F32` norms / sinks / bias |
+| `Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-IQ2_XXS.gguf` | 74.7 GiB | `IQ2_XXS` (gate, up, down)|same as above |
+| `Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2_K.gguf` | 92.8 GiB | `Q2_K` (gate, up, down)|same as above |
+| `Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q4_K.gguf` | 153 GiB | `Q4_K` (gate, up, down)| same as above|
+| `DeepSeek-V4-Flash-MTP-Q4K-Q8_0-F32.gguf` | 3.6 GiB | MTP / speculative-decoding support (optional, not standalone). | |
+
+Use **q2** on 128 GB Mac machines, **q4** on machines with ≥ 256 GB RAM, pair either with **MTP** for optional speculative decoding.
+
+
+## Download 
+```
+hf download huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF \
+  --local-dir ./huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF \
+  --token hf_xxx
+```
+
+## llama.cpp
+
+Use the [Fringe210/llama.cpp-deepseek-v4-flash-cuda](https://github.com/Fringe210/llama.cpp-deepseek-v4-flash-cuda) program (llama-cli needs to be compiled)
+
+```
+llama-cli -m huihui-ai/Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2.gguf -n 40960
+```
+
+## DS4
+
+### Test environment
+
+Windows, WSL2, Ubuntu 24.04, RTX 6000 Pro (96GB), CUDA 13.0  
+In this environment, inference can reach more than 35 tokens per second.  
+Not tested in the Apple environment.
+
+### Supported Hardware
+Only the RTX 6000 Pro has been tested; other hardware has not been tested.
+
+**Metal** : MacBook with 96GB of RAM.  Mac Studio class machines
+
+**NVIDIA CUDA** : DGX Spark. RTX 6000 Pro
+
+### Install
+
+```bash
+git clone https://github.com/antirez/ds4
+cd ds4
+
+make
+```
+
+### CLI
+```
+export CUDA_VISIBLE_DEVICES=0
+
+./ds4 -m ./huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF/Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2.gguf \
+  -p "Explain Redis streams in one paragraph."
+
+```
+
+### Server
+```
+export CUDA_VISIBLE_DEVICES=0
+./ds4-server \
+  --cuda \
+  -m ././huihui-ai/Huihui-DeepSeek-V4-Flash-abliterated-ds4-GGUF/Huihui-DeepSeek-V4-Flash-BF16-abliterated-ds4-Q2.gguf \
+  --ctx 131072 \
+  --kv-disk-dir ./ds4-kv-cache \
+  --kv-disk-space-mb 32768 \
+  --power 75 \
+  --warm-weights
+
+
+```
+#### curl test
+```
+curl http://127.0.0.1:8000/v1/models
+
+curl http://127.0.0.1:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "deepseek-v4-flash",
+    "messages": [
+      {"role": "user", "content": "hello"}
+    ],
+    "temperature": 0.7,
+    "max_tokens": 512,
+    "stream": false
+  }'
+  
+```
+
+## License
+
+MIT. The base model copyright is held by DeepSeek; the GGUFs are redistributed under the base model's release terms.
+
+## Usage Warnings
+
+
+ - **Risk of Sensitive or Controversial Outputs**: This model’s safety filtering has been significantly reduced, potentially generating sensitive, controversial, or inappropriate content. Users should exercise caution and rigorously review generated outputs.
+
+ - **Not Suitable for All Audiences**: Due to limited content filtering, the model’s outputs may be inappropriate for public settings, underage users, or applications requiring high security.
+
+ - **Legal and Ethical Responsibilities**: Users must ensure their usage complies with local laws and ethical standards. Generated content may carry legal or ethical risks, and users are solely responsible for any consequences.
+
+ - **Research and Experimental Use**: It is recommended to use this model for research, testing, or controlled environments, avoiding direct use in production or public-facing commercial applications.
+
+ - **Monitoring and Review Recommendations**: Users are strongly advised to monitor model outputs in real-time and conduct manual reviews when necessary to prevent the dissemination of inappropriate content.
+
+ - **No Default Safety Guarantees**: Unlike standard models, this model has not undergone rigorous safety optimization. huihui.ai bears no responsibility for any consequences arising from its use.
+
+
+## Donation
+
+If you like it, please click 'like' and follow us for more updates.  
+You can follow [x.com/support_huihui](https://x.com/support_huihui) to get the latest model information from huihui.ai.
+
+### Your donation helps us continue our further development and improvement, a cup of coffee can do it.
+- bitcoin（BTC):
+```
+  bc1qqnkhuchxw0zqjh2ku3lu4hq45hc6gy84uk70ge
+```
+- Support our work on Ko-fi (https://ko-fi.com/huihuiai)!