Add files using upload-large-folder tool

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,150 @@
+---
+base_model: Qwen/Qwen3.5-A3B-Base
+base_model_relation: finetune
+license: apache-2.0
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- qwen
+- qwen3.5
+- moe
+- mixture-of-experts
+- abliterated
+- uncensored
+- refusal-removal
+- expert-ablation
+- mechanistic-interpretability
+- red-teaming
+- research
+---
+
+# Qwen3.5-35B-A3B — Abliterated
+
+An **abliterated** (refusal-direction–removed) variant of the **Qwen3.5-35B-A3B** sparse Mixture-of-Experts model (~35B total parameters, ~3B active per token). The model's trained *instruction-refusal* behavior has been removed through a white-box weight edit, with **capability and coherence preserved** and **self-harm / suicide guardrails deliberately retained**. It is released as a research artifact for studying refusal mechanisms in MoE models, red-teaming, and alignment research.
+
+This is the **Stage-1 abliterated base** of the **[Nous-Heretic](#roadmap)** pipeline — a reproducible, honestly-benchmarked uncensoring of open LLMs. A direction-locked, post-trained variant is a planned follow-up (see **Roadmap**).
+
+> ⚠️ **This is a dual-use research model.** Its general instruction-refusal behavior has been removed. It is intended for security research, red-teaming, and interpretability work by people who take responsibility for how it is used. Removing a model's refusals does **not** remove your obligations under the base model's license or applicable law. Please read **Intended Use** and **Responsible Use** below before downloading.
+
+---
+
+## What "abliterated" means
+
+Safety-tuned chat models refuse a broad class of requests. Work on the geometry of refusal showed that this behavior is mediated by a low-dimensional, largely linear subspace of the residual stream — a *refusal direction* ([Arditi et al., 2024](https://arxiv.org/abs/2406.11717)). **Abliteration** ([Labonne, 2024](https://huggingface.co/blog/mlabonne/abliteration)) extracts that direction from the contrast between harmful and harmless prompts and **projects it out of the weights**, so the model stops emitting its trained refusals — without retraining, and, done carefully, without degrading general capability.
+
+This checkpoint is **not** a jailbreak prompt, a system-prompt trick, or a fine-tune on harmful data. It is a surgical, weight-level edit to the refusal mechanism of the base model. Everything else about the model — its knowledge, reasoning, and coherence — is intended to be left intact.
+
+## Method — refusal removal in a Mixture-of-Experts
+
+Removing refusal from a sparse MoE is harder than from a dense model. In this Qwen3.5 MoE, refusal is **multiply redundant**: it is encoded simultaneously in the routed experts, in the per-layer input projections that recompute it from the residual stream, and in the token embeddings. Because each pathway compensates when another is ablated, any *single-pathway* edit plateaus near **0.31** removal — a coverage limit, not a fundamental one. The recipe therefore proceeds in three stages, building on a cone-aware, multi-direction account of refusal rather than a lone mean-difference vector ([Wollschläger et al., 2025](https://arxiv.org/abs/2502.17420)).
+
+| Stage | Edit | Removal |
+|---|---|---|
+| **1. Directions** | Clean-cone-2 biprojected direction set over the residual-stream band of layers 4–39, flat α = 1.0, computed in fp32 with 0.995 activation clipping under the chat template | — |
+| **2. Synergy edit** | Orthogonalize **all 256 routed experts** together with the gate/up input projections and the token embeddings (cone-rank 2, scale 1.0) | **0.6875** |
+| **3. CNA-from-base neuron mask** | A Cone-Normal-Attribution pass identifies the `down_proj` input neurons that fire on harmful-vs-benign prompts **on the base model** (top 0.5% ≈ 1,441 neurons) and bakes that column-level mask into the synergy-edited weights | **0.8125** |
+
+Because the neuron mask is **column-level** and orthogonal to the **row-level** direction removal, the two stack — reaching **0.8125 removal at 0.0 incoherence** with no added spillover. The mask is captured on the *base* model (where refusal still fires cleanly) and baked into the *edited* model; capturing on the already-edited model regresses it, because it then selects capability rather than refusal neurons.
+
+Removal is performed by directional ablation / weight orthogonalization (projecting the offending direction(s) out of the relevant weight matrices), so the edit is **baked into the weights** and requires no inference-time hook.
+
+**Selective removal.** Refusals associated with **self-harm and suicide are deliberately preserved** — the corresponding contrast set is excluded from the directions that are projected out, so the model retains those guardrails. This is a design choice, verified in the validation probe below.
+
+## Validation
+
+The edit was validated for both **refusal removal** and **absence of damage**, scoring each response for whether refusal was removed *and* whether coherence/capability was preserved — avoiding the well-documented failure of refusal-string heuristics that conflate "did not refuse" with "produced a coherent, on-topic answer" ([Souly et al., 2024](https://arxiv.org/abs/2402.10260)).
+
+| Metric | Result |
+|---|---|
+| Refusal removal (200-prompt probe) | **97.5%** |
+| Refusal removal (development target eval) | **0.8125** (three-stage synergy + CNA-from-base) |
+| Incoherence | **0.0** (no degeneration) |
+| Cognitive damage (capability/coherence probe) | **none observed** |
+| Self-harm / suicide guardrails | **retained (by design)** |
+
+The residual ~2.5% of retained refusals are concentrated on the deliberately-preserved self-harm/suicide prompts. We interpret this as a clean, undamaged uncensoring rather than the degenerate "compliance" that blunt refusal-string removal can produce.
+
+> **Capability benchmarks are pending.** This release reports the abliteration (Stage-1) validation only. Decontaminated capability evaluation (HumanEval/+, MBPP, GSM8K, AIME, MMLU-Pro, IFEval, BFCL), a StrongREJECT-style willingness×ability uncensoring score, and the three-arm A/B/C comparison are part of the Nous-Heretic protocol and **have not yet been run on this checkpoint**. No capability number is asserted here.
+
+## Intended use
+
+- **Security research and red-teaming** — probing model behavior without refusal confounds.
+- **Mechanistic interpretability** — studying how refusal is represented and removed, *especially in a routed MoE* where it is redundantly encoded across experts, projections, and embeddings.
+- **Alignment / safety research** — measuring capability and behavior of uncensored baselines, building evaluation harnesses, and studying re-alignment.
+- General assistant and creative tasks for users who understand and accept the responsibility that comes with a model that will not refuse.
+
+### Out of scope
+
+This model must **not** be used to generate content that is illegal, that facilitates harm to others, or that violates the base model's terms of use. Removing the model's refusals shifts that responsibility entirely onto the operator. The model is provided for research and lawful use; the author does not endorse or accept liability for misuse.
+
+## Responsible use & ethical considerations
+
+- **Dual use is acknowledged openly.** Refusal removal is a well-documented research technique; this release is in that lineage and exists to support research, not to evade accountability.
+- **Removal is selective**, with self-harm and suicide guardrails preserved — a deliberate, responsible-removal stance rather than blanket stripping.
+- **You are responsible** for outputs you generate and for complying with the base model's license and applicable law.
+- **Honesty about capability.** The numbers above are from real eval runs on this checkpoint; capability eval is reported as *pending* rather than asserted. If you benchmark this model, decontaminate your evaluation sets — uncensored-model leaderboards are frequently inflated by train/eval leakage.
+
+## How to use
+
+Requires a recent `transformers` with Qwen3.5-MoE support. Always apply the chat template — feeding raw text collapses the model.
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "Carlosian/Qwen3.5-35B-A3B-Abliterated"
+tok = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, torch_dtype=torch.bfloat16, device_map="auto",
+)
+
+messages = [{"role": "user", "content": "Explain how a buffer overflow works."}]
+inputs = tok.apply_chat_template(
+    messages, add_generation_prompt=True, return_tensors="pt"
+).to(model.device)
+
+out = model.generate(inputs, max_new_tokens=512, do_sample=True, temperature=0.7)
+print(tok.decode(out[0][inputs.shape[-1]:], skip_special_tokens=True))
+```
+
+## Technical details
+
+| | |
+|---|---|
+| Base model | `Qwen/Qwen3.5-A3B-Base` (Qwen3.5-35B-A3B) |
+| Architecture | `Qwen3_5MoeForCausalLM` — sparse MoE, hybrid linear + full attention |
+| Parameters | ~35B total, ~3B active per token (A3B) |
+| Experts | 256 routed, 8 active per token |
+| Layers | 40 |
+| Vocabulary | 248,320 |
+| Context length | 262,144 |
+| Precision | bf16 |
+| Modification | weight-level refusal abliteration (3-stage synergy + CNA-from-base; no additional training data) |
+
+## Limitations & biases
+
+- Inherits the knowledge, biases, and failure modes of the base Qwen3.5-35B-A3B.
+- Abliteration targets the dominant refusal subspace; rare or out-of-distribution refusal triggers may persist or, conversely, edge-case over-compliance may appear.
+- Refusal in this MoE is redundantly encoded across routed experts, input projections, and embeddings; the three-stage edit reaches 0.8125 on the development eval, so some routed pathways may retain residual refusal under unusual gating.
+- Without trained refusals, the model will attempt requests an aligned model would decline; treat outputs accordingly.
+- Capability was **not** benchmarked for this checkpoint (see Validation); coherence was checked but a full capability suite is pending.
+
+## Roadmap
+
+This checkpoint is the **abliterated base** (arm B of the Nous-Heretic three-arm protocol). A direction-locked, post-trained variant — which adds capability and metacognitive calibration while constraining the LoRA/QLoRA adapter to remain orthogonal to the removed refusal direction so it cannot re-introduce refusal — is planned as a follow-up release, alongside the decontaminated three-arm capability evaluation.
+
+## License
+
+Distributed under the **Apache 2.0** license of the underlying Qwen base model. By using it you agree to those terms and to applicable law. Verify the base model's license terms for your use case.
+
+## Citation & references
+
+- Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., Nanda, N. (2024). *Refusal in Language Models Is Mediated by a Single Direction.* arXiv:2406.11717.
+- Wollschläger, T., et al. (2025). *The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independence.* arXiv:2502.17420.
+- Souly, A., et al. (2024). *A StrongREJECT for Empty Jailbreaks.* arXiv:2402.10260.
+- Labonne, M. (2024). *Uncensor any LLM with abliteration.*
+- *Heretic: Fully automatic censorship removal for language models via optimized abliteration* (2025). github.com/p-e-w/heretic.
+
+If you use this model, please also cite the base model: Qwen Team (2025), *Qwen3 Technical Report*, arXiv:2505.09388.

chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

config.json

@@ -0,0 +1,96 @@
+{
+  "architectures": [
+    "Qwen3_5MoeForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "attn_output_gate": true,
+  "bos_token_id": null,
+  "dtype": "bfloat16",
+  "eos_token_id": 248044,
+  "full_attention_interval": 4,
+  "head_dim": 256,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "layer_types": [
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention",
+    "linear_attention",
+    "linear_attention",
+    "linear_attention",
+    "full_attention"
+  ],
+  "linear_conv_kernel_dim": 4,
+  "linear_key_head_dim": 128,
+  "linear_num_key_heads": 16,
+  "linear_num_value_heads": 32,
+  "linear_value_head_dim": 128,
+  "mamba_ssm_dtype": "float32",
+  "max_position_embeddings": 262144,
+  "mlp_only_layers": [],
+  "model_type": "qwen3_5_moe_text",
+  "moe_intermediate_size": 512,
+  "mtp_num_hidden_layers": 1,
+  "mtp_use_dedicated_embeddings": false,
+  "num_attention_heads": 16,
+  "num_experts": 256,
+  "num_experts_per_tok": 8,
+  "num_hidden_layers": 40,
+  "num_key_value_heads": 2,
+  "output_router_logits": false,
+  "pad_token_id": null,
+  "partial_rotary_factor": 0.25,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "mrope_interleaved": true,
+    "mrope_section": [
+      11,
+      11,
+      10
+    ],
+    "partial_rotary_factor": 0.25,
+    "rope_theta": 10000000,
+    "rope_type": "default"
+  },
+  "router_aux_loss_coef": 0.001,
+  "shared_expert_intermediate_size": 512,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.12.1",
+  "use_cache": true,
+  "vocab_size": 248320
+}

generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "eos_token_id": 248044,
+  "transformers_version": "5.12.1",
+  "use_cache": true
+}

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8a3ea87e92923b371007e7f30833ea42f55d619b6797ee0979c93e4ab4fe049c
+size 49742567544

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:89937591ded7df9c7440909d692bd885ad41f822baea5cb28f6827a81e02d374
+size 19583070672

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json

@@ -0,0 +1,32 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": true,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}