--- base_model: - qwen/Qwen3.6-27B tags: - text-generation-inference - image - transformers - unsloth - qwen3_6 - reasoning - chain-of-thought - lora - sft - multimodal - vision - tool-use - function-calling - long-context - agent - science license: apache-2.0 language: - en - zh - es - ru - ja pipeline_tag: image-text-to-text datasets: - Jackrong/Claude-opus-4.6-TraceInversion-9000x - Jackrong/Claude-opus-4.7-TraceInversion-5000x ---

🪐 Qwopus3.6-27B-v2

SFT Release

Reasoning-Enhanced Dense Language Model Fine-Tuned on Qwen3.6-27B

🧬 Trace Inversion & Negentropy 🧠 27B Parameters 🔥 3-Stage Curriculum SFT 🛠️ Vision & Tool-use Support

💡 What is Qwopus3.6-27B-v2?

🪐 Qwopus3.6-27B-v2 is a reasoning-enhanced dense language model built on top of Qwen3.6-27B. By leveraging a multi-stage curriculum learning pipeline and augmented with Trace Inversion datasets (claude-opus-4.6/4.7-traceInversion), it reverse-engineers the compressed "Reasoning Bubbles" of commercial LLMs into structured, step-by-step synthetic reasoning traces, successfully eliminating logical shortcuts and knowledge fractures.

🧩 Structured Reasoning Injects reconstructed deep CoT chains to eliminate logical shortcuts via Trace Inversion.

🪶 Style Consistency Enforces strict constraints on the format and convergence of <think> tags.

🔁 Distillation Alignment Ensures high-quality cross-source SFT data alignment to narrow the capacity gap.

⚡ RL Scalability Sets up a stable formatting pipeline optimized for downstream Reinforcement Learning (RL).

## 💡 1. Base Model, Training Library & Cooperation

🧠 1.1 Base Model Specifications (Qwen3.6-27B)

Qwen3.6-27B is a state-of-the-art dense large language model developed by Alibaba Cloud. Boasting 27 billion parameters, this base model natively supports long-context modeling and is engineered for agentic workflows, complex logical reasoning, and multimodal capabilities.

Attribute	Specifications & Details
🧠 Architecture	Dense Transformer / 27 Billion Parameters
🏢 Developer	Alibaba Cloud (DAMO Academy)
📄 Context Window	Native Support Up to 32K / 128K context length
🎯 Focus Domains	Agentic Coding, Deep Logic Reasoning, Multimodal Tasks (Vision & Tool-use)
🧬 Distillation Strategy	Cross-source SFT alignment and multi-teacher distillation to close the capability gap.
⚡ RL Scalability	Optimized for downstream Reinforcement Learning alignment and self-critical learning loops.

🧪 1.2 Hardware Cooperation & Joint Collaboration

This project is built in close collaboration and joint effort with engineer Kyle Hessling to achieve stable fine-tuning of our 27B parameter model.

👉 You can follow him for hardware and model training updates on X / Twitter: @KyleHessling1

🦥 1.3 Fine-tuning Framework (Unsloth)

Our model training is accelerated by the cool library from Unsloth. Special thanks to the Unsloth team for their highly efficient, memory-optimized training framework.

👉 You can visit their documentation for more fine-tuning guidance: unsloth.ai/docs

⚙️ 1.4 Custom MTP Heads Processing & Automation Tooling

This release features a custom splitting and merging methodology designed specifically for Qwen series Multi-Token Prediction (MTP) heads. The automation skill and complete processing pipeline scripts are open-sourced in qwen-mtp-gguf.

🌟If you find this toolkit helpful, please support the project by leaving a star on GitHub!

> [!TIP] > **Vision & Tool Calling Support**: Qwopus3.6-27B-v2 natively supports vision and tool-use capabilities. To enable vision functionality, download `mmproj.gguf` from the [GGUF Repository](https://huggingface.co/Jackrong/Qwopus3.6-27B-v2-GGUF) and place it in the same directory as the main `.gguf` file. > [!WARNING] > **Community Release Notice**: Qwopus3.6-27B-v2 is an **experimental community release** and has not undergone complete safety evaluations or standard benchmarking. It is intended solely for research and exploration. --- ## 📖 2. Background & Motivation

⚠️ 2.1 The Distillation Dilemma of "Reasoning Bubbles"

In the field of reasoning distillation, commercial closed-source models (such as GPT-4o or Claude 3.5 Sonnet) typically hide their raw step-by-step thinking processes, displaying only highly compressed "Reasoning Bubbles" to users. Direct imitation of these summaries introduces a severe "Information Entropy Trap"—student models struggle to mimic jumpy, step-skipped conclusions without undergoing the underlying logic derivations, resulting in reasoning fractures and poor generalizability.

🧬 2.2 Trace Inversion & Negentropy Reconstruction

To address this challenge, we introduce Trace Inversion datasets to reconstruct the full reasoning pathway. By using a specialized logical reconstructor, Trace-Inverter-4B, we reverse-engineer the compressed reasoning bubbles into a complete, step-by-step Learnable Chain-of-Thought (CoT).

This model integrates:

claude-opus-4.6-traceInversion-9000x: 9,000 high-value, fully reconstructed step-by-step reasoning trajectories.
claude-opus-4.7-traceInversion-5000x: 5,000 complex multi-turn logic and mathematics samples optimized for negative entropy reconstruction.

This ensures the student model learns continuous, rigorous logical derivations rather than shortcut-ridden summaries.

--- ## ⚡ 3. Reasoning Efficiency & MTP Speedup

⚡ Reasoning Efficiency & MTP Acceleration

A compact view of how many output tokens are needed to produce correct answers, and how the MTP variant improves inference throughput.

🚀 MTP Speedup 1.66x Qwopus3.6-27B-v2-MTP reaches 1.66x the official Qwen3.6 speed

🧠 Correct-Answer Tokens 918.7 Average tokens when Qwopus answers correctly

📉 Total Token Reduction 15.0% 627,325 vs 738,238 output tokens

✅ Token Conversion 4.64 Correct answers per 10,000 output tokens

Efficiency framing: These measurements are based on parsed outputs from the same 350-question MMLU-style evaluation set. The goal is not only to compare accuracy, but also to measure how much reasoning text each model spends to reach correct answers.

🎯 3.1 Correct-Answer Token Cost

Metric Definition	Qwen3.6-27B	Qwopus3.6-27B-v2	Efficiency Gain
Definition A: average output tokens on correctly answered questions only.	1,433.3 tokens	918.7 tokens	35.9% fewer tokens
Definition B: total output tokens divided by the number of correct answers, including token cost from wrong answers.	2,511.0 tokens	2,155.8 tokens	14.2% less systemic overhead

📈 3.2 Token Conversion Efficiency

Metric	Qwen3.6-27B	Qwopus3.6-27B-v2	Delta
Correct answers per 10,000 output tokens	3.98	4.64	+16.6%
Total output token cost	738,238 tokens	627,325 tokens	15.0% fewer tokens

🧩 3.3 Chain-of-Thought Length Comparison

CoT Extraction Mode	Qwen3.6-27B	Qwopus3.6-27B-v2	Reduction
Normal thinking endings: text before the closing `</think>` tag only.	1,680.3 tokens 5,169.4 chars	798.5 tokens 2,370.0 chars	52.5% shorter

--- ## 📊 4. Evaluation & Benchmarks

📊 Evaluation & Performance Metrics

Detailed benchmark results on MMLU-Pro, SWE-bench, frontend page layout generation, creative coding, and agentic reasoning.

📚 MMLU-Pro Subset 87.43% +2.57 pp vs Qwen3.6-27B

🏆 SWE-bench Verified 75.25% 152 / 202 Resolved (Win)

🎨 Web Design 100% 5 / 5 Pages Validated

⚡ Speed (RTX 5090) 43.9 Avg tok/s (Q5_K_M)

📚 4.1 MMLU-Pro Selected Subset

Evaluation format: All models in this MMLU-Pro comparison were tested as unquantized, full-format 16-bit checkpoints. The selected evaluation subset covers 7 categories, with 50 questions per category and 350 questions in total.

Model	Correct / Total	Accuracy
Qwopus3.6-27B-v2	306 / 350	87.43%
Qwen3.6-27B	297 / 350	84.86%

Category	Qwen3.6-27B	Qwopus3.6-27B-v2	Delta
Biology	96%	96%	0 pp
Business	88%	94%	+6 pp
Computer Science	82%	84%	+2 pp
Mathematics	90%	88%	-2 pp
Physics	76%	86%	+10 pp
Chemistry	74%	80%	+6 pp
Health	88%	84%	-4 pp

Summary: On the selected 350-question MMLU-Pro evaluation set, Qwopus3.6-27B-v2 achieved 87.43% accuracy, outperforming Qwen3.6-27B at 84.86%. Qwopus3.6-27B-v2 is stronger in Business, Computer Science, Physics, and Chemistry, while Qwen3.6-27B remains ahead in Mathematics and Health.

Scope note: due to limited resources, only randomly sampled questions from MMLU-Pro were evaluated. To keep the evaluation fair and transparent, model response logs will be placed in the repository test_data folder.

💻 4.2 SWE-bench Verified (controlled-202 slice)

Model / Configuration	Sampling	Resolved	Empty Patches	Resolve %
Qwopus 3.6 27B v2 (dense)	temp 1.0, step 275, single-slot	152 / 202	1	75.25%

Execution Details: 19h 29m wall-clock on a single RTX 5090 using a 160K fp16 context window. Every instance successfully exited with Submitted status. 0 step-limit hits and 0 context-overflow failures occurred. Median trajectory length was 67 / 275 steps.

⚡

Run agentic harnesses hot: Raising sampling temperature to temp=1.0 with thinking-on effectively eliminates the reasoning-loop failure mode where earlier finetunes hit 78 empty patches. Greedy decoding (temp=0.1) forces the finetune to over-deliberate and loop inside the <think> block, whereas a higher temperature enables the model to utilize the full breadth of reasoning paths established during training.

⚡ 4.3 Throughput & VRAM Allocation (RTX 5090)

Metric	Qwopus 3.6 35B-A3B (MoE, Q5)	Qwopus 3.6 27B V2 (Dense, Q5)
Average Throughput	161.9 tok/s	43.9 tok/s
Throughput Range	154.4 / 164.8 tok/s	43.1 / 44.6 tok/s
VRAM Usage	~25 GB (65K q8 context)	~31 GB (160K fp16 context)
Completion Tokens (Suite)	106,688 tokens	119,036 tokens
Total Runtime (Suite)	11.1 min	45.3 min

Architecture Trade-off: The MoE wins on raw throughput by ~3.7x due to its A3B routing pattern. However, the Dense 27B model offsets this with superior per-token reasoning depth. We recommend the Dense 27B model for complex agentic workflows, long-context reasoning, and code execution, and the MoE model for fast, high-throughput generations. Tight throughput variance (±0.75 tok/s) indicates the dense model is fully memory-bandwidth-bound.

🎨 4.4 Web Design, WebGL Canvas, and Agentic Task Breakdown

🎨 Web Design Layout Generation (All 5 validated end-to-end)

Prompt / Brief	Size (KB)	Tokens	Time	Reasoning Tokens
SaaS Landing Page (AI Observability)	60.3 KB	23,801	552 s	836
Analytics Dashboard (Light Theme)	42.1 KB	15,390	354 s	1,898
Designer Portfolio (Kinetic Typography)	32.5 KB	11,612	265 s	1,459
Pricing Page (3 Tiers + FAQ)	26.6 KB	9,360	213 s	1,077
Mobile App Marketing Page	42.3 KB	16,590	382 s	1,650

🌌 Canvas / WebGL Creative Coding (Selected Publications)

Sketch Name	Size (KB)	Tokens	Time	Configuration & Metrics
Particle Attractor (Fluid Swarm)	9.4 KB	4,308	97 s	temp 1.0 · 1,513 chars reasoning
Generative Flowfield (Ink Agents)	13.9 KB	7,237	163 s	temp 1.0 · 6,269 chars reasoning
Soft-Body Physics Sandbox	18.0 KB	6,827	154 s	temp 0.75 · 1,665 chars reasoning (shipped clean first run)
Audio-Reactive Visualizer	10.7 KB	5,731	129 s	temp 1.0 · 7,645 chars reasoning

*Note: Mandelbulb and Three.js crystal scene outputs are excluded due to aesthetic standards and parked in excluded-canvas/.

🧠 Agentic Tasks (5 prompts + 1 structured extraction nothink rerun)

Task Brief	Completion Tokens	Reasoning Characters	Time
Multi-step Planning (URL shortener deploy)	2,238	7,067	50 s
Tool-use Planning (Flights, Hotel, Weather)	1,262	2,807	28 s
Code Debugging (4-bug BST K-th smallest)	1,753	5,225	39 s
Structured Extraction (Roster from prose)	1,721	4,245	39 s
Self-Critique Loop (Palindrome optimization)	1,255	3,309	28 s
Structured Extraction (No-think)	351	0 (nothink)	8 s

code_debug: Successfully caught all 4 bugs (sort order, = vs ==, useless loop, off-by-one errors).
self_critique: Followed the structured instruction loop (INITIAL → CRITIQUE → IMPROVED) and optimized a palindrome algorithm to O(n²) expand-around-center.
multi_step_planning: Designed a robust 10-step deployment plan with Dockerfile hand-off and explicit pip dependencies.
tool_use_json: Resolved a 3-tool sequence (search_flights, book_hotel, get_weather) with completely valid argument shapes.

--- ## 🗺️ 5. Training & Data Pipeline Overview The training process fuses **Trace Inversion** data augmentation with a **Three-Stage Curriculum Learning** pipeline. The core engineering focuses on expanding context length gradually while training on reconstructed reasoning traces to guarantee format stability. ```text [ 🗺️ Trace Inversion: Reconstructing Distillation Workflow ] A. Surrogate Model Training (Trace Inverter) Open-source Model (GLM-5.1 / DS-V4) ──► Complete Reasoning Chain ──► [ Qwen3-235B Compression ] ──► Reasoning Bubbles │ │ └──────────► [ Training ] ◄─────────┘ (Base: Qwen3-4B-Instruct) (Result: Trace-Inverter-4B) B. Inversion Phase: Reconstructing Claude-4.7-Max _______________________________________________________ | | | Claude-4.7-Max API ──► Compressed Bubbles + Answer | |_______________________________________________________| │ ▼ [ 🧠 Trace-Inverter-4B (Logic Reconstructor) ] ──► Synthetic Deep Reasoning Trace (Learnable CoT) │ ▼ [ 🧩 Data Splicing ] ◄────────── (Original Prompt + Response) (Embed reconstructed CoT in tags, splicing with original prompt/response) │ ▼ (Result: claude-opus-4.6/4.7 inverted sets) C. Final SFT Curriculum Pipeline ___________________________________________ | | | Base Model (Qwen3.6-27B) | |___________________________________________| │ ▼ [ 📦 Phase 1: Format Inception ] ──► [ 🛠️ Phase 2: Complexity Expansion ] ──► [ 🚀 Phase 3: Long-Context SFT ] ( < 4096 tokens ) ( 4096 - 8192 tokens ) ( 8192 - 32K tokens ) (Short-context stable format) (Medium-complexity reasoning) (Long/Multi-turn / 10% replay) │ │ └─────────────────────────────┬─────────────────────────────────────────┘ ▼ _____________________________________________ | | | 🌟 Final Model: Qwopus3.6-27B-v2 | |_____________________________________________| ``` --- ## 🎯 6. Three-Stage Curriculum Learning To steadily scale up the reasoning quality under long-context inference, **Qwopus3.6-27B-v2** adopts a Curriculum Learning strategy, progressively mixing longer and more complex reasoning templates:

Curriculum Stage	Focus & Sample Characteristics	Strategy Details
📦 Stage 1: Format Inception	• Limit context within 4,096 tokens • Emphasize stable reasoning templates	Focuses on short-to-medium length, cleanly formatted reasoning samples. The primary goal is to establish a reliable, structured reasoning output format (such as auto-closing `<think>` tags), preventing premature exposure to complex chains from causing format collapse.
🛠️ Stage 2: Complexity Expansion	• Extend length to 4,096 - 8,192 tokens • Introduce high-difficulty logic samples	Gradually increases the ratio of complex reasoning chains. By aligned distillation with "teacher models" whose reasoning style distributions closely match the Qwen3.6 base, the capacity gap is controlled to achieve highly efficient knowledge transfer.
🚀 Stage 3: Long-Context SFT	• Progressively scale window up to 32K tokens • 10% high-quality short sample replay	In this stage, the model is pushed to deep reasoning scenarios under ultra-long context and multi-turn dialogues. To prevent capacity drift or degradation of short-instruction comprehension during long-text training, a 10% replay of high-quality short samples is strictly enforced.

--- ## 🎨 7. Trace Inversion Case Studies (5 Key Domains Showcase) To demonstrate how **Trace Inversion** reconstructs logical continuity and eliminates negative entropy, the following interactive panels show the contrast between raw compressed "Reasoning Bubbles" and the fully step-by-step reconstructed chain-of-thought (Learnable CoT) under 5 typical scenarios: ### 📐 Domain 1: Mathematics (Probability Calculation)

📐 Case 1: Probability Calculation

Math Case

💬 Query

A bag contains 4 red balls and 6 blue balls. If two balls are drawn at random without replacement, what is the probability that both are red?

📋 Compressed Reasoning Bubble

First draw red: 4/10. Second draw red: 3/9. Multiply: (4/10) * (3/9) = 12/90 = 2/15.

✅ Final Answer

The probability that both balls drawn are red is 2/15.

2/15

💡 Synthesized Reasoning Trace (Learnable CoT)

STEP 1
Define

Identify total items and counts: Total balls = 4 (red) + 6 (blue) = 10 balls. The goal is to compute the probability of drawing two red balls in succession without replacement: P(R1 ∩ R2).

STEP 2
First Draw

Calculate probability of the first event (drawing a red ball first): P(R1) = Number of red balls / Total balls = 4/10. This simplifies to 2/5.

STEP 3
Second Draw

Calculate probability of the second event given that the first was red: Since there is no replacement, there are now 9 balls left in the bag, and 3 of them are red. P(R2 | R1) = 3/9 = 1/3.

STEP 4
Multiply

Apply the multiplication rule for dependent events: P(R1 ∩ R2) = P(R1) × P(R2 | R1). Substitute values: P(R1 ∩ R2) = (4/10) × (3/9) = 12/90.

STEP 5
Simplify

Simplify the resulting fraction: 12/90 can be divided by 6 in both numerator and denominator: 12÷6 = 2, 90÷6 = 15. The final probability is 2/15.

🏆

Conclusion: The probability of drawing both red balls is 2/15 (~13.33%).

### 🚀 Domain 2: Physics (Kinematics)

🚀 Case 2: Kinematics Formula

Physics Case

💬 Query

A car starts from rest and accelerates uniformly at 3 m/s² for 6 seconds. How far does the car travel during this time?

📋 Compressed Reasoning Bubble

Use kinematics equation d = v0*t + 0.5*a*t^2. Initial velocity v0 = 0, a = 3, t = 6. Calculate 0.5 * 3 * 36. Output 54 meters.

✅ Final Answer

The car travels 54 meters.

Answer: 54 meters

💡 Synthesized Reasoning Trace (Learnable CoT)

STEP 1
Goal

Identify the given variables and goal: Calculate the displacement (d) traveled under uniform acceleration from rest.

STEP 2
Extract

Identify given values: Initial velocity (v₀) = 0 m/s (starts from rest), acceleration (a) = 3 m/s², and duration (t) = 6 seconds.

STEP 3
Match Formula

Select the kinematic formula relating displacement to acceleration and time: d = v₀t + ½at².

STEP 4
Compute

Substitute the values: d = (0)×(6) + ½×3×(6)². Simplify: first term is 0; 6² = 36; 3 × 36 = 108; ½ × 108 = 54 meters.

STEP 5
Verify

Verify using average velocity method: final velocity v_final = v₀ + at = 0 + 3×6 = 18 m/s. Average velocity v_avg = (0 + 18)/2 = 9 m/s. Distance d = v_avg × t = 9 × 6 = 54 meters. The calculation is robust.

🏆

Conclusion: The car travels 54 meters during this uniform acceleration period.

### 💻 Domain 3: Coding (Algorithm Logic)

💻 Case 3: Algorithm Logic

Coding Case

💬 Query

Given an array of integers, find the sum of all even numbers.

📋 Compressed Reasoning Bubble

Filter evens using x % 2 == 0, then sum them up. Iterate through array, checking modulus.

✅ Final Answer

Here is the Python implementation:

def sum_even_numbers(arr):
    return sum(x for x in arr if x % 2 == 0)

💡 Synthesized Reasoning Trace (Learnable CoT)

STEP 1
Iterate

Initialize a tracking sum variable to 0. Prepare to iterate through the array from index 0 to N-1.

STEP 2
Check Even

For each element x, apply the modulus operator: check if x % 2 == 0. If True, the number is even.

STEP 3
Accumulate

If the condition is met, add x to the tracking sum. Otherwise, proceed to the next iteration.

STEP 4
Edge Cases

Check for empty arrays or arrays with no even numbers. In both cases, the returned sum is 0. Negative numbers are handled correctly (e.g. -2 % 2 == 0 is True).

STEP 5
Complexity

Verify complexity: The algorithm runs in O(N) time as it inspects each element exactly once, and O(1) auxiliary space.

🏆

Conclusion: Summing even numbers completed with O(N) time complexity.

### 🧠 Domain 4: Logical Reasoning (Syllogism)

🧠 Case 4: Syllogism & Set Inclusion

Logic Case

💬 Query

If all philosophers are thinkers, and some thinkers are writers, is it necessarily true that some philosophers are writers?

📋 Compressed Reasoning Bubble

Class inclusion. 'Philosophers' is a subset of 'Thinkers'. 'Writers' overlaps with 'Thinkers', but the overlap could lie entirely outside the 'Philosophers' subset. Thus, it does not necessarily follow.

✅ Final Answer

No, it does not necessarily follow.

Answer: No, it does not follow

💡 Synthesized Reasoning Trace (Learnable CoT)

STEP 1
Set Definition

Define the sets: P = Philosophers, T = Thinkers, W = Writers. Transcribe premises: Premise 1 is P ⊆ T. Premise 2 is T ∩ W ≠ ∅.

STEP 2
Objective

Determine if the conclusion P ∩ W ≠ ∅ is a logical necessity under all models.

STEP 3
Venn Analysis

Visualize via Venn diagrams. Circle P is completely inside Circle T. Circle W intersects Circle T. W can intersect only the region of T that excludes P. In this layout, P ∩ W = ∅.

STEP 4
Counterexample

Create a semantic counterexample: Let T = Humans, P = French people, W = Japanese people. All French people are Humans. Some Humans are Japanese. Does it follow that some French people are Japanese? No.

STEP 5
Conclusion

Since a valid counterexample exists, the argument forms an invalid syllogism (specifically, Fallacy of the Undistributed Middle). The conclusion is not guaranteed.

🏆

Conclusion: The syllogism is invalid; the conclusion does not necessarily follow.

### 💡 Domain 5: Core Theory (Reasoning Bubble vs. Learnable CoT)

💡 Case 5: Proprietary API "Reasoning Bubbles" vs. Reconstructed "Learnable CoT"

Core Theory

💬 Core Problem

What is the difference between proprietary API "Reasoning Bubbles" and reconstructed "Learnable Chain-of-Thought (CoT)"? Why is direct fine-tuning on Reasoning Bubbles problematic?

📋 Compressed Reasoning Bubble

Reasoning bubbles are condensed summaries of thinking. They hide step-by-step logic, making direct imitation hard for small models. Reconstructed CoT fills in missing steps, providing a smooth learning gradient for model distillation.

✅ Final Answer

Proprietary "Reasoning Bubbles" serve as compressed, high-level summaries that cause cognitive gaps in student models. "Learnable CoT" is a step-by-step, fully-derived logical chain that restores intermediate gradients, enabling effective reasoning distillation.

Bubble = What was thought; CoT = How it was thought

💡 Synthesized Reasoning Trace (Learnable CoT)

STEP 1
Define

Define the Concepts: Reasoning Bubbles are high-level summaries written for humans, whereas Learnable CoT consists of explicit, step-by-step derivations optimized for model learning.

STEP 2
Entropy

Information Loss (Entropy Trap): API summaries hide intermediate calculation steps. Direct imitation introduces severe "information entropy traps"—student models are forced to skip logical steps, causing logical confusion.

STEP 3
Gradient

Causal Gradients: Reconstructed CoT unfolds every derivation, calculation, and code design block. By providing continuous token-level supervision, it creates stable logical gradients for model weights to converge on.

STEP 4
Inversion

Trace Inversion Process: Using the problem and the final answer as hard endpoints, and the reasoning bubble as clues, the inversion model reverse-engineers a complete, rigorous reasoning path (Learnable CoT).

🏆

Conclusion: Reconstructed CoT turns dark black-box data into open, highly learnable logic gradients.

--- ## 🚀 Context Length and Long-Context Usage During fine-tuning, this model was trained with a maximum sequence length of **32K tokens**. The training data mixture was also constructed around samples up to **32K tokens**, so the "Context Length Distribution" shown in this model card reflects the fine-tuning data distribution rather than a hard architectural limit. The model still inherits the native long-context capability of the Qwen3.6 base model. Therefore, longer context windows such as **128K** or **256K** may be available in compatible inference runtimes, depending on the backend and configuration. For practical long-context inference beyond 32K, especially when using **llama.cpp / GGUF**, it is recommended to enable **RoPE/YaRN scaling** instead of only increasing `n_ctx` / `--ctx-size`. Directly setting a larger context window without RoPE scaling may work in some cases, but it can be less stable and may not achieve the expected long-context performance. This is consistent with Qwen community guidance for long-context GGUF usage: **128K context generally requires YaRN/RoPE scaling**, and it is not necessarily enabled by default in llama.cpp. For example, Qwen maintainers have noted that "128K context length needs YaRN" and that it should be explicitly enabled when supported by the runtime. Reference: https://huggingface.co/Qwen/Qwen2.5-72B-Instruct-GGUF/discussions/2 Community feedback also suggests that RoPE/YaRN scaling can improve long-context stability for this model family. One user reported that, on **HermesAgent-20**, `Qwopus3.6-35B-A3B-v1` performed better when extending from **32K to 128K via RoPE scaling** than when directly setting a **128K context window** without scaling, with scores of **83 vs. 72** in their setup. This result may vary depending on the backend, quantization type, KV cache settings, hardware, and benchmark configuration, but it is consistent with the recommendation to use RoPE/YaRN scaling for contexts beyond 32K. Example llama.cpp configuration for extending from 32K to 128K: ```bash ./llama-server \ -m model.gguf \ --ctx-size 131072 \ --rope-scaling yarn \ --rope-scale 4 \ --yarn-orig-ctx 32768 ``` For 256K context, users may need to adjust the scaling factor accordingly and validate the result in their own workload: ```bash ./llama-server \ -m model.gguf \ --ctx-size 262144 \ --rope-scaling yarn \ --rope-scale 8 \ --yarn-orig-ctx 32768 ``` Please note that long-context behavior may vary depending on the inference backend, quantization type, KV cache settings, available memory, and task type. For best results, users should benchmark their own target workload when using contexts beyond 32K. --- ## 🤝 8. Collaboration & Training Details This model is a collaborative milestone achieved with hardware engineer **Kyle Hessling**. You can follow him on X / Twitter: [@KyleHessling1](https://x.com/KyleHessling1) to keep up with the latest hardware infrastructure and distributed training updates. 🙏

Dimension	Details & Infrastructure
🖥️ Training Hardware	NVIDIA DGX Cluster / H100 / RTX 6000 Pro
⚙️ Fine-tuning Framework	Unsloth (used for highly efficient SFT of dense models and memory optimization)

--- ## ⚠️ 9. Known Training & Deployment Issues (IMPORTANT) While the 27B dense model architecture is relatively stable, certain low-level framework compatibility issues may still surface during large-scale parameter updates and complex long-context training. **It is highly recommended to monitor the following technical risk points during secondary fine-tuning and deployment:**

Module / Component	Issue & Troubleshooting Diagnostics
🔀 Weight Merge (LoRA Merger)	When merging LoRA adapters back into the base model, it is highly susceptible to peak memory out-of-memory (OOM) errors. Ensure the merging host has sufficient virtual memory or perform the low-precision merge on the CPU.
🛠️ Dependency Compatibility	PEFT, Transformers 5.x fusion mode, and Unsloth patches may occasionally cause module import failures (ImportError) or weight mapping conflicts. Please align your dependency versions with those provided in our finetuning-guide repository.

> [!CAUTION] > **Local Fine-Tuning & Deployment Warning**: If you attempt to run secondary fine-tuning or merge adapter weights locally, please proceed with caution and be prepared to manually patch model definition files or pin dependency versions strictly. --- ## 📚 10. Resources & Guides 👉 **[GitHub Repository: Jackrong-llm-finetuning-guide](https://github.com/R6410418/Jackrong-llm-finetuning-guide.git)** Access the repository to dive into the codebase and reproduce our results locally or on Google Colab. --- ## 🙏 11. Acknowledgements Special thanks to: - The Qwen team for providing the powerful Qwen3.6 base model. - Unsloth for providing the highly efficient fine-tuning framework. - Open-source datasets and community contributors. - **Kyle Hessling** for the close collaboration on this project. --- ## 📖 12. Citation ```bibtex @misc{jackrong_qwopus36_27b_v2, title = {Qwopus3.6-27B-v2}, author = {Jackrong}, year = {2026}, publisher = {Hugging Face} } ```