FGFGFGGDFGDFGSD/qwen3-0.6b-claude-opus-4.7-rus-distilled-GGUF

Qwen3-0.6B-Distill-Opus-v1

Overview

This is an experimental distillation of Qwen3-0.6B-Base. The model was trained using a LoRA adapter (r=24) on a single NVIDIA T4 GPU. The goal was to transfer encyclopedic and linguistic knowledge from Claude Opus 4.7 and ChatGPT 5.4 into a compact 596M parameter architecture. (it was fail i guess)

Technical Specifications

• Parameters: 596M
• Training Method: LoRA (r=24, alpha=48)
• Base Model: Qwen3-0.6B-Base
• Hardware: NVIDIA T4

Evaluation Results

The following table compares this distilled version against stock models and competitors in the sub-2B category.

The distill model was in Q4 quant!!! (approx 1-3% lie)

Model	MMLU (5-shot)	GSM8K (Math)	Parameters
Qwen3-0.6B Distil	44.57	6.00	596M
Qwen3-0.6B (Official)	52.81	59.59	596M
Llama-3.2-1B (Official)	39.00	44.40	1.2B
Gemma-2-2B (Official)	51.30	61.10	2.6B
Qwen2.5-0.5B (Official)	37.90	40.10	0.5B
SmolLM2-1.7B (Official)	45.00	40.00	1.7B

Detailed MMLU Breakdown (57 Categories)

Final Average Accuracy: 46.23%

Category	Accuracy (%)
Marketing	70.09
Sociology	63.18
Computer Security	63.00
High School Psychology	62.94
High School Geography	60.61
High School Biology	59.68
International Law	59.50
Management	59.22
High School Microeconomics	57.56
High School European History	56.97
Security Studies	55.10
High School Government And Politics	54.92
Electrical Engineering	54.48
Clinical Knowledge	54.34
Medical Genetics	54.00
Us Foreign Policy	54.00
Nutrition	53.27
High School Computer Science	52.00
College Medicine	50.29
College Computer Science	50.00
Miscellaneous	49.43
Professional Medicine	48.16
Human Sexuality	47.33
High School World History	47.26
High School Macroeconomics	46.67
High School Us History	46.57
High School Chemistry	46.31
Jurisprudence	46.30
Astronomy	46.05
Public Relations	45.45
High School Statistics	45.37
Anatomy	45.19
College Biology	45.14
Moral Disputes	45.09
World Religions	45.03
Philosophy	44.69
Prehistory	42.90
Conceptual Physics	42.55
College Chemistry	42.00
Professional Psychology	40.52
Virology	40.36
Human Aging	39.01
Professional Accounting	39.01
Business Ethics	39.00
College Mathematics	38.00
High School Physics	37.75
Logical Fallacies	36.81
Formal Logic	35.71
Machine Learning	35.71
Elementary Mathematics	35.19
Econometrics	33.33
Abstract Algebra	33.00
Professional Law	31.49
High School Mathematics	31.48
College Physics	28.43
Moral Scenarios	27.49
Global Facts	20.00

Knowledge Density (Perplexity)

A significant drop in Perplexity (PPL) indicates successful knowledge compression from the teacher models:

• Stock PPL: 10.3566
• Distilled PPL: 8.9864
• Delta: -1.3702

Limitations and Critical Warnings

• Hallucinations: The model is highly prone to severe factual errors in conversational contexts.
• Reasoning Decay: Mathematical logic (GSM8K) has significantly degraded compared to the stock base model.
• Inference Stability: Users may encounter token loops or "stuck" states. Thinking tags/Reasoning chains are not supported in this version.

Recommended Inference Settings

For stable results, use the following configuration:

• Temperature: 0.35
• System Prompt: You - ai assistant
• Stop Strings: ### Question:, ### Response:
• Repetition Penalty: 1.1 - 1.2