Text Generation
Transformers
Safetensors
GGUF
English
qwen3_5
image-text-to-text
sql
text-to-sql
fine-tuned
lora
sft
trl
unsloth
neo-deep-agent-lab
modal
conversational
Eval Results (legacy)
Instructions to use Shumatsurontek/Qwen3.5-4B-neo with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Shumatsurontek/Qwen3.5-4B-neo with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Shumatsurontek/Qwen3.5-4B-neo") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] pipe(text=messages)# Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("Shumatsurontek/Qwen3.5-4B-neo") model = AutoModelForMultimodalLM.from_pretrained("Shumatsurontek/Qwen3.5-4B-neo", device_map="auto") messages = [ { "role": "user", "content": [ {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"}, {"type": "text", "text": "What animal is on the candy?"} ] }, ] inputs = processor.apply_chat_template( messages, add_generation_prompt=True, tokenize=True, return_dict=True, return_tensors="pt", ).to(model.device) outputs = model.generate(**inputs, max_new_tokens=40) print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:])) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use Shumatsurontek/Qwen3.5-4B-neo with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf Shumatsurontek/Qwen3.5-4B-neo # Run inference directly in the terminal: llama cli -hf Shumatsurontek/Qwen3.5-4B-neo
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf Shumatsurontek/Qwen3.5-4B-neo # Run inference directly in the terminal: llama cli -hf Shumatsurontek/Qwen3.5-4B-neo
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf Shumatsurontek/Qwen3.5-4B-neo # Run inference directly in the terminal: ./llama-cli -hf Shumatsurontek/Qwen3.5-4B-neo
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf Shumatsurontek/Qwen3.5-4B-neo # Run inference directly in the terminal: ./build/bin/llama-cli -hf Shumatsurontek/Qwen3.5-4B-neo
Use Docker
docker model run hf.co/Shumatsurontek/Qwen3.5-4B-neo
- LM Studio
- Jan
- vLLM
How to use Shumatsurontek/Qwen3.5-4B-neo with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Shumatsurontek/Qwen3.5-4B-neo" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Shumatsurontek/Qwen3.5-4B-neo", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker
docker model run hf.co/Shumatsurontek/Qwen3.5-4B-neo
- SGLang
How to use Shumatsurontek/Qwen3.5-4B-neo with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Shumatsurontek/Qwen3.5-4B-neo" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Shumatsurontek/Qwen3.5-4B-neo", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Shumatsurontek/Qwen3.5-4B-neo" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Shumatsurontek/Qwen3.5-4B-neo", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }' - Ollama
How to use Shumatsurontek/Qwen3.5-4B-neo with Ollama:
ollama run hf.co/Shumatsurontek/Qwen3.5-4B-neo
- Unsloth Studio
How to use Shumatsurontek/Qwen3.5-4B-neo with Unsloth Studio:
Install Unsloth Studio (macOS, Linux, WSL)
curl -fsSL https://unsloth.ai/install.sh | sh # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for Shumatsurontek/Qwen3.5-4B-neo to start chatting
Install Unsloth Studio (Windows)
irm https://unsloth.ai/install.ps1 | iex # Run unsloth studio unsloth studio -H 0.0.0.0 -p 8888 # Then open http://localhost:8888 in your browser # Search for Shumatsurontek/Qwen3.5-4B-neo to start chatting
Using HuggingFace Spaces for Unsloth
# No setup required # Open https://huggingface.co/spaces/unsloth/studio in your browser # Search for Shumatsurontek/Qwen3.5-4B-neo to start chatting
- Pi
How to use Shumatsurontek/Qwen3.5-4B-neo with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Shumatsurontek/Qwen3.5-4B-neo
Configure the model in Pi
# Install Pi: npm install -g @mariozechner/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "Shumatsurontek/Qwen3.5-4B-neo" } ] } } }Run Pi
# Start Pi in your project directory: pi
- OpenClaw new
How to use Shumatsurontek/Qwen3.5-4B-neo with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Shumatsurontek/Qwen3.5-4B-neo
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "Shumatsurontek/Qwen3.5-4B-neo" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
- Docker Model Runner
How to use Shumatsurontek/Qwen3.5-4B-neo with Docker Model Runner:
docker model run hf.co/Shumatsurontek/Qwen3.5-4B-neo
- Lemonade
How to use Shumatsurontek/Qwen3.5-4B-neo with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull Shumatsurontek/Qwen3.5-4B-neo
Run and chat with the model
lemonade run user.Qwen3.5-4B-neo-{{QUANT_TAG}}List all available models
lemonade list
- Hermes Agent
How to use Shumatsurontek/Qwen3.5-4B-neo with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf Shumatsurontek/Qwen3.5-4B-neo
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default Shumatsurontek/Qwen3.5-4B-neo
Run Hermes
hermes
- Atomic Chat
Add benchmark results (mmlu,gsm8k,arc_challenge,hellaswag)
Browse files
README.md
CHANGED
|
@@ -160,136 +160,19 @@ or production use without human review of generated SQL.
|
|
| 160 |
|
| 161 |
## Benchmark Results
|
| 162 |
|
| 163 |
-
|
| 164 |
-
|
| 165 |
-
>
|
| 166 |
-
|
| 167 |
-
|
|
| 168 |
-
|---|---|---|---|
|
| 169 |
-
| **
|
| 170 |
-
| **
|
| 171 |
-
| **
|
| 172 |
-
| **
|
| 173 |
-
| **
|
| 174 |
-
| **
|
| 175 |
-
| **
|
| 176 |
-
| **mmlu_high_school_chemistry** | acc_stderr | 5.2% | **5.7%** | π’ +0.5% |
|
| 177 |
-
| **mmlu_professional_medicine** | acc | 86.0% | **78.0%** | π΄ -8.0% |
|
| 178 |
-
| **mmlu_professional_medicine** | acc_stderr | 5.0% | **5.9%** | π’ +1.0% |
|
| 179 |
-
| **mmlu_moral_scenarios** | acc | 46.0% | **42.0%** | π΄ -4.0% |
|
| 180 |
-
| **mmlu_moral_scenarios** | acc_stderr | 7.1% | **7.0%** | π΄ -0.1% |
|
| 181 |
-
| **mmlu_high_school_government_and_politics** | acc | 96.0% | **94.0%** | π΄ -2.0% |
|
| 182 |
-
| **mmlu_high_school_government_and_politics** | acc_stderr | 2.8% | **3.4%** | π’ +0.6% |
|
| 183 |
-
| **mmlu_high_school_psychology** | acc | 96.0% | **94.0%** | π΄ -2.0% |
|
| 184 |
-
| **mmlu_high_school_psychology** | acc_stderr | 2.8% | **3.4%** | π’ +0.6% |
|
| 185 |
-
| **mmlu_high_school_statistics** | acc | 62.0% | **60.0%** | π΄ -2.0% |
|
| 186 |
-
| **mmlu_high_school_statistics** | acc_stderr | 6.9% | **7.0%** | π’ +0.1% |
|
| 187 |
-
| **mmlu_high_school_mathematics** | acc | 48.0% | **58.0%** | π’ +10.0% |
|
| 188 |
-
| **mmlu_high_school_mathematics** | acc_stderr | 7.1% | **7.0%** | π΄ -0.1% |
|
| 189 |
-
| **mmlu_abstract_algebra** | acc | 58.0% | **60.0%** | π’ +2.0% |
|
| 190 |
-
| **mmlu_abstract_algebra** | acc_stderr | 7.0% | **7.0%** | π΄ -0.1% |
|
| 191 |
-
| **mmlu_computer_security** | acc | 88.0% | **82.0%** | π΄ -6.0% |
|
| 192 |
-
| **mmlu_computer_security** | acc_stderr | 4.6% | **5.5%** | π’ +0.9% |
|
| 193 |
-
| **mmlu_miscellaneous** | acc | 82.0% | **78.0%** | π΄ -4.0% |
|
| 194 |
-
| **mmlu_miscellaneous** | acc_stderr | 5.5% | **5.9%** | π’ +0.4% |
|
| 195 |
-
| **mmlu_high_school_european_history** | acc | 82.0% | **76.0%** | π΄ -6.0% |
|
| 196 |
-
| **mmlu_high_school_european_history** | acc_stderr | 5.5% | **6.1%** | π’ +0.6% |
|
| 197 |
-
| **mmlu_high_school_computer_science** | acc | 88.0% | **84.0%** | π΄ -4.0% |
|
| 198 |
-
| **mmlu_high_school_computer_science** | acc_stderr | 4.6% | **5.2%** | π’ +0.6% |
|
| 199 |
-
| **mmlu_high_school_physics** | acc | 68.0% | **64.0%** | π΄ -4.0% |
|
| 200 |
-
| **mmlu_high_school_physics** | acc_stderr | 6.7% | **6.9%** | π’ +0.2% |
|
| 201 |
-
| **mmlu_human_sexuality** | acc | 84.0% | **86.0%** | π’ +2.0% |
|
| 202 |
-
| **mmlu_human_sexuality** | acc_stderr | 5.2% | **5.0%** | π΄ -0.3% |
|
| 203 |
-
| **mmlu_machine_learning** | acc | 56.0% | **58.0%** | π’ +2.0% |
|
| 204 |
-
| **mmlu_machine_learning** | acc_stderr | 7.1% | **7.0%** | π΄ -0.0% |
|
| 205 |
-
| **mmlu_astronomy** | acc | 94.0% | **94.0%** | βͺ 0.0% |
|
| 206 |
-
| **mmlu_astronomy** | acc_stderr | 3.4% | **3.4%** | βͺ 0.0% |
|
| 207 |
-
| **mmlu_professional_law** | acc | 52.0% | **52.0%** | βͺ 0.0% |
|
| 208 |
-
| **mmlu_professional_law** | acc_stderr | 7.1% | **7.1%** | βͺ 0.0% |
|
| 209 |
-
| **mmlu_high_school_us_history** | acc | 90.0% | **84.0%** | π΄ -6.0% |
|
| 210 |
-
| **mmlu_high_school_us_history** | acc_stderr | 4.3% | **5.2%** | π’ +0.9% |
|
| 211 |
-
| **mmlu_management** | acc | 94.0% | **94.0%** | βͺ 0.0% |
|
| 212 |
-
| **mmlu_management** | acc_stderr | 3.4% | **3.4%** | βͺ 0.0% |
|
| 213 |
-
| **mmlu_college_computer_science** | acc | 72.0% | **74.0%** | π’ +2.0% |
|
| 214 |
-
| **mmlu_college_computer_science** | acc_stderr | 6.4% | **6.3%** | π΄ -0.1% |
|
| 215 |
-
| **mmlu_virology** | acc | 60.0% | **60.0%** | βͺ 0.0% |
|
| 216 |
-
| **mmlu_virology** | acc_stderr | 7.0% | **7.0%** | βͺ 0.0% |
|
| 217 |
-
| **mmlu_jurisprudence** | acc | 82.0% | **86.0%** | π’ +4.0% |
|
| 218 |
-
| **mmlu_jurisprudence** | acc_stderr | 5.5% | **5.0%** | π΄ -0.5% |
|
| 219 |
-
| **mmlu_college_medicine** | acc | 82.0% | **80.0%** | π΄ -2.0% |
|
| 220 |
-
| **mmlu_college_medicine** | acc_stderr | 5.5% | **5.7%** | π’ +0.2% |
|
| 221 |
-
| **mmlu_professional_psychology** | acc | 86.0% | **80.0%** | π΄ -6.0% |
|
| 222 |
-
| **mmlu_professional_psychology** | acc_stderr | 5.0% | **5.7%** | π’ +0.8% |
|
| 223 |
-
| **mmlu_conceptual_physics** | acc | 86.0% | **88.0%** | π’ +2.0% |
|
| 224 |
-
| **mmlu_conceptual_physics** | acc_stderr | 5.0% | **4.6%** | π΄ -0.3% |
|
| 225 |
-
| **mmlu_social_sciences** | acc | 83.0% | **79.3%** | π΄ -3.7% |
|
| 226 |
-
| **mmlu_social_sciences** | acc_stderr | 1.5% | **1.6%** | π’ +0.1% |
|
| 227 |
-
| **mmlu_electrical_engineering** | acc | 82.0% | **84.0%** | π’ +2.0% |
|
| 228 |
-
| **mmlu_electrical_engineering** | acc_stderr | 5.5% | **5.2%** | π΄ -0.2% |
|
| 229 |
-
| **mmlu_international_law** | acc | 90.0% | **88.0%** | π΄ -2.0% |
|
| 230 |
-
| **mmlu_international_law** | acc_stderr | 4.3% | **4.6%** | π’ +0.4% |
|
| 231 |
-
| **mmlu_global_facts** | acc | 42.0% | **36.0%** | π΄ -6.0% |
|
| 232 |
-
| **mmlu_global_facts** | acc_stderr | 7.0% | **6.9%** | π΄ -0.2% |
|
| 233 |
-
| **mmlu_high_school_geography** | acc | 90.0% | **80.0%** | π΄ -10.0% |
|
| 234 |
-
| **mmlu_high_school_geography** | acc_stderr | 4.3% | **5.7%** | π’ +1.4% |
|
| 235 |
-
| **mmlu_philosophy** | acc | 82.0% | **82.0%** | βͺ 0.0% |
|
| 236 |
-
| **mmlu_philosophy** | acc_stderr | 5.5% | **5.5%** | βͺ 0.0% |
|
| 237 |
-
| **mmlu_world_religions** | acc | 86.0% | **84.0%** | π΄ -2.0% |
|
| 238 |
-
| **mmlu_world_religions** | acc_stderr | 5.0% | **5.2%** | π’ +0.3% |
|
| 239 |
-
| **mmlu_sociology** | acc | 88.0% | **84.0%** | π΄ -4.0% |
|
| 240 |
-
| **mmlu_sociology** | acc_stderr | 4.6% | **5.2%** | π’ +0.6% |
|
| 241 |
-
| **mmlu_stem** | acc | 73.5% | **72.3%** | π΄ -1.1% |
|
| 242 |
-
| **mmlu_stem** | acc_stderr | 1.4% | **1.4%** | π’ +0.0% |
|
| 243 |
-
| **mmlu_marketing** | acc | 92.0% | **92.0%** | βͺ 0.0% |
|
| 244 |
-
| **mmlu_marketing** | acc_stderr | 3.9% | **3.9%** | βͺ 0.0% |
|
| 245 |
-
| **mmlu_public_relations** | acc | 74.0% | **64.0%** | π΄ -10.0% |
|
| 246 |
-
| **mmlu_public_relations** | acc_stderr | 6.3% | **6.9%** | π’ +0.6% |
|
| 247 |
-
| **mmlu_college_mathematics** | acc | 60.0% | **52.0%** | π΄ -8.0% |
|
| 248 |
-
| **mmlu_college_mathematics** | acc_stderr | 7.0% | **7.1%** | π’ +0.1% |
|
| 249 |
-
| **mmlu_elementary_mathematics** | acc | 80.0% | **74.0%** | π΄ -6.0% |
|
| 250 |
-
| **mmlu_elementary_mathematics** | acc_stderr | 5.7% | **6.3%** | π’ +0.6% |
|
| 251 |
-
| **mmlu_logical_fallacies** | acc | 82.0% | **74.0%** | π΄ -8.0% |
|
| 252 |
-
| **mmlu_logical_fallacies** | acc_stderr | 5.5% | **6.3%** | π’ +0.8% |
|
| 253 |
-
| **mmlu_medical_genetics** | acc | 90.0% | **92.0%** | π’ +2.0% |
|
| 254 |
-
| **mmlu_medical_genetics** | acc_stderr | 4.3% | **3.9%** | π΄ -0.4% |
|
| 255 |
-
| **mmlu_econometrics** | acc | 66.0% | **64.0%** | π΄ -2.0% |
|
| 256 |
-
| **mmlu_econometrics** | acc_stderr | 6.8% | **6.9%** | π’ +0.1% |
|
| 257 |
-
| **mmlu_us_foreign_policy** | acc | 80.0% | **80.0%** | βͺ 0.0% |
|
| 258 |
-
| **mmlu_us_foreign_policy** | acc_stderr | 5.7% | **5.7%** | βͺ 0.0% |
|
| 259 |
-
| **mmlu** | acc | 77.1% | **75.0%** | π΄ -2.1% |
|
| 260 |
-
| **mmlu** | acc_stderr | 0.8% | **0.8%** | π’ +0.0% |
|
| 261 |
-
| **mmlu_college_biology** | acc | 90.0% | **88.0%** | π΄ -2.0% |
|
| 262 |
-
| **mmlu_college_biology** | acc_stderr | 4.3% | **4.6%** | π’ +0.4% |
|
| 263 |
-
| **mmlu_moral_disputes** | acc | 64.0% | **66.0%** | π’ +2.0% |
|
| 264 |
-
| **mmlu_moral_disputes** | acc_stderr | 6.9% | **6.8%** | π΄ -0.1% |
|
| 265 |
-
| **mmlu_high_school_biology** | acc | 90.0% | **88.0%** | π΄ -2.0% |
|
| 266 |
-
| **mmlu_high_school_biology** | acc_stderr | 4.3% | **4.6%** | π’ +0.4% |
|
| 267 |
-
| **mmlu_professional_accounting** | acc | 52.0% | **54.0%** | π’ +2.0% |
|
| 268 |
-
| **mmlu_professional_accounting** | acc_stderr | 7.1% | **7.1%** | π΄ -0.0% |
|
| 269 |
-
| **mmlu_security_studies** | acc | 74.0% | **70.0%** | π΄ -4.0% |
|
| 270 |
-
| **mmlu_security_studies** | acc_stderr | 6.3% | **6.6%** | π’ +0.3% |
|
| 271 |
-
| **mmlu_high_school_macroeconomics** | acc | 74.0% | **68.0%** | π΄ -6.0% |
|
| 272 |
-
| **mmlu_high_school_macroeconomics** | acc_stderr | 6.3% | **6.7%** | π’ +0.4% |
|
| 273 |
-
| **mmlu_high_school_microeconomics** | acc | 88.0% | **88.0%** | βͺ 0.0% |
|
| 274 |
-
| **mmlu_high_school_microeconomics** | acc_stderr | 4.6% | **4.6%** | βͺ 0.0% |
|
| 275 |
-
| **mmlu_clinical_knowledge** | acc | 78.0% | **78.0%** | βͺ 0.0% |
|
| 276 |
-
| **mmlu_clinical_knowledge** | acc_stderr | 5.9% | **5.9%** | βͺ 0.0% |
|
| 277 |
-
| **mmlu_business_ethics** | acc | 84.0% | **84.0%** | βͺ 0.0% |
|
| 278 |
-
| **mmlu_business_ethics** | acc_stderr | 5.2% | **5.2%** | βͺ 0.0% |
|
| 279 |
-
| **mmlu_college_chemistry** | acc | 54.0% | **54.0%** | βͺ 0.0% |
|
| 280 |
-
| **mmlu_college_chemistry** | acc_stderr | 7.1% | **7.1%** | βͺ 0.0% |
|
| 281 |
-
| **mmlu_other** | acc | 77.7% | **76.3%** | π΄ -1.4% |
|
| 282 |
-
| **mmlu_other** | acc_stderr | 1.5% | **1.6%** | π’ +0.0% |
|
| 283 |
-
| **mmlu_prehistory** | acc | 80.0% | **76.0%** | π΄ -4.0% |
|
| 284 |
-
| **mmlu_prehistory** | acc_stderr | 5.7% | **6.1%** | π’ +0.4% |
|
| 285 |
-
| **mmlu_humanities** | acc | 76.5% | **73.9%** | π΄ -2.6% |
|
| 286 |
-
| **mmlu_humanities** | acc_stderr | 1.6% | **1.7%** | π’ +0.1% |
|
| 287 |
-
| **mmlu_high_school_world_history** | acc | 90.0% | **86.0%** | π΄ -4.0% |
|
| 288 |
-
| **mmlu_high_school_world_history** | acc_stderr | 4.3% | **5.0%** | π’ +0.7% |
|
| 289 |
-
| **mmlu_college_physics** | acc | 68.0% | **66.0%** | π΄ -2.0% |
|
| 290 |
-
| **mmlu_college_physics** | acc_stderr | 6.7% | **6.8%** | π’ +0.1% |
|
| 291 |
-
| **mmlu_formal_logic** | acc | 68.0% | **64.0%** | π΄ -4.0% |
|
| 292 |
-
| **mmlu_formal_logic** | acc_stderr | 6.7% | **6.9%** | π’ +0.2% |
|
| 293 |
|
| 294 |
## Citation
|
| 295 |
|
|
|
|
| 160 |
|
| 161 |
## Benchmark Results
|
| 162 |
|
| 163 |
+
Evaluated against baseline [`unsloth/Qwen3.5-4B`](https://huggingface.co/unsloth/Qwen3.5-4B) using [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness) on NVIDIA L40S.
|
| 164 |
+
|
| 165 |
+
> Evaluated on 50 samples per task.
|
| 166 |
+
|
| 167 |
+
| Benchmark | Baseline | Finetuned | Delta |
|
| 168 |
+
|---|:---:|:---:|:---:|
|
| 169 |
+
| **MMLU: STEM** | 73.5 | 71.8 | π΄ -1.7 |
|
| 170 |
+
| **MMLU: HUMANITIES** | 76.5 | 73.4 | π΄ -3.1 |
|
| 171 |
+
| **HELLASWAG** | 48.0 | 52.0 | π’ +4.0 |
|
| 172 |
+
| **MMLU** | 77.1 | 74.7 | π΄ -2.4 |
|
| 173 |
+
| **MMLU: OTHER** | 77.7 | 75.8 | π΄ -1.8 |
|
| 174 |
+
| **ARC_CHALLENGE** | 60.0 | 60.0 | βͺ 0.0 |
|
| 175 |
+
| **MMLU: SOCIAL SCIENCES** | 83.0 | 79.7 | π΄ -3.3 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 176 |
|
| 177 |
## Citation
|
| 178 |
|