Instructions to use AesSedai/Qwen3.6-35B-A3B-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use AesSedai/Qwen3.6-35B-A3B-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Use Docker

docker model run hf.co/AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

LM Studio
Jan
Ollama
How to use AesSedai/Qwen3.6-35B-A3B-GGUF with Ollama:
```
ollama run hf.co/AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M
```

Unsloth Studio

How to use AesSedai/Qwen3.6-35B-A3B-GGUF with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for AesSedai/Qwen3.6-35B-A3B-GGUF to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for AesSedai/Qwen3.6-35B-A3B-GGUF to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for AesSedai/Qwen3.6-35B-A3B-GGUF to start chatting

How to use AesSedai/Qwen3.6-35B-A3B-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

OpenClaw new

How to use AesSedai/Qwen3.6-35B-A3B-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use AesSedai/Qwen3.6-35B-A3B-GGUF with Docker Model Runner:
```
docker model run hf.co/AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M
```

Lemonade

How to use AesSedai/Qwen3.6-35B-A3B-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Qwen3.6-35B-A3B-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use AesSedai/Qwen3.6-35B-A3B-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default AesSedai/Qwen3.6-35B-A3B-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

Q6_K?

by qenme - opened Apr 16

Discussion

qenme

Apr 16

Hi AesSedai,

Big fan; do you have plans for a Q6_K release?

AesSedai

Owner Apr 17

•

edited Apr 17

Hi, I don't really do Q6_K with these recipes simply because of how the mixtures stack:

Q4_K is Q4_K for the Up and Gate and Q5_K for the Down ffns
Q5_K is Q5_K for the Up and Gate and Q6_K for the Down ffns

so a Q6_K theoretically would be:

Q6_K is Q6_K for the Up and Gate and Q8_0 for the Down ffns

and with the rest of the model in Q8_0 already, you're only shaving a small amount of size off of the Q8_0 to step the Up and Gate to the Q6_K level. Honestly IMO you'd just run the Q8_0 at that point?

Benasd

Apr 17

@AesSedai Q6_K will be just enough to fit in RTX 5090. Maybe use Q6_K for Up, Down and Gate would also be a good option?

AesSedai

Owner Apr 18

I'll look into it later today and see where it lands on the chart

qenme

Apr 18

Hi, I don't really do Q6_K with these recipes simply because of how the mixtures stack:

Q4_K is Q4_K for the Up and Gate and Q5_K for the Down ffns

Q5_K is Q5_K for the Up and Gate and Q6_K for the Down ffns

so a Q6_K theoretically would be:

Q6_K is Q6_K for the Up and Gate and Q8_0 for the Down ffns

and with the rest of the model in Q8_0 already, you're only shaving a small amount of size off of the Q8_0 to step the Up and Gate to the Q6_K level. Honestly IMO you'd just run the Q8_0 at that point?

Thanks for the insight!

AesSedai

Owner Apr 19

I've uploaded a Q6_K for Up/Gate/Down, it's a little smaller but not my much. I won't do that regularly but this is a pretty small MoE so it doesn't take much storage to do.

Benasd

Apr 19

I've uploaded a Q6_K for Up/Gate/Down

Huge thanks!!

mancub

Apr 22

I've uploaded a Q6_K for Up/Gate/Down, it's a little smaller but not my much. I won't do that regularly but this is a pretty small MoE so it doesn't take much storage to do.

Oh please make the 27B too...I like your Q6_Ks because I run them on a 2x3090 with tons of context. In general your quants appear to be hitting the mark 99% of the time on first shot queries.

AesSedai

Owner Apr 22

Hi @mancub , I don't really quantize the dense models because they don't have the same PPL / KLD benefit as the MoE's do (hence why we call it MoE-optimized!).

I did try it on a dense model before but it just doesn't have the same improvement effect compared to the regular quantization recipes, sorry :(

mancub

Apr 22

Thanks for doing this great work anyway!

IHaveNoClueAndIMustPost

May 17

Been trying several versions of the 35B model hoping to move from the 27B variant for more speed but haven't been convinced. The quality drop has been noticeable.

But this Q6_K model is on a whole other level as the PPL hints at. Compared to the other variants I've tried it excels not only at code but also in discussions on how to tackle various problems and implementations.. Yes, a MoE model requires more hand-holding in the form of plans and documentation, but the kicker is that this model is smart enough to write them itself with only basic prompting. Alternatively, you could have the 27B or another larger model do it for you while you grab a coffee. For anyone who already follows a plan → execute → evaluate workflow or similar, it's zero extra effort for a lot gained in PP/TG performance.

That said, I'm only doing Python stuff and staying below 128K context, so YMMV.

mancub

May 18

I use VSCode+CC as my main setup, so I ended up using the MoE model (Q4 though) as a lower-tier model (sonet/haiku) which CC now uses for agents (and it's really fast and good at it). The 27B I relegated as opus and that's like the main figuring things out model I guess.

Problem is CC wants 200k+ tokens budget and while I do have around 200k with 27B, it still runs out, so something isn't fully set up yet. But I feel spoiled now being able to run the 27B as the main model, and the 35B as an agentic one, LOL. With speculative tokens my 3x3090s deliver very usable performance in vllm.

AesSedai

Owner May 18

I've updated these quants with MTP support.

qenme

May 18

I've updated these quants with MTP support.

Fire, thanks man.

AesSedai

Owner May 18

I just realized I messed up and didn't do the fused gate+up, so the imatrix wasn't right (since my imatrix was based on that). I'm re-quanting these and will have it corrected shortly. Sorry D:

AesSedai

Owner May 18

Follow-up, fixed the issue and have the corrects quants (fused + MTP) uploaded now.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment