moolm Qwen3.8-27B SFT (LoRA)
QLoRA adapter that teaches Qwen/Qwen3.8-27B to write, explain, and refactor MOOCode, reason about Torchship design, and drive the live #340 agent harness (tool calls).
This is an adapter-only release (~897 MB). Load it on top of the base model; it is not a full merge.
Intended use
- MOOCode authoring, explanation, review, refactor, and dialect translation (mooR / classic LambdaMOO-style)
- Agentic multi-turn tool use against Torchship-style MCP tool schemas
- Chat-style design / principles / conventions Q&A grounded in the moolm corpus
Out of scope: general-purpose chat replacement for the base model; vision/video (this SFT is text/tool oriented); unsupervised production writes without a sandbox.
Training data
Built by the moolm pipeline from mined Torchship sessions plus synthesized, compile-checked examples.
| Source file | sft.reasoned.jsonl |
| Rows in file | 18,485 |
| Kept for this run | 7,946 complete examples with sequence length in [1, 4096] |
| Dropped | 3,135 over-budget rows (never truncated) |
| Tracks (full file) | ~57% agentic / ~43% chat |
| Dialects (full file) | ~99% mooR, ~1% LambdaMOO |
| Sources (full file) | Torchship, Sindome, HellMOO, FallMOO |
Training only supervised the weight=1 assistant turn in each window. Tool schemas were omitted from the chat template (tool calls retained). Nearly all rows carry reasoning_content from the reasoning backfill.
Training details
| Hyperparameter | Value |
|---|---|
| Base | Qwen/Qwen3.8-27B (local text path; VL tower unused) |
| Method | QLoRA (NF4, double quant) + LoRA |
| Hardware | 1× NVIDIA A100-SXM4-40GB |
| Max sequence | 4096 (complete examples only) |
| LoRA rank / alpha / dropout | 32 / 64 / 0.05 |
| Target modules | q/k/v/o_proj, gate/up/down_proj, in_proj_{a,b,qkv,z}, out_proj |
| Trainable params | 233,455,616 (~0.86%) |
| Epochs | 2 |
| Effective batch | 16 (per_device=1, grad_accum=16) |
| LR | 1e-4 cosine, warmup 70 steps |
| Optim | paged AdamW 8-bit |
| Precision | bf16 compute |
| Attention | SDPA |
| Seed | 7 |
| Final train loss | ~0.575 |
Run artifact: training/output/qwen38-sft-full/final (completed 2026-08-16).
How to use
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
base_id = "Qwen/Qwen3.8-27B"
adapter_id = "moolm/candle-qwen38-27b-sft-1p" # replace ORG
tokenizer = AutoTokenizer.from_pretrained(adapter_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
base_id,
torch_dtype=torch.bfloat16,
device_map="auto",
trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter_id)
For 4-bit loading, pass a BitsAndBytesConfig when constructing the base model, then attach the adapter the same way.
Limitations
- Sequence budget for this adapter is 4096; longer agent windows from the dataset were excluded.
- Behavior is skewed toward Torchship / mooR; classic LambdaMOO coverage is thin.
- Agentic outputs assume Torchship
#340-style tools; other harnesses need their own schemas in context. - See Evaluation for held-out compile / tool metrics on A100-unused rows.
Evaluation
Held-out set drawn from the A100-unused slice of sft.reasoned.jsonl: complete
examples whose training encoding exceeded 4096 tokens (never truncated into the
SFT run). Stratified by family from the length band that still fits eval context.
| Split | a100_unused (qwen38_unused_eval.jsonl) |
| Samples scored | 128 |
| Encoded length (train measure) | 4203–12269 (median 9865) |
| Completions with ```moocode fences | 49 |
| Compile rate (moocheck) | 98.0% (48/49) |
| Gold tool-call rows | 41 |
| Tool-name exact set match | 0.0% |
| Mean tool-name F1 | 0.000 |
| Mean content token Jaccard | 0.202 |
| Adapter | /tank/data/Dev/moolm/training/output/qwen38-sft-full/final |
| Eval timestamp (UTC) | 2026-08-17T04:34:47.043036+00:00 |
Compile rate by family (rows with code fences)
| family | ok/n | rate |
|---|---|---|
| agentic_task | 33/34 | 97.1% |
| agentic_turn | 7/7 | 100.0% |
| principled_pushback | 8/8 | 100.0% |
Generation: greedy-ish sampling (temperature=0.3), thinking enabled then stripped
for scoring; prompts capped so prompt+new tokens fit the eval context budget.
License
Apache 2.0, following the base Qwen3.8-27B license. You must also comply with Qwen's terms when redistributing derivatives.
- Downloads last month
- 22
Model tree for moolm/candle-qwen38-27b-sft-1p
Base model
Qwen/Qwen3.8-27B
from peft import PeftModel from transformers import AutoModelForCausalLM base_model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.8-27B") model = PeftModel.from_pretrained(base_model, "moolm/candle-qwen38-27b-sft-1p")