j2ampn/qwen3-8b-distractor-lora-v8

Qwen3-8B Diagnostic Distractor LoRA v8

This is a PEFT LoRA adapter trained to propose diagnostic wrong answers for sixth-grade and middle-school Number mathematics questions. Given a trusted question, correct answer, and topic, it is trained to return exactly three different distractors. Each distractor names a distinct student misconception, shows the question-specific arithmetic that misconception would produce, and reports the resulting answer.

The outputs are hypotheses for assessment authors, not diagnoses of learners. Generated content must be parsed and independently checked before it reaches a student.

Model and artifact identity

• Base model: unsloth/Qwen3-8B-bnb-4bit
• Immutable base revision: 1deaf68f694c40dbce295da300851729d759b21a
• Adapter type: causal-LM LoRA, rank 32, alpha 32, dropout 0
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, and down_proj
• PEFT version recorded by the artifact: 0.19.1
• Recovered ZIP: 325,882,649 bytes; SHA-256 e00dcb7653e9baa19fb103bbe0712b419fccb281724b487847de7f03a960c7fb
• adapter_model.safetensors: 349,243,752 bytes; SHA-256 e949ee36800f429ba5dc02b761aa54bf8037af6d406fac9e5a04c7b68cce4a12

The recovered ZIP named the correct base model but stored revision: null in adapter_config.json. For this publication, that metadata field was set to the immutable revision in the training receipt. Adapter weights and tokenizer files were not changed.

Output contract

The model was supervised to emit only one JSON object with this shape:

{
  "distractors": [
    {
      "misconception": "<short misconception>",
      "computation": "<arithmetic> = <value>",
      "answer": "<value>"
    },
    {
      "misconception": "...",
      "computation": "...",
      "answer": "..."
    },
    {
      "misconception": "...",
      "computation": "...",
      "answer": "..."
    }
  ]
}

The intended constraints are:

1. exactly three distractors;
2. three distinct, specific misconceptions;
3. three distinct wrong answers, none equal to the key; and
4. each computation evaluates to its paired answer for the supplied question.

These are generation targets, not guarantees. Enforce them in trusted code.

Usage

Qwen3 support requires a current Transformers release. The base is a bitsandbytes 4-bit checkpoint, so use a compatible CUDA environment.

import json
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

ADAPTER = "j2ampn/qwen3-8b-distractor-lora-v8"
BASE = "unsloth/Qwen3-8B-bnb-4bit"
BASE_REVISION = "1deaf68f694c40dbce295da300851729d759b21a"

SYSTEM_PROMPT = """You are an expert middle-school mathematics assessment writer. Given a "Number" strand math question and its correct answer, produce exactly three diagnostic distractors (wrong answers) for a multiple-choice version.

For each distractor provide, in this order:
- misconception: the specific student misconception or procedural error.
- computation: the exact arithmetic a student with THAT misconception performs on THIS question, written as a plain expression that ends in '= <answer>' (e.g. "0.4 ÷ 0.2 = 2"). Use only digits, + - × ÷, parentheses, decimals, and fractions a/b.
- answer: the value the computation evaluates to. It MUST equal the computation's result.

Rules:
- Exactly 3 distractors, each tagged to a distinct misconception.
- Each answer is exactly what a student making that misconception would compute (numerically consistent with the misconception and its shown computation).
- The three answers must all be different, and none may equal the correct answer.

Respond with ONLY a JSON object, no prose, in this exact schema:
{"distractors": [{"misconception": "<short misconception>", "computation": "<arithmetic> = <value>", "answer": "<value>"}, {"misconception": "...", "computation": "...", "answer": "..."}, {"misconception": "...", "computation": "...", "answer": "..."}]}"""

tokenizer = AutoTokenizer.from_pretrained(ADAPTER)
base = AutoModelForCausalLM.from_pretrained(
    BASE,
    revision=BASE_REVISION,
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {
        "role": "user",
        "content": (
            "Question: What is 3/4 of 20?\n"
            "Correct answer: 15\n"
            "Topic: Fractions"
        ),
    },
]
input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    output_ids = model.generate(
        input_ids=input_ids,
        max_new_tokens=512,
        do_sample=False,
    )

text = tokenizer.decode(
    output_ids[0, input_ids.shape[1]:],
    skip_special_tokens=True,
)
payload = json.loads(text)
print(payload)

The registered model-only track uses greedy generation as shown above. The separate best-of-N result below does not come from this single call.

Training

“One-shot” means one planned training invocation, not one training example. The run used QLoRA SFT against the pinned 4-bit base for three planned epochs with:

• 3,572 verified training rows and a deterministic prompt-grouped train/validation split;
• response-only loss, maximum sequence length 2,048;
• batch size 1, gradient accumulation 8;
• learning rate 1.5e-4, cosine schedule, 5% warmup;
• 8-bit AdamW, weight decay 0.01, seed 42; and
• automatic restoration of the checkpoint with lowest validation loss.

The selected checkpoint was outputs_v8/checkpoint-403, with validation loss 0.051324423402547836. The frozen 140-item benchmark was not used for training, validation, checkpoint selection, or tuning.

Training combined programmatically generated misconception procedures with filtered real-question targets. No raw training dataset is included in this model repository.

Evaluation

The following are the final deterministic hard-gate results from TABLE_V8_RESULTS.md. Brackets are 95% intervals.

Metric	Opus generator baseline	v8 model-only	v8 verifier-guided best-of-4
Valid exactly-3 output	97.1% (136/140) [92.9, 98.9]	100.0% (140/140) [97.3, 100.0]	100.0% (140/140) [97.3, 100.0]
No answer equals key	94.3% (132/140) [89.1, 97.1]	94.3% (132/140) [89.1, 97.1]	96.4% (135/140) [91.9, 98.5]
Three distinct answers	85.0% (119/140) [78.2, 90.0]	80.7% (113/140) [73.4, 86.4]	96.4% (135/140) [91.9, 98.5]
Three distinct misconceptions	97.1% (136/140) [92.9, 98.9]	100.0% (140/140) [97.3, 100.0]	100.0% (140/140) [97.3, 100.0]
Hardened computation validity	40.3% (170/422) [34.2, 47.4]	79.3% (333/420) [73.3, 84.5]	84.8% (356/420) [79.0, 89.3]

Model-only is deterministic greedy output from the adapter. Best-of-4 is a system track: it adds three seeded sampled candidates to the greedy candidate, then uses trusted local code to select by structure, key safety, distinctness, and hardened computation checks. It must not be described as model-only performance.

Good Distractor Rate (GDR), Good@3, holistic diagnostic-quality/plausibility, and the registered overall win rule are unavailable / not demonstrated. There is no accepted independent judge or completed human review for those holistic measures. There are also no observed student option-pick frequencies; the evaluation cannot establish that any distractor is frequently selected by students.

Aggregate machine-readable results are in benchmark_summary.json. No protected benchmark questions, raw Eedi records, or prediction rows are distributed here.

Limitations

• The model can emit malformed JSON, a correct answer as a distractor, duplicate answers, invalid arithmetic, or a misconception label that does not explain its answer.
• Deterministic arithmetic checks do not establish student plausibility, diagnostic usefulness, or misconception-to-answer validity in every case.
• The benchmark covers English middle-school Number content and should not be generalized to other subjects, languages, ages, or high-stakes decisions.
• A generated misconception is a content-design hypothesis. It must not be used as an automated diagnosis of a learner.
• Results do not measure observed student choice frequency.
• The best-of-4 figures require the repository verifier and additional generation; loading this adapter alone reproduces only the model-only track.

Safety, privacy, and local game use

The intended Wayline game path runs inference locally. Trusted code supplies only the question, correct answer, topic, and fixed prompt. It does not send a learner's name, profile/session ID, answer selection, confidence, or progress to the model. No Hugging Face or provider credential is embedded in these files or required after local download.

Treat raw model text as untrusted. Parse exact JSON, reject key collisions and duplicates, evaluate arithmetic, apply curriculum constraints, and use human review for released content. The model should not directly grade students, change progression, or present unverified material to children.

For a local llama.cpp game runtime, the base and adapter must first be merged and exported to a separately validated GGUF. This repository does not claim that the adapter by itself is a production-ready game package.

License and data terms

The adapter is released under Apache-2.0, matching the pinned base model's declared license and the owner's prior adapter release. The source repository describes raw Eedi Kaggle inputs under separate CC BY-NC 4.0 terms. No raw Eedi data is included here, and the model license does not replace any source-data terms that apply when reproducing the training pipeline.

Source and frozen evidence

• Source repository: https://github.com/jsonjj/diagnostic-distractor-slm
• Frozen source revision: 5aa7146b0fa7fc71efbd10feac4e57c2839e05a4
• Final results: TABLE_V8_RESULTS.md
• Artifact validation: v8_artifact_validation.json
• Final benchmark record: benchmark_v8_final.json
• Training receipt: v8_training_receipt.json