professorsynapse/eh-qwen3-4b-seq-sft-dpo-seed2-lora

Epistemic Humility: Qwen3-4B sequential SFT then DPO adapter (seed 2)

A LoRA adapter for the second stage of a two-stage abstention regimen: supervised fine-tuning first to induce refusal behavior at all, then direct preference optimization (DPO) on top of it to refine where the refusal boundary sits. This is seed 2 of three.

The cold-start comparison that this extends found that preference training alone does not induce abstention on this model at this scale. The question this arm answers is whether preference training helps once SFT has already installed the behavior.

Status

Pre-registered extension. This adapter belongs to the sequential extension signed off on 2026-06-14 as a prospective addition to the locked matrix, recorded in section "Amendment A / v0.4 status" of the protocol and in the amendment governance note.

Its numbers are reported separately from the headline matrix and are never pooled with it. The protocol states the rule directly: mixed-stage results must be labeled as extension results unless a later signed revision explicitly supersedes the matrix.

Training

• Foundation model: unsloth/Qwen3-4B-bnb-4bit, loaded in 4-bit, maximum sequence length 2048.
• Stage 1: the same-seed headline SFT adapter (professorsynapse/eh-qwen3-4b-headline-sft-seed2-lora, revision 23ae0043bd794be8ede1122effd9ccfecb9d85aa), merged into a 16-bit model.
• Stage 2 (this adapter): direct preference optimization (DPO) trained on top of that merged 16-bit model.
• Seed: 2.
• LoRA: rank 32, alpha 64, dropout 0.05, applied to q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj.
• Optimization: learning rate 5e-6, 1 epoch, per-device batch 2, gradient accumulation 4. The materialized recipe carries no explicit beta override, so the run took the trainer default; PROTOCOL v0.3 section 3.1a registers beta 0.1 as the pre-registered default for this arm.
• Training file: dpo_train.jsonl, SHA-256 39e2ba8c9bc1b41ef1b7e797f80637c276ba150c97055962bbc4e2b550bd17b5 as recorded in the run record.

Training data

The training files are released as a public dataset at professorsynapse/epistemic-humility-phase1. All arms are built from one frozen question set: a known set and an unknown set of distinct source questions, split so that train and dev question keys are disjoint after normalizing question text. The per-method row expansion (one row per question for SFT, one chosen/rejected pair per question for DPO, several labeled rows per question for KTO) follows from each format rather than from a different budget.

Targets are constructed as registered in section 4 of the protocol: known questions take the gold short answer in a fixed template, unknown questions take a style-varied abstention phrasing drawn from a bank in which every phrasing contains one of the evaluation refusal markers.

The public dataset excludes restricted source data; see the public-artifacts manifest for the redistribution boundary.

Evaluation

How to read these numbers

The behavioral surface is SelfAware (Yin et al., 2023), a question set built to separate questions that have an answer from questions that do not: 3,369 rows per seed, 1,032 unknown-labeled and 2,337 known-labeled. Four metrics carry the result, all defined in section 3.4 of the manuscript:

• Refusal recall: percentage of unknown rows the model refused. Higher is better.
• Over-refusal: percentage of known rows the model refused. Lower is better.
• Correct-on-known: among known rows the model chose to answer, the percentage answered correctly. Its denominator is the answered subset, not all known rows.
• Truthful: percentage of all rows either correctly answered (known) or correctly refused (unknown).

This adapter (seed 2)

Metric	Value
Refusal recall	65.89%
Over-refusal	18.36%
Correct-on-known	25.84%
Truthful	34.82%

Source: amendment_a_selfaware_summary.csv, row seed2_dpo_clean / sft_dpo_seed2_lowmem.

Across the three seeds this arm averages refusal recall 52.81%, over-refusal 14.59%, truthfulness 31.18% (section 4.2 of the manuscript).

How to load

The stage-1 model this adapter was trained on is a local 16-bit merge that is not itself published. Rebuild it from the two published pieces, then apply this adapter on top. Merging a 4-bit base with its LoRA into 16-bit reproduces the training-time construction; it is not guaranteed to be bit-identical to the local artifact.

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

FOUNDATION = "unsloth/Qwen3-4B-bnb-4bit"
STAGE1 = "professorsynapse/eh-qwen3-4b-headline-sft-seed2-lora"
STAGE1_REVISION = "23ae0043bd794be8ede1122effd9ccfecb9d85aa"
ADAPTER = "professorsynapse/eh-qwen3-4b-seq-sft-dpo-seed2-lora"
REVISION = "62c2cf65d93509ee86bdedb257512f9055a4ff1a"

tokenizer = AutoTokenizer.from_pretrained(FOUNDATION)
base = AutoModelForCausalLM.from_pretrained(FOUNDATION, device_map="auto")

stage1 = PeftModel.from_pretrained(base, STAGE1, revision=STAGE1_REVISION)
stage1 = stage1.merge_and_unload()

model = PeftModel.from_pretrained(stage1, ADAPTER, revision=REVISION)

Intended use and limits

This is a research artifact for studying abstention and the refusal-recall against over-refusal trade-off. It is not a deployment-ready assistant.

• One model family at one scale (Qwen3-4B), one primary evaluation surface.
• Extension evidence, reported separately from the pre-registered headline matrix and not pooled with it.
• Loading requires reconstructing the stage-1 merge described above.

Provenance

• Revision this card describes: 62c2cf65d93509ee86bdedb257512f9055a4ff1a
• Local source run directory: synaptic-tuner/toolset-training-artifacts/runs/local/4b/sft_dpo__4b__amendment_a__seed2/20260616_164539/final_model
• Run record: sft_dpo__4b__amendment_a__seed2.json
• Registered protocol and extension sign-off: PROTOCOL v0.3
• Amendment governance: papers/common/amendment-governance.md
• Staging registry: docs/checkpoint-staging.md
• Release record: docs/public-artifacts.md
• Paper: Training regimen manuscript
• Project repository: https://github.com/ProfSynapse/Epistemic-Humility-Research

License

Apache-2.0, matching the unsloth/Qwen3-4B-bnb-4bit base model license recorded in the staging registry.

Citation

Cite the paper and the exact Hugging Face revision shown on this page.

@misc{rosenbaum2026abstention,
  title  = {Teaching Small Language Models to Say I Don't Know: A Controlled
            Comparison of SFT, DPO, KTO, and GRPO on Model-Specific Abstention Data},
  author = {Rosenbaum, Joseph},
  year   = {2026},
  note   = {Synaptic Labs},
  howpublished = {\url{https://github.com/ProfSynapse/Epistemic-Humility-Research}}
}