README.md

---

base_model: Qwen/Qwen2.5-Coder-1.5B-Instruct
base_model_relation: adapter
new_version: joshuasundance/mypo-qwen2.5-coder-1.5b-dpo-v3
datasets:
- joshuasundance/mypo-4k-rfc
language:
- en
- code
library_name: peft
license: apache-2.0
pipeline_tag: text-generation
model_name: mypo-qwen2.5-coder-1.5b-dpo-v2
tags:
- generated_from_trainer
- dpo
- trl
- preference-optimization
- lora
- peft
- python
- type-hints
- code
- qwen2.5-coder
- mypo
- hf_jobs
- codecarbon
- carbon-emissions
co2_eq_emissions:
  emissions: 238.0
  source: "CodeCarbon v3.2.6 (measured)"
  training_type: "fine-tuning"
  geographical_location: "Virginia, USA (AWS us-east-1)"
  hardware_used: "1 x NVIDIA A10G (HF Jobs a10g-large)"
model-index:
- name: mypo-qwen2.5-coder-1.5b-dpo-v2
  results:
  - task:
      type: text-generation
      name: Python type-hinted code generation
    dataset:
      name: mypo-4k-rfc
      type: joshuasundance/mypo-4k-rfc
      split: validation
    metrics:
    - type: pass_rate
      name: parse rate
      value: 0.973
    - type: pass_rate
      name: black pass rate
      value: 0.120
    - type: pass_rate
      name: ruff pass rate
      value: 0.940
    - type: pass_rate
      name: mypy --strict pass rate
      value: 0.060
    - type: coverage
      name: annotation slot coverage
      value: 0.000
    - type: win_rate
      name: preference win-rate vs gold (chosen)
      value: 0.000
  - task:
      type: text-generation
      name: Python code generation
    dataset:
      name: HumanEval+
      type: humaneval-plus
      split: test
    metrics:
    - type: pass_rate
      name: pass@1 (base tests)
      value: 0.6707317073170732
    - type: pass_rate
      name: pass@1 (plus tests)
      value: 0.5914634146341463
---


# Model Card for mypo-qwen2.5-coder-1.5b-dpo-v2

> **⚠️ Superseded by [`mypo-qwen2.5-coder-1.5b-dpo-v3`](https://huggingface.co/joshuasundance/mypo-qwen2.5-coder-1.5b-dpo-v3).** v3 fixes the v2 no-op issue (matched LoRA α=r=256, lr=5e-5, β=0.3, warm-started from SFT, published as a merged model). This v2 adapter is preserved as a reproducibility artifact and case study in how DPO training telemetry can look healthy while generations remain unchanged.

DPO LoRA adapter for [`Qwen/Qwen2.5-Coder-1.5B-Instruct`](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct), trained on [`joshuasundance/mypo-4k-rfc`](https://huggingface.co/datasets/joshuasundance/mypo-4k-rfc), where `chosen` = type-hinted Python and `rejected` = unhinted Python. v2 uses a higher LoRA rank and all-linear targeting compared to v1.

This is a **PEFT/LoRA adapter**, not a merged model.

## TL;DR — honest assessment

Training telemetry looked great (`rewards/accuracies → 1.0`, `rewards/margins → ~0.44`), but post-hoc characterization shows **this adapter barely shifts generation behavior from the base model** under greedy decoding. Two measurements confirm the no-op under different decoding regimes:

**Batched eval (n=150 stratified held-out validation prompts, run `2026-04-22-qwen2.5-1.5b`):**

| metric | base | **this adapter (dpo-v2)** | SFT adapter | gold (`chosen`) |
| --- | --- | --- | --- | --- |
| parse rate | 97.3 % | **97.3 %** | 100 % | 100 % |
| `ruff` pass rate | 93.3 % | **94.0 %** | 96.0 % | 100 % |
| `black` pass rate | 12.0 % | **12.0 %** | 97.3 % | 98.0 % |
| `mypy --strict` pass rate | 6.0 % | **6.0 %** | 92.7 % | 100 % |
| mean annotation slot coverage | 0.000 | **0.000** | 0.953 | 0.955 |
| mean `ruff` violations / sample | 0.47 | **0.46** | 0.07 | 0.00 |
| mean `mypy` errors / sample | 2.30 | **2.35** | 0.13 | 0.00 |
| preference win rate vs `chosen` | — | **0 %** | 49.0 % | — |
| preference win rate vs base | — | **50 %** (tie) | 100 % | — |

**Single-prompt validation (n=30, batch=1, no padding):**

| metric | base | **this adapter (dpo-v2)** | SFT adapter |
| --- | --- | --- | --- |
| `mypy --strict` pass rate | 0.0 % | **0.0 %** | 73.3 % |
| mean annotation slot coverage | 0.000 | **0.000** | 0.971 |
| `black` pass rate | 6.7 % | **6.7 %** | 100 % |

v2 is indistinguishable from base under **both** decoding regimes — batched and single-prompt — confirming the no-op is not an artifact of batching, tokenization, or adapter-loading. The adapter is active (outputs differ from base in non-annotation ways), but the policy shift in annotation dimensions is effectively zero.

### Why the training metrics look strong but generation doesn't change

DPO optimizes a ranking objective on `log p(chosen) − log p(rejected)`. Achieving `rewards/accuracies = 1.0` only requires the log-ratio to tilt in the right direction — it can be satisfied by infinitesimal weight movements when the LoRA scaling factor is small.

Here the effective LoRA scale is `alpha / r = 16 / 256 = 0.0625`, and the DPO learning rate is `1e-6` (200× lower than the SFT recipe's `2e-4`). The product — tiny deltas × tiny scale × small lr — left the adapter weights too small to change argmax generations under greedy decoding, even though they were enough to move relative log-probabilities.

For a **dpo-v3**, either raise `lora_alpha` to match `r` (→ `alpha=256`, scale=1.0) or switch to `rslora=True`, and increase `lr` to `1e-5`–`5e-5`. Those are documented remedies in the "LoRA without regret" literature.

Full characterization is published in [`joshuasundance/mypo-training`](https://huggingface.co/joshuasundance/mypo-training/tree/main/reports/2026-04-22-qwen2.5-1.5b).

External benchmark note: on the latest canonical full HumanEval+ run (n=164),
this adapter reaches **110 / 164 = 67.1 %** pass@1 on base tests and
**97 / 164 = 59.1 %** on plus tests, versus the untuned Qwen base at
`112 / 164` and `99 / 164`. So v2 remains close to base out of domain, but it
still does not beat the base model on general HumanEval+ correctness.

## Quick start

```python

import torch

from peft import PeftModel

from transformers import AutoModelForCausalLM, AutoTokenizer



base_id = "Qwen/Qwen2.5-Coder-1.5B-Instruct"

adapter_id = "joshuasundance/mypo-qwen2.5-coder-1.5b-dpo-v2"



tokenizer = AutoTokenizer.from_pretrained(base_id)

base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16, device_map="auto")

model = PeftModel.from_pretrained(base, adapter_id)



messages = [{"role": "user", "content": "Write a typed Python function that returns the nth Fibonacci number."}]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)

out = model.generate(inputs, max_new_tokens=256, do_sample=False)

print(tokenizer.decode(out[0, inputs.shape[-1]:], skip_special_tokens=True))

```

## Training

Trained with [TRL](https://github.com/huggingface/trl) `DPOTrainer` on a single NVIDIA A10G via Hugging Face Jobs. Training script: [`mypo_dpo_train_v2.py`](https://huggingface.co/joshuasundance/mypo-training/blob/main/mypo_dpo_train_v2.py). Job id: `69e8a5b0d2fd2eb837d766b1`.

### Data

- **Dataset:** `joshuasundance/mypo-4k-rfc` (train + validation concatenated → 6361 preference pairs).
- **Format:** native DPO `(prompt, chosen, rejected)`.

### Hyperparameters

| Group | Setting |
| --- | --- |
| Quantization | 4-bit NF4 + double-quant, `bnb_4bit_compute_dtype=bfloat16` |
| LoRA | `r=256`, `alpha=16`, `dropout=0.05`, `target_modules="all-linear"`, `task_type=CAUSAL_LM` |
| Optimization | `paged_adamw_8bit`, `lr=1e-6`, cosine schedule, `warmup_steps=100` |
| Batching | `per_device_train_batch_size=1`, `gradient_accumulation_steps=8` (effective 8) |
| DPO | `beta=0.1`, `loss_type="sigmoid"`, reference adapter toggled off in-place |
| Schedule | 3 epochs, `max_length=2048` |
| Precision | bf16, gradient checkpointing on |
| Seed | 42 |

### Final training metrics (from job logs)

| Metric | Value |
| --- | --- |
| `train_runtime` | 10938.4 s (~3h 02m) |
| `train_loss` | 0.5153 |
| `rewards/accuracies` (final step) | 1.0 |
| `rewards/margins` (final step) | ~0.44 |
| `rewards/chosen` (final step) | ~0.12 |
| `rewards/rejected` (final step) | ~-0.32 |
| `mean_token_accuracy` (final step) | 0.8214 |

## Environmental impact

CodeCarbon telemetry from the training run (`emissions.csv` in this repo):

| Metric | Value |
| --- | --- |
| Duration | 10938.4 s |
| Energy consumed | 0.6456 kWh |
| CO₂e emissions | **0.2383 kg** |
| GPU energy / power | 0.4306 kWh / 141.8 W avg |
| CPU energy / power | 0.0565 kWh / 19.2 W avg |
| RAM energy / power | 0.1585 kWh / 54.0 W |
| Hardware | 1 × NVIDIA A10G, AMD EPYC 7R32 (48 vCPU), 187 GB RAM |
| Region | AWS `us-east-1` (Virginia), PUE 1.0 |
| Tracker | codecarbon 3.2.6, `tracking_mode=machine` |

## Framework versions

- TRL 1.2.0, Transformers 5.5.4, PyTorch 2.11.0, Datasets 4.8.4, Tokenizers 0.22.2, PEFT ≥ 0.12, bitsandbytes ≥ 0.44.

## License

Apache 2.0 (inherits from the [Qwen2.5-Coder-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-1.5B-Instruct) base model).

## Citations

### CodeCarbon (emissions tracking)

Emissions for this run were measured with [CodeCarbon](https://codecarbon.io/) v3.2.6.

```bibtex

@software{codecarbon,

  author  = {Benoit Courty and Victor Schmidt and Sasha Luccioni and Goyal-Kamal and MarionCoutarel and Boris Feld and Jérémy Lecourt and LiamConnell and Amine Saboni and Inimaz and supatomic and Mathilde Léval and Luis Blanche and Alexis Cruveiller and Ouminasara and Franklin Zhao and Aditya Joshi and Alexis Bogroff and Hugues de Lavoreille and Niko Laskaris and Edoardo Abati and Douglas Blank and Ziyao Wang and Armin Catovic and Marc Alencon and Michał Stęchły and Christian Bauer and Lucas Otávio N. de Araújo and JPW and MinervaBooks},

  title   = {{CodeCarbon: Estimate and track carbon emissions from machine learning computing}},

  year    = 2024,

  doi     = {10.5281/zenodo.11171501},

  url     = {https://github.com/mlco2/codecarbon}

}

```

### DPO

```bibtex

@inproceedings{rafailov2023direct,

    title        = {{Direct Preference Optimization: Your Language Model is Secretly a Reward Model}},

    author       = {Rafael Rafailov and Archit Sharma and Eric Mitchell and Christopher D. Manning and Stefano Ermon and Chelsea Finn},

    year         = 2023,

    booktitle    = {Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023}

}

```

### TRL

```bibtex

@software{vonwerra2020trl,

  title   = {{TRL: Transformer Reinforcement Learning}},

  author  = {von Werra, Leandro and Belkada, Younes and Tunstall, Lewis and Beeching, Edward and Thrush, Tristan and Lambert, Nathan and Huang, Shengyi and Rasul, Kashif and Gallouédec, Quentin},

  license = {Apache-2.0},

  url     = {https://github.com/huggingface/trl},

  year    = 2020

}

```