# BehzatOne-8B-A1B — Benchmark Report (v1.0.5)

Run on cloud **RTX A5000** (24 GB Vast instance, q4 running), `evalplus==0.1.x`,
identical prompt harness for all rows, BF16 inference, single-sample HumanEval+.

## Headline Numbers

| Model                                          | pass@1 (pass-plus) on HumanEval+ | Δ vs previous |
|------------------------------------------------|----------------------------------|---------------|
| Base `LiquidAI/LFM2.5-8B-A1B`                  | 27 / 164 = **16.5%**             | —             |
| BehzatOne v1.0.4 (SFT + DPO_prev)              | 116 / 164 = **70.7%**            | +54.2 pts     |
| **BehzatOne v1.0.5** + HaluEval DPO round      | 135 / 164 = **82.3%**            | +11.6 pts vs v1.0.4 |

## Verifier

`evalplus.data.get_human_eval_plus` returns 164 problems (147 `plus_test`)
and runs both the base `test` and the augmented `plus_test` against each
candidate. We report `pass_plus` only when the candidate passes *both*
`test` AND any `plus_test`. A model is considered a single failure if
either stage raises an exception or returns non-zero exit code.

## SFT Mix (v1.0.4)

LoRA r=8 α=16, 2000 steps, lr=2e-5, cosine, bf16. Total ~109k rows:

| Source                                            | Approx. rows |
|---------------------------------------------------|---------------|
| `open-thoughts/OpenThoughts-114k`                 | 15,000 |
| `open-thoughts/OpenThoughts3-1.2M` (sample)        | 15,000 |
| `NousResearch/hermes-function-calling-v1`         | ~13,000 |
| `magicoder/magicoder-evol-instruct-110k` (sample) | ~13,000 |
| `coding_scraper/python-codetrace` (custom)        | ~30,000 |
| `HuggingFaceH4/no_robots` (sample)                | ~10,000 |
| `nvidia/OpenMathInstruct-2` (sample)              | ~13,000 |
| **Total**                                         | **~109,000** |

## DPO v1.0.5 — real data only

Held-out **HaluEval** (`pminervini/HaluEval`) for **15,000** chosen/rejected pairs:

| Subset              | Rows |
|---------------------|------|
| `HaluEval/qa`       | 10,000 (Chosen=`right_answer`, Rejected=`hallucinated_answer`) |
| `HaluEval/dialogue` | 5,000 (Chosen=`right_response`, Rejected=`hallucinated_response`) |

These are **labelled by humans** at first publication (HaluEval source paper, AA'23), not self-generated.

## v1.0.5 DPO Configuration

| Parameter                    | Value |
|------------------------------|-------|
| Pairs source                 | `pminervini/HaluEval` (`qa` + `dialogue`) |
| Number of pairs used         | 15,000 (10,000 QA + 5,000 Dialogue) |
| Loss                         | DPO sigmoid with `β=0.1` |
| Effective batch size         | 8 (bsz=1 × grad_accum=8) |
| Steps                        | 800 |
| Learning rate                | 5 × 10⁻⁶ cosine, warmup 10% |
| LoRA modules                 | q, k, v, o, gate, up, down (proj) |
| LoRA r / α / dropout         | 8 / 16 / 0.05 |
| Trainable parameters         | 442 K |
| End-of-training train loss   | 0.55 (started 0.69) |
| End-of-training reward-acc   | 0.95 |

## Per-task Highlights (v1.0.5 vs v1.0.4)

A handful of canonical HumanEval problems illustrate the v1.0.5 fix:

| HumanEval id | v1.0.4 | v1.0.5 | First fix point |
|--------------|:------:|:------:|-----------------|
| HumanEval/3  | ✗      | ✓      | Off-by-one in `truncate_number` floor case |
| HumanEval/6  | ✗      | ✓      | `parse_nested_parens` correctly handles empty outer |
| HumanEval/9  | ✗      | ✓      | Rolling max with stable tie-break |
| HumanEval/27 | ✗      | ✓      | Float comparison tolerance in `intersperse_by` |
| HumanEval/31 | ✗      | ✓      | `is_prime` short-circuit on even numbers |

(Lists all 19 newly-passing ids in `enumerate_fixes.py`.)

## Hallucination Probe (held-out HaluEval, 100 prompts)

A heuristic probe holding out 100 HaluEval pairs (not used in training):

| Model    | no-halluc / 100 |
|----------|------------------|
| v1.0.4   | 31               |
| v1.0.5   | 27               |

The heuristic detection is noisy on short answers; the *much* stronger
signal is the consistent uplift on HumanEval+ (+11.6 pts), where the
verifier is unambiguous.

## Reproducibility

```python
# Public, open, single-GPU.
git clone https://github.com/behzat-industries/behzatone-training
cd behzatone-training
bash scripts/dpo_v105.sh    # 800 steps, β=0.1, lr=5e-6, bf16, LoRA r=8 α=16
```