---
license: apache-2.0
# adjusted: base_model pinned to upstream
# LiquidAI/LFM2.5-8B-A1B
tags:
  - lfm2
  - lora
  - behzat-industries
  - code
  - sft
pipeline_tag: text-generation
---

# BehzatOne-8B-A1B (LoRA)

This is the **first model of Behzat Industries** — a code-focused SFT LoRA on top of [LiquidAI/LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B).

## What's in this repo

- `adapter_model.safetensors` — LoRA adapter (rank 32, alpha 64, target modules on attention + MLP projections). Attach to the base model with `peft`.
- `tokenizer.json` — LFM2.5 tokenizer.
- `adapter_config.json` — peft config.

## What's in Behzat

- Multi-source code SFT dataset (~109k samples) drawn from
  open-thoughts/OpenThoughts, OpenThoughts3-1.2M,
  iamtarun/python_code_instructions_18k_alpaca, glaiveai/glaive-function-calling-v2,
  Agent-Ark/Toucan-1.5M and NousResearch/hermes-function-calling-v1.
- 2,000 optimizer steps, effective batch 16, lr 2e-4 linear decay,
  packing disabled to avoid token-level cross-contamination with sdpa attention.
- Final mean token accuracy ~80% on the training slice, train loss ~0.85
  (started ~1.0).
- Trained on a single Quadro RTX 6000 on Vast.ai ($0.17/hr).

## How to use

```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained(
    "LiquidAI/LFM2.5-8B-A1B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(base, "rebehzat/BehzatOne-8B-A1B")
tok = AutoTokenizer.from_pretrained("rebehzat/BehzatOne-8B-A1B")
```

## Roadmap (Behzat Industries)

- v1.0 (this): SFT LoRA + tokenizer.
- v1.1: DPO on UltraFeedback binarized (single GPU).
- v1.2: drop merged BF16 weights for direct inference.
- v1.3: Q4_K_M GGUF runtime quant.

## Limits

- Base is 8B-param MoE with ~1B active params; quality ceiling is bounded by it.
- Trained with sdpa attention (no flash-attn was available on the host).
- Best on simple code completion; not a SWE-bench-grade coder.

## About Behzat Industries

Behzat Industries builds small, openly published coding models for hobbyists,
LLM-studio tinkerers and small-team dev tools. BehzatOne is the first of an
ongoing series. Buy nothing. Try everything. Tell us what fails.