File size: 2,309 Bytes
6572764 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 | ---
license: apache-2.0
# adjusted: base_model pinned to upstream
# LiquidAI/LFM2.5-8B-A1B
tags:
- lfm2
- lora
- behzat-industries
- code
- sft
pipeline_tag: text-generation
---
# BehzatOne-8B-A1B (LoRA)
This is the **first model of Behzat Industries** — a code-focused SFT LoRA on top of [LiquidAI/LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B).
## What's in this repo
- `adapter_model.safetensors` — LoRA adapter (rank 32, alpha 64, target modules on attention + MLP projections). Attach to the base model with `peft`.
- `tokenizer.json` — LFM2.5 tokenizer.
- `adapter_config.json` — peft config.
## What's in Behzat
- Multi-source code SFT dataset (~109k samples) drawn from
open-thoughts/OpenThoughts, OpenThoughts3-1.2M,
iamtarun/python_code_instructions_18k_alpaca, glaiveai/glaive-function-calling-v2,
Agent-Ark/Toucan-1.5M and NousResearch/hermes-function-calling-v1.
- 2,000 optimizer steps, effective batch 16, lr 2e-4 linear decay,
packing disabled to avoid token-level cross-contamination with sdpa attention.
- Final mean token accuracy ~80% on the training slice, train loss ~0.85
(started ~1.0).
- Trained on a single Quadro RTX 6000 on Vast.ai ($0.17/hr).
## How to use
```python
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained(
"LiquidAI/LFM2.5-8B-A1B",
torch_dtype=torch.bfloat16,
device_map="auto",
)
model = PeftModel.from_pretrained(base, "rebehzat/BehzatOne-8B-A1B")
tok = AutoTokenizer.from_pretrained("rebehzat/BehzatOne-8B-A1B")
```
## Roadmap (Behzat Industries)
- v1.0 (this): SFT LoRA + tokenizer.
- v1.1: DPO on UltraFeedback binarized (single GPU).
- v1.2: drop merged BF16 weights for direct inference.
- v1.3: Q4_K_M GGUF runtime quant.
## Limits
- Base is 8B-param MoE with ~1B active params; quality ceiling is bounded by it.
- Trained with sdpa attention (no flash-attn was available on the host).
- Best on simple code completion; not a SWE-bench-grade coder.
## About Behzat Industries
Behzat Industries builds small, openly published coding models for hobbyists,
LLM-studio tinkerers and small-team dev tools. BehzatOne is the first of an
ongoing series. Buy nothing. Try everything. Tell us what fails.
|