---
base_model: Qwen/Qwen3.8-27B
library_name: peft
pipeline_tag: image-text-to-text
license: other
language:
- en
- fr
tags:
- unsloth
- lora
- qlora
- cyber
- image-text-to-text
- vision
- video
- peft

model-index:
- name: Qwen3.8-27B-cyber — LoRA
  results:
  - task:
      type: text-generation
      name: Causal language modeling
    dataset:
      name: cyber SFT holdout
      type: dataset_cyber.jsonl
    metrics:
    - type: loss
      value: 0.741224
      name: eval_loss
---

# Qwen3.8-27B-cyber — LoRA

Instruction-tuned cybersecurity assistant (offensive, defensive, GRC, architecture, SOC/DFIR, RSSI).

| | |
|---|---|
| **Base model** | [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) |
| **Domain** | cyber |
| **Method** | LoRA / QLoRA (Unsloth) · rank 8 · α 16 |
| **Quantization at train** | bf16 LoRA |
| **Context** | 2048 tokens |
| **Dataset** | `dataset_cyber.jsonl` · train 57718 / eval 584 |
| **GPU** | NVIDIA RTX PRO 6000 Blackwell Server Edition (95.0 GiB) |
| **Wall time** | 53.0 min |
| **Modalities kept** | vision, video |

This checkpoint continues a strong general model and specialises it on a curated SFT corpus of cybersecurity procedures: pentest / red team, SOC and DFIR, cloud and identity, GRC (ISO, NIST, NIS2, DORA), and RSSI / project-management questions. Answers are meant to be concrete (controls, detections, hardening), not generic essays.


## What changed vs the reference

Reference = the published base checkpoint [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B), plus the first in-run loss (LoRA ≈ 0 at step 0).

| Metric | Reference (base / first log) | This fine-tune | Δ |
|---|---:|---:|---:|
| Train loss (first → last logged) | 2.8025 | 0.1832 | -93.5% |
| Train loss (best) | — | 0.6627 | — |
| Eval loss (holdout, first → last) | 0.9783 | 0.7412 | -24.2% |

The **first logged train loss** is the closest in-run proxy for the base model (LoRA starts near zero). Option F, when executed, adds an independent holdout comparison against the frozen merged base.


## Training data

- File: `dataset_cyber.jsonl`
- Path used at train time: `/content/drive/MyDrive/finetuning/dataset_cyber.jsonl`
- Split: `0.01` holdout, seed 42
- Format: chat-templated SFT (`messages` / `instruction`+`output` / `### Instruction` + `### Response`)

## Training procedure

| Hyperparameter | Value |
|---|---|
| Epochs | 1 |
| Learning rate | 0.0002 |
| Warmup ratio | 0.05 |
| Device batch | 4 |
| Grad accum | 2 |
| Effective batch | 8 |
| Optim | adamw_8bit |
| Packing | True |
| LoRA targets | `['q_proj', 'k_proj', 'v_proj', 'o_proj', 'out_proj', 'gate_proj', 'up_proj', 'down_proj']` |

## Intended use

Authorized defensive work, tabletop exercises, control design, detection engineering, audit readiness, and explaining attack techniques **without** weaponized payloads.

**Out of scope:** Do not use it to attack systems you do not own, to generate exploit payloads, or as a substitute for a licensed auditor or incident commander.

## Multimodal

Kept towers: **vision, video**. Vision/audio layers were frozen during text SFT (vision=False, audio=False). Load the merged Transformers folder (or GGUF + `mmproj`) to keep image / video / audio.


## How to use

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = 'Qwen/Qwen3.8-27B'
adapter = "nico248000000000/Qwen3.8-27B-cyber-LoRA"
tok = AutoTokenizer.from_pretrained(adapter, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    base, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,
)
model = PeftModel.from_pretrained(model, adapter)
```

## Limitations

- Domain shift: quality drops outside the SFT topics.
- Eval above is **holdout loss** (and optional targeted checks). It is not a public leaderboard.
- The base model license and acceptable-use policy still apply.

## License

`other` — inherit and respect the license of `Qwen/Qwen3.8-27B`.