Upload Qwen3.8-27B_merged_16bit

README.md

@@ -1,21 +1,121 @@
 ---
 base_model: Qwen/Qwen3.8-27B
-tags:
-- text-generation-inference
-- transformers
-- unsloth
-- qwen3_5
-license: apache-2.0
+library_name: transformers
+pipeline_tag: image-text-to-text
+license: other
 language:
 - en
+- fr
+tags:
+- unsloth
+- lora
+- qlora
+- cyber
+- image-text-to-text
+- vision
+- video
+
+model-index:
+- name: Qwen3.8-27B-cyber
+  results:
+  - task:
+      type: text-generation
+      name: Causal language modeling
+    dataset:
+      name: cyber SFT holdout
+      type: dataset_cyber.jsonl
+    metrics:
+    - type: loss
+      value: 0.741224
+      name: eval_loss
 ---
 
-# Uploaded finetuned  model
+# Qwen3.8-27B-cyber
+
+Instruction-tuned cybersecurity assistant (offensive, defensive, GRC, architecture, SOC/DFIR, RSSI).
+
+| | |
+|---|---|
+| **Base model** | [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B) |
+| **Domain** | cyber |
+| **Method** | LoRA / QLoRA (Unsloth) · rank 8 · α 16 |
+| **Quantization at train** | bf16 LoRA |
+| **Context** | 2048 tokens |
+| **Dataset** | `dataset_cyber.jsonl` · train 57718 / eval 584 |
+| **GPU** | NVIDIA RTX PRO 6000 Blackwell Server Edition (95.0 GiB) |
+| **Wall time** | 53.0 min |
+| **Modalities kept** | vision, video |
+
+This checkpoint continues a strong general model and specialises it on a curated SFT corpus of cybersecurity procedures: pentest / red team, SOC and DFIR, cloud and identity, GRC (ISO, NIST, NIS2, DORA), and RSSI / project-management questions. Answers are meant to be concrete (controls, detections, hardening), not generic essays.
+
+
+## What changed vs the reference
+
+Reference = the published base checkpoint [`Qwen/Qwen3.8-27B`](https://huggingface.co/Qwen/Qwen3.8-27B), plus the first in-run loss (LoRA ≈ 0 at step 0).
+
+| Metric | Reference (base / first log) | This fine-tune | Δ |
+|---|---:|---:|---:|
+| Train loss (first → last logged) | 2.8025 | 0.1832 | -93.5% |
+| Train loss (best) | — | 0.6627 | — |
+| Eval loss (holdout, first → last) | 0.9783 | 0.7412 | -24.2% |
+
+The **first logged train loss** is the closest in-run proxy for the base model (LoRA starts near zero). Option F, when executed, adds an independent holdout comparison against the frozen merged base.
+
+
+## Training data
+
+- File: `dataset_cyber.jsonl`
+- Path used at train time: `/content/drive/MyDrive/finetuning/dataset_cyber.jsonl`
+- Split: `0.01` holdout, seed 42
+- Format: chat-templated SFT (`messages` / `instruction`+`output` / `### Instruction` + `### Response`)
+
+## Training procedure
+
+| Hyperparameter | Value |
+|---|---|
+| Epochs | 1 |
+| Learning rate | 0.0002 |
+| Warmup ratio | 0.05 |
+| Device batch | 4 |
+| Grad accum | 2 |
+| Effective batch | 8 |
+| Optim | adamw_8bit |
+| Packing | True |
+| LoRA targets | `['q_proj', 'k_proj', 'v_proj', 'o_proj', 'out_proj', 'gate_proj', 'up_proj', 'down_proj']` |
+
+## Intended use
+
+Authorized defensive work, tabletop exercises, control design, detection engineering, audit readiness, and explaining attack techniques **without** weaponized payloads.
+
+**Out of scope:** Do not use it to attack systems you do not own, to generate exploit payloads, or as a substitute for a licensed auditor or incident commander.
+
+## Multimodal
+
+Kept towers: **vision, video**. Vision/audio layers were frozen during text SFT (vision=False, audio=False). Load the merged Transformers folder (or GGUF + `mmproj`) to keep image / video / audio.
+
+
+## How to use
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+import torch
+
+repo = "nico248000000000/Qwen3.8-27B-cyber"
+tok = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    repo, torch_dtype=torch.bfloat16, device_map="auto", trust_remote_code=True,
+)
+messages = [{"role": "user", "content": 'How do you design a reliable Zero Trust infrastructure? Answer with a concrete control list.'}]
+inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
+print(tok.decode(model.generate(inputs, max_new_tokens=256)[0], skip_special_tokens=True))
+```
+
+## Limitations
 
-- **Developed by:** nico248000000000
-- **License:** apache-2.0
-- **Finetuned from model :** Qwen/Qwen3.8-27B
+- Domain shift: quality drops outside the SFT topics.
+- Eval above is **holdout loss** (and optional targeted checks). It is not a public leaderboard.
+- The base model license and acceptable-use policy still apply.
 
-This qwen3_5 model was trained 2x faster with [Unsloth](https://github.com/unslothai/unsloth) and Huggingface's TRL library.
+## License
 
-[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>](https://github.com/unslothai/unsloth)
+`other` — inherit and respect the license of `Qwen/Qwen3.8-27B`.

config.json

@@ -2,9 +2,7 @@
     "architectures": [
         "Qwen3_5ForConditionalGeneration"
     ],
-    "bos_token_id": null,
     "torch_dtype": "bfloat16",
-    "eos_token_id": 248046,
     "image_token_id": 248056,
     "language_model_only": false,
     "model_name": "Qwen/Qwen3.8-27B",
@@ -123,7 +121,6 @@
     },
     "tie_word_embeddings": false,
     "unsloth_version": "2026.8.18",
-    "use_cache": false,
     "video_token_id": 248057,
     "vision_config": {
         "deepstack_visual_indexes": [],

generation_config.json

@@ -1,7 +1,7 @@
 {
+  "bos_token_id": 248044,
   "do_sample": true,
   "eos_token_id": [
-    248046,
     248046,
     248044
   ],

model-00001-of-00018.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:ad9559c2657909f8fbc572968eb70876b492463c852ab42418af5a4e9b41b566
+oid sha256:ba0ce20aae489ad196733da5064bcdf159a1fe84f53336648196e1ebb7751b1c
 size 3966730552

model-00002-of-00018.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:625ecb1fc2f1a31645f4058b28c801ea8f0bc857221f66924e56297e06bfd306
+oid sha256:06a148c01bfbe3faa14a5f184a7ff29a706f7ae1c8b2705d2058e26d17a001fb
 size 3043080328

model-00004-of-00018.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:5144613f85e0891598f358c11ef7683959d75f45712ead4ecc39493242529b58
+oid sha256:511e34063187882659753c4d93f3859f93c019fd438d8813071921c81d9a3f1a
 size 3988973152

model-00005-of-00018.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:8af79f78494386beac8071441ffef0fad0c6b720598602953651235f0d68e52b
+oid sha256:635cb53446dc74f219740fc59e18b774f877b803b9722e289ca62575a6efa701
 size 2099339864

model-00006-of-00018.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:c2075046094cf0be65a59d3a3ea90fc43f9feba91df3eac4b3acaeadf67f04cc
+oid sha256:0bc5214fac607f0e6cc92eec3789d4b8559410ef9fce66621ba8158e8410dae0
 size 3979553696

model-00008-of-00018.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d934f4b1ff850631a392425347e671682e490e7114239b9b075853241cefd73a
+oid sha256:7192c5b66185d3592927daabee1cc19e6f6e0ce75988ee20e824b624765fda79
 size 3979553696

tokenizer_config.json

@@ -21,7 +21,7 @@
     "vision_eos_token": "<|vision_end|>"
   },
   "pad_token": "<|endoftext|>",
-  "padding_side": "right",
+  "padding_side": "left",
   "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
   "processor_class": "Qwen3VLProcessor",
   "split_special_tokens": false,
@@ -295,6 +295,5 @@
       "normalized": false,
       "special": true
     }
-  },
-  "chat_template": "{%- set image_count = namespace(value=0) %}\n{%- set video_count = namespace(value=0) %}\n{%- macro render_content(content, do_vision_count, is_system_content=false) %}\n    {%- if content is string %}\n        {{- content }}\n    {%- elif content is iterable and content is not mapping %}\n        {%- for item in content %}\n            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}\n                {%- if is_system_content %}\n                    {{- raise_exception('System message cannot contain images.') }}\n                {%- endif %}\n                {%- if do_vision_count %}\n                    {%- set image_count.value = image_count.value + 1 %}\n                {%- endif %}\n                {%- if add_vision_id %}\n                    {{- 'Picture ' ~ image_count.value ~ ': ' }}\n                {%- endif %}\n                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}\n            {%- elif 'video' in item or item.type == 'video' %}\n                {%- if is_system_content %}\n                    {{- raise_exception('System message cannot contain videos.') }}\n                {%- endif %}\n                {%- if do_vision_count %}\n                    {%- set video_count.value = video_count.value + 1 %}\n                {%- endif %}\n                {%- if add_vision_id %}\n                    {{- 'Video ' ~ video_count.value ~ ': ' }}\n                {%- endif %}\n                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}\n            {%- elif 'text' in item %}\n                {{- item.text }}\n            {%- else %}\n                {{- raise_exception('Unexpected item type in content.') }}\n            {%- endif %}\n        {%- endfor %}\n    {%- elif content is none or content is undefined %}\n        {{- '' }}\n    {%- else %}\n        {{- raise_exception('Unexpected content type.') }}\n    {%- endif %}\n{%- endmacro %}\n{%- if not messages %}\n    {{- raise_exception('No messages provided.') }}\n{%- endif %}\n{%- set reasoning_instructions = '' %}\n{%- if enable_thinking is undefined or enable_thinking is true %}\n    {%- set resolved_reasoning_effort = reasoning_effort|default('xhigh') %}\n    {%- if resolved_reasoning_effort not in ('xhigh', 'medium', 'low') %}\n        {{- raise_exception('Unexpected reasoning effort ' ~ reasoning_effort ~ '. Supported types are xhigh (default), medium, and low.') }}\n    {%- endif %}\n    {%- if resolved_reasoning_effort == 'xhigh' %}\n        {%- set reasoning_instructions = 'Reasoning effort is set to xhigh. Please think carefully through the task, validate key assumptions, consider plausible alternatives, and prioritize correctness, consistency, and clarity in the final answer.' %}\n    {%- elif resolved_reasoning_effort == 'low' %}\n        {%- set reasoning_instructions = 'Reasoning effort is set to low. Keep your thinking brief and focused, moving directly to the conclusion without unnecessary elaboration.' %}\n    {%- endif %}\n{%- endif %}\n{%- if tools and tools is iterable and tools is not mapping %}\n    {{- '<|im_start|>system\\n' }}\n    {%- if reasoning_instructions %}\n        {{- reasoning_instructions + '\\n\\n' }}\n    {%- endif %}\n    {{- \"# Tools\\n\\nYou have access to the following functions:\\n\\n<tools>\" }}\n    {%- for tool in tools %}\n        {{- \"\\n\" }}\n        {{- tool | tojson }}\n    {%- endfor %}\n    {{- \"\\n</tools>\" }}\n    {{- '\\n\\nIf you choose to call a function ONLY reply in the following format with NO suffix:\\n\\n<tool_call>\\n<function=example_function_name>\\n<parameter=example_parameter_1>\\nvalue_1\\n</parameter>\\n<parameter=example_parameter_2>\\nThis is the value for the second parameter\\nthat can span\\nmultiple lines\\n</parameter>\\n</function>\\n</tool_call>\\n\\n<IMPORTANT>\\nReminder:\\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\\n- Required parameters MUST be specified\\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\\n</IMPORTANT>' }}\n    {%- if messages[0].role == 'system' %}\n        {%- set content = render_content(messages[0].content, false, true)|trim %}\n        {%- if content %}\n            {{- '\\n\\n' + content }}\n        {%- endif %}\n    {%- endif %}\n    {{- '<|im_end|>\\n' }}\n{%- else %}\n    {%- if messages[0].role == 'system' %}\n        {%- set content = render_content(messages[0].content, false, true)|trim %}\n        {%- if content %}\n            {{- '<|im_start|>system\\n' + (reasoning_instructions + '\\n\\n' if reasoning_instructions else '')  + content + '<|im_end|>\\n' }}\n        {%- elif reasoning_instructions %}\n            {{- '<|im_start|>system\\n' + reasoning_instructions + '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- elif reasoning_instructions %}\n        {{- '<|im_start|>system\\n' + reasoning_instructions + '<|im_end|>\\n' }}\n    {%- endif %}\n{%- endif %}\n{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}\n{%- for message in messages[::-1] %}\n    {%- set index = (messages|length - 1) - loop.index0 %}\n    {%- if ns.multi_step_tool and message.role == \"user\" %}\n        {%- set content = render_content(message.content, false)|trim %}\n        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}\n            {%- set ns.multi_step_tool = false %}\n            {%- set ns.last_query_index = index %}\n        {%- endif %}\n    {%- endif %}\n{%- endfor %}\n{%- if ns.multi_step_tool %}\n    {{- raise_exception('No user query found in messages.') }}\n{%- endif %}\n{%- for message in messages %}\n    {%- set content = render_content(message.content, true)|trim %}\n    {%- if message.role == \"system\" %}\n        {%- if not loop.first %}\n            {{- raise_exception('System message must be at the beginning.') }}\n        {%- endif %}\n    {%- elif message.role == \"user\" %}\n        {{- '<|im_start|>' + message.role + '\\n' + content + '<|im_end|>' + '\\n' }}\n    {%- elif message.role == \"assistant\" %}\n        {%- set reasoning_content = '' %}\n        {%- if message.reasoning_content is string %}\n            {%- set reasoning_content = message.reasoning_content %}\n        {%- endif %}\n        {%- set reasoning_content = reasoning_content|trim %}\n        {%- if preserve_thinking is undefined or preserve_thinking is true or loop.index0 > ns.last_query_index %}\n            {{- '<|im_start|>' + message.role + '\\n<think>\\n' + reasoning_content + '\\n</think>\\n\\n' + content }}\n        {%- else %}\n            {{- '<|im_start|>' + message.role + '\\n' + content }}\n        {%- endif %}\n        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}\n            {%- for tool_call in message.tool_calls %}\n                {%- if tool_call.function is defined %}\n                    {%- set tool_call = tool_call.function %}\n                {%- endif %}\n                {%- if loop.first %}\n                    {%- if content|trim %}\n                        {{- '\\n\\n<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n                    {%- else %}\n                        {{- '<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n                    {%- endif %}\n                {%- else %}\n                    {{- '\\n<tool_call>\\n<function=' + tool_call.name + '>\\n' }}\n                {%- endif %}\n                {%- if tool_call.arguments is defined and tool_call.arguments != '' %}\n                    {%- for args_name, args_value in tool_call.arguments|items %}\n                        {{- '<parameter=' + args_name + '>\\n' }}\n                        {%- set args_value = args_value | string if args_value is string else args_value | tojson | safe %}\n                        {{- args_value }}\n                        {{- '\\n</parameter>\\n' }}\n                    {%- endfor %}\n                {%- endif %}\n                {{- '</function>\\n</tool_call>' }}\n            {%- endfor %}\n        {%- endif %}\n        {{- '<|im_end|>\\n' }}\n    {%- elif message.role == \"tool\" %}\n        {%- if loop.previtem and loop.previtem.role != \"tool\" %}\n            {{- '<|im_start|>user' }}\n        {%- endif %}\n        {{- '\\n<tool_response>\\n' }}\n        {{- content }}\n        {{- '\\n</tool_response>' }}\n        {%- if not loop.last and loop.nextitem.role != \"tool\" %}\n            {{- '<|im_end|>\\n' }}\n        {%- elif loop.last %}\n            {{- '<|im_end|>\\n' }}\n        {%- endif %}\n    {%- else %}\n        {{- raise_exception('Unexpected message role.') }}\n    {%- endif %}\n{%- endfor %}\n{%- if add_generation_prompt %}\n    {{- '<|im_start|>assistant\\n' }}\n    {%- if enable_thinking is defined and enable_thinking is false %}\n        {{- '<think>\\n\\n</think>\\n\\n' }}\n    {%- else %}\n        {{- '<think>\\n' }}\n    {%- endif %}\n{%- endif %}"
-}
+  }
+}