AbhishekMallick/incident-triage-grpo-train

incident-triage-sft (LoRA adapter)

SFT-trained PEFT adapter for incident triage tasks on top of Qwen/Qwen2.5-1.5B-Instruct. The model is optimized to improve structured triage quality across diagnosis, policy, blast radius, and PR-quality heads.

Model Details

• Developed by: OpenEnv Hackathon project team
• Model type: Causal LM adapter (LoRA / PEFT)
• Base model: Qwen/Qwen2.5-1.5B-Instruct
• Language: English
• License: Apache-2.0 (inherits base model compatibility requirements)
• Frameworks: Transformers + PEFT

What this model is for

This adapter is designed for assistant-style incident triage:

• infer likely root cause from noisy multi-signal incident context
• recommend policy-level response actions
• estimate blast radius and urgency
• generate PR/remediation guidance in a concise format

Intended and out-of-scope use

Intended use

• internal incident simulation and benchmarking
• triage copilots for engineering on-call workflows
• structured response drafting where a human reviewer remains in-the-loop

Out-of-scope

• fully autonomous production incident response without human approval
• legal/compliance sign-off
• medical, financial, or safety-critical decision automation

How to run

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

base_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id = "YOUR_USERNAME/incident-triage-sft"

tokenizer = AutoTokenizer.from_pretrained(base_id, trust_remote_code=True)
base_model = AutoModelForCausalLM.from_pretrained(
    base_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)
model = PeftModel.from_pretrained(base_model, adapter_id)

prompt = "Incident: API latency spike after region failover. Provide diagnosis, policy, blast radius, and PR plan."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=256, temperature=0.2)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Training

Stage 1: SFT

• Started from Qwen/Qwen2.5-1.5B-Instruct
• Trained as PEFT adapter for incident-triage instructions
• SFT loss decreased from approximately 2.67 -> 1.81 (~32% reduction)

Stage 2 (separate): GRPO refinement

GRPO results are documented in this project, but this specific model card is for the SFT checkpoint upload.

Evaluation summary (held-out hard tasks)

Compared against baseline-hf:

• hard_multi_signal_cascade: 0.39 -> 0.71 (delta +0.32)
• expert_stealth_regression: 0.42 -> 0.83 (delta +0.40)
• hard_pr_quality_breach: 0.34 -> 0.60 (delta +0.26)

Average composite delta across these tasks: approximately +0.33.

Per-head behavior improved most on:

• blast radius scoring
• PR-quality scoring
• diagnosis improved moderately
• policy remained strong and near saturation

Limitations and risks

• still sensitive to prompt format and missing context
• diagnosis remains the main bottleneck versus oracle-ceiling references
• may produce over-confident recommendations under ambiguous telemetry
• should always be reviewed by an on-call engineer

Recommended Hugging Face metadata (UI fields)

Use these values in the model page metadata panel:

• base_model: Qwen/Qwen2.5-1.5B-Instruct
• pipeline_tag: text-generation
• library_name: peft
• license: apache-2.0
• language: en
• tags: incident-triage, sft, lora, peft, openenv
• datasets: your SFT training dataset identifier(s)
• metrics: composite_score, optionally per-head metrics (diagnosis, policy, blast, pr)

Model card contact

Open an issue in this repository/project for bug reports, misuse reports, or benchmark reproduction requests.