---
license: apache-2.0
base_model: Qwen/Qwen2.5-0.5B-Instruct
tags:
  - text-classification
  - intent-classification
  - routing
  - lora
  - peft
  - agent-router
  - small-language-model
language:
  - en
pipeline_tag: text-classification
---

# Qwen2.5-0.5B Intent Router (LoRA fine-tuned)

A 500M-parameter sequence classifier fine-tuned to route user queries to one of
four tool categories for an internal GenAI assistant, achieving **95.3% accuracy**
and **3.3x lower latency** than a 70B-parameter LLM baseline on the same task.

## Model Description

This model classifies a user query into exactly one of four categories, intended
to sit at the entry point of an AI agent and decide which downstream tool/node
should handle the request:

| Category | Description |
|---|---|
| `rag_tool` | Query needs internal company knowledge (HR policy, internal docs, onboarding, internal tools) |
| `code_tool` | Query needs code written, debugged, explained, or reviewed |
| `web_search` | Query needs current/live/recent external information |
| `direct_answer` | Query is answerable from general knowledge, no lookup needed |

**Base model:** [Qwen/Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct)
**Fine-tuning method:** LoRA (r=8, alpha=16, dropout=0.1) applied to attention
projections (q_proj, k_proj, v_proj, o_proj), then merged into the base
weights for standalone deployment.
**Task head:** Sequence classification (AutoModelForSequenceClassification),
not generative/instruction-following output, chosen specifically for
single-forward-pass inference speed and zero output-parsing risk in production.

## Intended Use

Designed as the routing node in a multi-tool AI agent (e.g. a LangGraph graph),
where a fast, cheap classification decision is needed before dispatching to a
more expensive tool (RAG pipeline, code execution, web search, or direct LLM
response). Not intended as a general-purpose chat model -- it has no
generative/conversational capability; it only outputs one of the four labels.

## Training Data

581 synthetically generated employee queries (Llama-3.3-70B via Groq), covering
4 balanced categories, with two deliberate design choices:

1. ~20% of examples per category were generated as near-boundary/ambiguous
   cases (e.g., a query mentioning a coding library but actually asking for
   current release info), so the benchmark reflects realistic routing difficulty
   rather than only trivially separable examples.
2. Targeted augmentation after error analysis: an initial training run
   (530 examples) showed the model confusing technical-vocabulary queries with
   code_tool even when the query was actually asking for current/live info
   (e.g. "has there been an update to the AWS CLI recently"). 60 additional
   web_search examples specifically pairing technical vocabulary with
   recency framing were generated and added, improving web_search recall
   from 84.4% to 90.6%, with a minor, expected trade-off on two adjacent
   code_tool boundary cases (a normal small-data fine-tuning dynamic, not
   a regression -- net test accuracy was unchanged at 95.3%).

**Data quality controls:**
- Exact-hash deduplication during generation
- TF-IDF cosine-similarity near-duplicate detection and removal, run separately
  for (a) train-vs-test/val leakage and (b) train-internal redundancy
- Test set generated in a separate batch/prompt pass from train, specifically
  to avoid the test set being near-paraphrases of training examples

Train/val/test split: 581 / 94 / 127 examples. Test set was evaluated exactly
once, after training was finalized, to ensure a fair held-out estimate.

## Evaluation Results

### Accuracy comparison (vs. zero-shot LLM baseline)

Benchmarked against an independent LLM (zero-shot prompted, no examples, same
held-out 127-query test set) as a fair, methodologically-clean comparison point
-- the LLM judge was not used to generate the training data, to avoid
contaminating the comparison.

| Metric | This model (0.5B, fine-tuned) | LLM baseline (zero-shot) |
|---|---|---|
| Accuracy | 95.3% | 96.9% |
| Macro F1 | 95.2% | 96.6% |

### Latency comparison

Measured as real wall-clock time per query: local GPU forward pass (this model)
vs. a real API call to a 70B LLM via Groq (chosen specifically because Groq is
among the fastest LLM inference providers available -- making this a
conservative, harder-to-dispute speed comparison rather than picking an easy
target).

| Metric | This model | LLM baseline (Groq, 70B) |
|---|---|---|
| Mean latency | 110 ms | 365 ms |
| Speedup | 3.3x faster | -- |
| Parameter count | 0.5B | 70B (140x larger) |

### Per-class breakdown (this model, final test run)

| Category | Precision | Recall | F1 |
|---|---|---|---|
| code_tool | 0.941 | 0.941 | 0.941 |
| direct_answer | 0.926 | 0.962 | 0.943 |
| rag_tool | 0.946 | 1.000 | 0.972 |
| web_search | 1.000 | 0.906 | 0.951 |

## Honest Limitations

- The LLM baseline scores slightly higher on raw accuracy (96.9% vs 95.3%).
  This model's value proposition is not "beats the LLM outright" -- it is
  near-parity accuracy at 140x fewer parameters and 3.3x lower latency, which
  is the relevant trade-off for a high-throughput routing node that runs on
  every single user message.
- Two of the four LLM-baseline misclassifications, and several of this model's
  misclassifications, cluster on the same two genuinely ambiguous test
  examples (queries that blend web_search framing with general-knowledge
  framing), suggesting these specific examples may sit at a real boundary in
  the category definitions themselves, not purely a model weakness.
- Trained on synthetic, LLM-generated data -- phrasing diversity, while
  deliberately varied, reflects the generating model's style and may not fully
  capture real employee phrasing patterns in a live deployment.
- Dataset size (581 train examples) is small; performance on categories or
  phrasing patterns not represented in training is untested.

## Training Procedure

- Hardware: Google Colab T4 GPU
- LoRA: r=8, alpha=16, dropout=0.1, targeting q_proj/k_proj/v_proj/o_proj
- Trainable parameters: 1,084,928 / 495,121,280 (0.22%)
- Epochs: 8, learning rate 2e-4, batch size 16
- Tracked via MLflow

## Citation

If you use this model, please reference the base model:

Qwen2.5 Technical Report, Qwen Team, Alibaba Group