naveenllama007/qwen2.5-0.5b-intent-router

Qwen2.5-0.5B Intent Router (LoRA fine-tuned)

A 500M-parameter sequence classifier fine-tuned to route user queries to one of four tool categories for an internal GenAI assistant, achieving 95.3% accuracy and 3.3x lower latency than a 70B-parameter LLM baseline on the same task.

Model Description

This model classifies a user query into exactly one of four categories, intended to sit at the entry point of an AI agent and decide which downstream tool/node should handle the request:

Category	Description
rag_tool	Query needs internal company knowledge (HR policy, internal docs, onboarding, internal tools)
code_tool	Query needs code written, debugged, explained, or reviewed
web_search	Query needs current/live/recent external information
direct_answer	Query is answerable from general knowledge, no lookup needed

Base model: Qwen/Qwen2.5-0.5B-Instruct Fine-tuning method: LoRA (r=8, alpha=16, dropout=0.1) applied to attention projections (q_proj, k_proj, v_proj, o_proj), then merged into the base weights for standalone deployment. Task head: Sequence classification (AutoModelForSequenceClassification), not generative/instruction-following output, chosen specifically for single-forward-pass inference speed and zero output-parsing risk in production.

Intended Use

Designed as the routing node in a multi-tool AI agent (e.g. a LangGraph graph), where a fast, cheap classification decision is needed before dispatching to a more expensive tool (RAG pipeline, code execution, web search, or direct LLM response). Not intended as a general-purpose chat model -- it has no generative/conversational capability; it only outputs one of the four labels.

Training Data

581 synthetically generated employee queries (Llama-3.3-70B via Groq), covering 4 balanced categories, with two deliberate design choices:

1. ~20% of examples per category were generated as near-boundary/ambiguous cases (e.g., a query mentioning a coding library but actually asking for current release info), so the benchmark reflects realistic routing difficulty rather than only trivially separable examples.
2. Targeted augmentation after error analysis: an initial training run (530 examples) showed the model confusing technical-vocabulary queries with code_tool even when the query was actually asking for current/live info (e.g. "has there been an update to the AWS CLI recently"). 60 additional web_search examples specifically pairing technical vocabulary with recency framing were generated and added, improving web_search recall from 84.4% to 90.6%, with a minor, expected trade-off on two adjacent code_tool boundary cases (a normal small-data fine-tuning dynamic, not a regression -- net test accuracy was unchanged at 95.3%).

Data quality controls:

• Exact-hash deduplication during generation
• TF-IDF cosine-similarity near-duplicate detection and removal, run separately for (a) train-vs-test/val leakage and (b) train-internal redundancy
• Test set generated in a separate batch/prompt pass from train, specifically to avoid the test set being near-paraphrases of training examples

Train/val/test split: 581 / 94 / 127 examples. Test set was evaluated exactly once, after training was finalized, to ensure a fair held-out estimate.

Evaluation Results

Accuracy comparison (vs. zero-shot LLM baseline)

Benchmarked against an independent LLM (zero-shot prompted, no examples, same held-out 127-query test set) as a fair, methodologically-clean comparison point -- the LLM judge was not used to generate the training data, to avoid contaminating the comparison.

Metric	This model (0.5B, fine-tuned)	LLM baseline (zero-shot)
Accuracy	95.3%	96.9%
Macro F1	95.2%	96.6%

Latency comparison

Measured as real wall-clock time per query: local GPU forward pass (this model) vs. a real API call to a 70B LLM via Groq (chosen specifically because Groq is among the fastest LLM inference providers available -- making this a conservative, harder-to-dispute speed comparison rather than picking an easy target).

Metric	This model	LLM baseline (Groq, 70B)
Mean latency	110 ms	365 ms
Speedup	3.3x faster	--
Parameter count	0.5B	70B (140x larger)

Per-class breakdown (this model, final test run)

Category	Precision	Recall	F1
code_tool	0.941	0.941	0.941
direct_answer	0.926	0.962	0.943
rag_tool	0.946	1.000	0.972
web_search	1.000	0.906	0.951

Honest Limitations

• The LLM baseline scores slightly higher on raw accuracy (96.9% vs 95.3%). This model's value proposition is not "beats the LLM outright" -- it is near-parity accuracy at 140x fewer parameters and 3.3x lower latency, which is the relevant trade-off for a high-throughput routing node that runs on every single user message.
• Two of the four LLM-baseline misclassifications, and several of this model's misclassifications, cluster on the same two genuinely ambiguous test examples (queries that blend web_search framing with general-knowledge framing), suggesting these specific examples may sit at a real boundary in the category definitions themselves, not purely a model weakness.
• Trained on synthetic, LLM-generated data -- phrasing diversity, while deliberately varied, reflects the generating model's style and may not fully capture real employee phrasing patterns in a live deployment.
• Dataset size (581 train examples) is small; performance on categories or phrasing patterns not represented in training is untested.

Training Procedure

• Hardware: Google Colab T4 GPU
• LoRA: r=8, alpha=16, dropout=0.1, targeting q_proj/k_proj/v_proj/o_proj
• Trainable parameters: 1,084,928 / 495,121,280 (0.22%)
• Epochs: 8, learning rate 2e-4, batch size 16
• Tracked via MLflow

Citation

If you use this model, please reference the base model:

Qwen2.5 Technical Report, Qwen Team, Alibaba Group