---
license: apache-2.0
base_model: unsloth/Qwen2.5-1.5B-Instruct
tags:
- gguf
- qwen2.5
- e-commerce
- intent-classification
- llama.cpp
datasets:
- custom
- mudasir13cs/E-commerce-query-rewriting-dataset
language:
- en
---

# Intent Classifier - GGUF Format

This repository contains GGUF format models for efficient inference with [llama.cpp](https://github.com/ggerganov/llama.cpp).

> **Note**: This is the GGUF (quantized) version of the model. For the full HuggingFace format model, see the [merged model repository](https://huggingface.co/mudasir13cs/E-commerce-intent-classifier).

## Model Information

- **Base Model**: `unsloth/Qwen2.5-1.5B-Instruct`
- **Task**: Intent classification for e-commerce
- **Fine-tuning Method**: LoRA (Low-Rank Adaptation) with Unsloth
- **LoRA Config**: R=32, Alpha=64, Dropout=0.05
- **Training Examples**: 3140 train, 392 validation, 394 test
- **Training Epochs**: 5
- **Effective Batch Size**: 32
- **Learning Rate**: 0.0002
- **Max Sequence Length**: 2048
- **Optimizer**: adamw_torch_fused
- **Precision**: bf16
- **Trained On**: NVIDIA RTX 4090 (24GB VRAM)

## Fine-Tuning Details

### Training Configuration

- **Framework**: Unsloth (optimized for fast training)
- **LoRA Rank**: 32
- **LoRA Alpha**: 64 (2x rank for optimal scaling)
- **LoRA Dropout**: 0.05
- **Batch Size**: 16 per device
- **Gradient Accumulation**: 2 steps
- **Learning Rate Schedule**: cosine
- **Weight Decay**: 0.01
- **Warmup Ratio**: 0.1

### Dataset

The model was fine-tuned on a custom e-commerce dataset containing:
- Pronoun resolution (30%)
- Ellipsis expansion (20%)
- Ordinal references (15%)
- Product name references (15%)
- Price/category queries (10%)
- Navigation commands (5%)
- Query refinements (5%)

Total: ~10,000 examples from real e-commerce product data (Flipkart, Amazon, MyOnlineShop).

## Available Formats

This repository contains multiple quantization levels:

- **f16**: Full precision (largest, best quality) - ~3GB
- **q4_k_m**: 4-bit quantization (smallest, recommended for most use cases) - ~1GB
- **q5_k_m**: 5-bit quantization (balanced quality/size) - ~1.2GB
- **q8_0**: 8-bit quantization (high quality, larger size) - ~1.8GB

## Usage with Different Backends

### 1. llama.cpp (Recommended for GGUF)

#### Installation

```bash
# Clone llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build && cd build
cmake .. -DGGML_CUDA=ON
cmake --build . --config Release -j
```

#### Basic Usage

```bash
# Using llama-cli
./llama.cpp/build/bin/llama-cli \
    -m path/to/intent-classifier-q4_k_m.gguf \
    -p "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"

# Using llama-server (for API access)
./llama.cpp/build/bin/llama-server \
    -m path/to/intent-classifier-q4_k_m.gguf \
    --port 8080
```

#### Python Example

```python
from llama_cpp import Llama

# Load model
llm = Llama(
    model_path="path/to/intent-classifier-q4_k_m.gguf",
    n_ctx=2048,  # Context window
    n_threads=4  # Number of CPU threads
)

# Prepare prompt (ChatML format)
prompt = "<|im_start|>system\nClassify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow_detail<|im_end|>"

# Generate
response = llm(
    prompt,
    max_tokens=128,
    temperature=0.7,
    stop=["<|im_end|>", "<|im_start|>"]
)

print(response['choices'][0]['text'])
```

### 2. Ollama

#### Import GGUF Model to Ollama

**Option 1: Using Modelfile (Recommended)**

1. Create a `Modelfile`:
```dockerfile
FROM ./intent-classifier-q4_k_m.gguf

TEMPLATE "<|im_start|>system\nClassify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow_detail<|im_end|>"

PARAMETER temperature 0.7
PARAMETER num_predict 128
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
```

2. Import the model:
```bash
ollama create intent-classifier -f Modelfile
```

**Option 2: Direct Import**
```bash
# Import GGUF file directly
ollama import intent-classifier-q4_k_m.gguf
```

#### Usage with Ollama

```bash
# Command line
ollama run intent-classifier "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"

# With context
ollama run intent-classifier "Context: Previous search: Smartphones\nQuery: show me that one"
```

#### Python API

```python
import requests

# Generate
response = requests.post(
    "http://localhost:11434/api/generate",
    json={
        "model": "intent-classifier",
        "prompt": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one",
        "stream": False,
        "options": {
            "temperature": 0.7,
            "num_predict": 128,
            "stop": ["<|im_end|>", "<|im_start|>"]
        }
    }
)

print(response.json()["response"])
```

#### Chat API

```python
import requests

response = requests.post(
    "http://localhost:11434/api/chat",
    json={
        "model": "intent-classifier",
        "messages": [
            {"role": "system", "content": "Classify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close."},
            {"role": "user", "content": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"}
        ],
        "stream": False
    }
)

print(response.json()["message"]["content"])
```

### 3. vLLM (For Merged Models)

> **Note**: vLLM works best with HuggingFace format models. Use the [merged model](https://huggingface.co/mudasir13cs/E-commerce-intent-classifier) instead of GGUF.

#### Installation

```bash
pip install vllm
```

#### Usage

```python
from vllm import LLM, SamplingParams

# Load merged model (not GGUF)
llm = LLM(
    model="mudasir13cs/E-commerce-intent-classifier",
    trust_remote_code=True,
    max_model_len=2048
)

# Prepare prompt
prompt = "<|im_start|>system\nClassify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow_detail<|im_end|>"

# Sampling parameters
sampling_params = SamplingParams(
    temperature=0.7,
    max_tokens=128,
    stop=["<|im_end|>", "<|im_start|>"]
)

# Generate
outputs = llm.generate([prompt], sampling_params)
print(outputs[0].outputs[0].text)
```

#### vLLM Server

```bash
# Start server
python -m vllm.entrypoints.openai.api_server \
    --model mudasir13cs/E-commerce-intent-classifier \
    --trust-remote-code \
    --port 8000

# Use OpenAI-compatible API
curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "intent-classifier",
        "prompt": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one",
        "max_tokens": 128,
        "temperature": 0.7
    }'
```

### 4. Text Generation Inference (TGI)

> **Note**: TGI works with HuggingFace format models. Use the [merged model](https://huggingface.co/mudasir13cs/E-commerce-intent-classifier).

#### Installation

```bash
# Using Docker (recommended)
docker pull ghcr.io/huggingface/text-generation-inference:latest
```

#### Usage

```bash
docker run --gpus all \
    -p 8080:80 \
    -v /path/to/model:/data \
    ghcr.io/huggingface/text-generation-inference:latest \
    --model-id mudasir13cs/E-commerce-intent-classifier \
    --trust-remote-code
```

#### Python Client

```python
from text_generation import Client

client = Client("http://localhost:8080")

response = client.generate(
    prompt="<|im_start|>system\nClassify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>\n<|im_start|>user\nContext:\nPrevious search: Smartphones\nState: SEARCH_RESULTS\nLast command: show_list\nProducts (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2\nProduct count: 6\n\nQuery: show me that one<|im_end|>\n<|im_start|>assistant\nshow_detail<|im_end|>",
    max_new_tokens=128,
    temperature=0.7,
    stop_sequences=["<|im_end|>", "<|im_start|>"]
)

print(response.generated_text)
```

### 5. Transformers (For Merged Models)

> **Note**: Use the [merged HuggingFace model](https://huggingface.co/mudasir13cs/E-commerce-intent-classifier) for Transformers.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load merged model
model = AutoModelForCausalLM.from_pretrained(
    "mudasir13cs/E-commerce-intent-classifier",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(
    "mudasir13cs/E-commerce-intent-classifier",
    trust_remote_code=True
)

# Prepare input
messages = [
    {"role": "system", "content": "Classify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close."},
    {"role": "user", "content": "Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one"}
]

# Apply chat template
prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

# Generate
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=128,
    temperature=0.7,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id
)

# Decode
response = tokenizer.decode(
    outputs[0][inputs.input_ids.shape[1]:],
    skip_special_tokens=True
)
print(response)
```

## Prompt Format

The model uses **ChatML format** (Qwen2.5's native format):

```
<|im_start|>system
Classify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.<|im_end|>
<|im_start|>user
Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one<|im_end|>
<|im_start|>assistant
show_detail<|im_end|>
```

### Input Structure

**System Message:**
```
Classify the user's intent based on the query and context. Intent can be: search, show_detail, go_back, or close.
```

**User Message (Context + Query):**
```
Context:
Previous search: [category]
State: [SEARCH_RESULTS|PRODUCT_DETAIL|INITIAL]
Last command: [show_list|show_item|go_back|close]
Products (N): [product1, product2, ...]
Product count: N

Query: [user query]
```

**Expected Output:**
```
show_detail
```

### Example Prompts

**Example 1: Pronoun Resolution**
```
Context:
Previous search: Smartphones
State: SEARCH_RESULTS
Last command: show_list
Products (6): iPhone 15 Pro, Samsung Galaxy S24, OnePlus 12, Google Pixel 8, Xiaomi 14, Nothing Phone 2
Product count: 6

Query: show me that one
```

**Example 2: Ellipsis Expansion**
```
Context:
Previous search: Laptops
State: SEARCH_RESULTS
Last command: show_list
Products (5): MacBook Pro, Dell XPS, HP Spectre, Lenovo ThinkPad, ASUS ZenBook
Product count: 5

Query: under 50000
```

**Example 3: Ordinal Reference**
```
Context:
Previous search: Headphones
State: SEARCH_RESULTS
Last command: show_list
Products (4): Sony WH-1000XM5, Bose QuietComfort, AirPods Max, Sennheiser Momentum
Product count: 4

Query: show me the second one
```

## Quantization Comparison

| Format | Size | Quality | Use Case |
|--------|------|---------|----------|
| f16 | ~3GB | Best | Maximum quality, sufficient VRAM |
| q8_0 | ~1.8GB | Excellent | High quality, moderate VRAM |
| q5_k_m | ~1.2GB | Very Good | Balanced quality/size |
| q4_k_m | ~1GB | Good | Smallest size, limited VRAM |

## Performance

- **Inference Speed**: Optimized for CPU and GPU (CUDA)
- **Memory Usage**: Significantly lower than original models
- **Quality**: Minimal quality loss with quantization
- **Test Accuracy**: 100% (394/394) on test set

## Download

Download the desired quantization level:

```bash
# Using huggingface-cli
huggingface-cli download mudasir13cs/E-commerce-intent-classifier-gguf \
    intent-classifier-q4_k_m.gguf \
    --local-dir ./models

# Or download all formats
huggingface-cli download mudasir13cs/E-commerce-intent-classifier-gguf \
    --local-dir ./models
```

## Related Models

- **Merged Model (HuggingFace Format)**: [`mudasir13cs/E-commerce-intent-classifier`](https://huggingface.co/mudasir13cs/E-commerce-intent-classifier)
  - Full model in HuggingFace format
  - Can be used with Transformers, Unsloth, or other HF-compatible libraries
  - Suitable for further fine-tuning or inference
  
- **Original LoRA Adapter**: See merged model repository for LoRA adapter details

## Backend Comparison

| Backend | Format | Best For | Pros | Cons |
|---------|--------|----------|------|------|
| **llama.cpp** | GGUF | CPU/GPU inference, edge devices | Fast, low memory, cross-platform | Limited to GGUF format |
| **Ollama** | GGUF | Local development, easy deployment | Simple API, auto-manages models | Requires model import |
| **vLLM** | HF | High-throughput serving | Very fast, batching support | Requires HF format, more memory |
| **TGI** | HF | Production serving | Optimized serving, Docker support | Requires HF format |
| **Transformers** | HF | Research, fine-tuning | Full flexibility, easy integration | Slower inference, more memory |

## Requirements

### For GGUF Models (llama.cpp, Ollama)
- [llama.cpp](https://github.com/ggerganov/llama.cpp) (for C/C++ usage)
- [llama-cpp-python](https://github.com/abetlen/llama-cpp-python) (for Python usage)
- [Ollama](https://ollama.ai/) (optional, for Ollama backend)
- CUDA (optional, for GPU acceleration)

### For Merged Models (vLLM, TGI, Transformers)
- [vLLM](https://github.com/vllm-project/vllm) (for high-throughput serving)
- [Text Generation Inference](https://github.com/huggingface/text-generation-inference) (for production serving)
- [Transformers](https://huggingface.co/docs/transformers) (for research/fine-tuning)
- CUDA (recommended for GPU acceleration)

## Citation

If you use this model, please cite:

```bibtex
@software{ecommerce_agent_models,
  title = {E-commerce Agent Models - Intent Classifier},
  author = {Syed Mudasir},
  year = {2025},
  url = {https://huggingface.co/mudasir13cs/E-commerce-intent-classifier-gguf}
}
```

## License

Apache 2.0

## Notes

- GGUF models are optimized for inference, not training
- Use q4_k_m for most production deployments
- f16 format is recommended for maximum quality if VRAM allows
- For training or further fine-tuning, use the [merged HuggingFace model](https://huggingface.co/mudasir13cs/E-commerce-intent-classifier)
- The model was trained on English e-commerce data and performs best on similar queries