---
license: apache-2.0
base_model: mistralai/Mistral-7B-Instruct-v0.3
tags:
- unsloth
- mistral
- lora
- education
- study-abroad
- parameter-efficient-tuning
- domain-adaptation
- nlp
- kaggle
datasets:
- millat/StudyAbroadGPT-Dataset
library_name: transformers
---

# StudyAbroadGPT-7B-LoRa-Kaggle

Parameter-efficient LoRA-adapted Mistral-7B-Instruct-v0.3 fine-tuned on synthetic study-abroad conversational data for domain-specific academic advising guidance.

**Status**: Generation and lightweight qualitative evaluation complete ✅ | Manual blinded scoring pending ⏳ | Factuality audit pending ⏳

## 🔗 Project Ecosystem

| Resource | Link |
|----------|------|
| Dataset | [millat/StudyAbroadGPT-Dataset](https://huggingface.co/datasets/millat/StudyAbroadGPT-Dataset) |
| Training Code | [codermillat/StudyAbroadGPT](https://github.com/codermillat/StudyAbroadGPT) |
| Dataset Generation | [codermillat/study-abroad-dataset](https://github.com/codermillat/study-abroad-dataset) |
| Evaluation Artifacts | [LoRA Paper evaluation workspace](https://github.com/codermillat/LoRA-Paper) |
| Research Paper | [arXiv:2504.15610](https://arxiv.org/abs/2504.15610) |
| Author ORCID | [0009-0005-7198-9893](https://orcid.org/0009-0005-7198-9893) |

## 📊 Model Details

### Architecture

| Component | Specification |
|-----------|---|
| **Base Model** | `mistralai/Mistral-7B-Instruct-v0.3` |
| **Base Model Size** | 7 billion parameters |
| **Quantization** | 4-bit NF4 (via Unsloth) |
| **Fine-Tuning Method** | LoRA (Low-Rank Adaptation) |
| **LoRA Rank (r)** | 16 |
| **LoRA Alpha (α)** | 32 |
| **Scaling Factor** | α / r = 2.0 |

### Trainable Parameters

- **Total Model Parameters**: ~7B
- **LoRA Trainable Parameters**: ~4.7M (0.07% of model)
- **LoRA Adapters**: Applied to Q, K, V, O projections (attention) + FFN layers (gate, up, down)

### Quantization Details

- **Method**: 4-bit NF4 quantization via Unsloth
- **Benefit**: Reduces model memory from ~14GB (fp16) to ~8GB
- **Trade-off**: Minimal impact on model quality with significant memory savings
- **Hardware Requirement**: 16GB+ VRAM GPU (e.g., Tesla T4, P100)

## 🎯 Training Details

### Training Data

- **Dataset**: [millat/StudyAbroadGPT-Dataset](https://huggingface.co/datasets/millat/StudyAbroadGPT-Dataset)
- **Training Samples**: 2,274 conversations
- **Test Samples**: 402 conversations
- **Average Turns/Conversation**: 5.2
- **Topics Covered**: Admissions, scholarships, visas, accommodation, cultural adaptation

### Training Configuration

| Parameter | Value |
|-----------|-------|
| **Batch Size** | 2 per device |
| **Gradient Accumulation** | 4 steps |
| **Learning Rate** | 2e-4 |
| **Warmup Ratio** | 0.03 |
| **Number of Epochs** | 4 |
| **Max Sequence Length** | 2048 tokens |
| **Optimizer** | adamw_8bit (8-bit optimizer) |
| **Max Gradient Norm** | 0.3 |
| **Learning Rate Scheduler** | Linear |
| **Effective Batch Size** | 8 (2 × 4) |

### Hardware & Resources

| Setting | Value |
|---------|-------|
| **GPU Tested** | Tesla T4 (Kaggle), Tesla P100 (Kaggle) |
| **RAM** | 16GB |
| **Training Time (T4)** | ~3-4 hours |
| **Training Time (P100)** | ~1-2 hours |
| **Monitoring** | Weights & Biases (WandB) |

## 🚀 Usage

### Option 1: Using Unsloth (Recommended for Inference)

```python
from unsloth import FastLanguageModel
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    max_seq_length=2048,
    dtype=torch.float16,
    load_in_4bit=True,
)

# Prepare for inference
FastLanguageModel.for_inference(model)

prompt = "What documents do I need for a UK student visa?"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.0,  # Deterministic
        do_sample=False,
        top_p=1.0
    )

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)
```

### Option 2: Using Transformers Library

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    subfolder="merged",
    torch_dtype="auto",
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle",
    subfolder="merged"
)

prompt = "How much should I budget for accommodation in London?"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.0,
    do_sample=False
)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

### Option 3: Using LoRA Adapter (Training)

```python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Load base model
base_model = AutoModelForCausalLM.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3",
    torch_dtype="auto",
    device_map="auto"
)

# Load LoRA adapters
model = PeftModel.from_pretrained(
    base_model,
    "millat/StudyAbroadGPT-7B-LoRa-Kaggle"
)

tokenizer = AutoTokenizer.from_pretrained(
    "mistralai/Mistral-7B-Instruct-v0.3"
)

# For continued training, unfreeze LoRA parameters
model.training_mode = True
```

### Option 4: Local Inference with Ollama/GGUF

The merged weights can be quantized to GGUF format for local inference:

```bash
# Clone the model
git clone https://huggingface.co/millat/StudyAbroadGPT-7B-LoRa-Kaggle

# Convert to GGUF (requires llama.cpp tools)
python convert.py studyabroadgpt-merged/

# Run locally with Ollama
ollama run studyabroadgpt-7b-merged
```

## 📊 Lightweight Evaluation Results

From companion 50-sample deterministic evaluation run:

### Generation Metrics

| Metric | Base Model | LoRA Model | Δ |
|--------|-----------|-----------|---|
| Avg Response Length (chars) | 1151.88 | 1178.74 | +26.86 |
| Avg Response Tokens | 252.26 | 254.72 | +2.46 |
| Avg Generation Time (sec) | 20.92 | 23.98 | +3.06 |
| Truncation Rate | 0.96 | 0.96 | 0.0 |

### Domain-Specific Term Coverage

| Term | Base | LoRA | Change |
|------|------|------|--------|
| University | 48% | 42% | -6% |
| Tuition | 12% | 22% | +10% |
| Scholarship | 20% | 16% | -4% |
| Visa | 8% | 6% | -2% |
| IELTS | 6% | 10% | +4% |

### Format Quality

| Format Indicator | Base | LoRA |
|---|---|---|
| Bullet/List Usage | 92% | 96% |
| Caveat Phrase Usage | 2% | 0% |

**Interpretation**: Lightweight qualitative findings only. Manual blinded scoring and factuality audit still pending before claiming quantified improvements.

## ⚠️ Important Limitations

### What This Model Is

✅ A domain-adapted assistant for study-abroad queries  
✅ Suitable for research and experimentation  
✅ Parameter-efficient alternative to full fine-tuning  
✅ Demonstrated on low-resource hardware (T4 GPU)  

### What This Model Is NOT

❌ **Not a policy authority** — Do not use as single source of truth  
❌ **Not factually verified** — Outputs may contain inaccuracies or hallucinations  
❌ **Not production-ready** — Requires validation layer before operational use  
❌ **Not comprehensive** — May miss edge cases or regional variations  
❌ **Not a replacement for official guidance** — Always verify with official sources  

### Recommended Usage

**✅ Safe to use for:**
- Educational chatbot prototyping
- Research on domain adaptation
- Fine-tuning experiments
- Data augmentation for related tasks

**❌ NOT safe to use for:**
- Direct immigration advice
- Official policy interpretation
- Time-sensitive information (visas, deadlines)
- High-stakes decision making without expert review

## 🔄 Model Variants

### Merged vs Adapter-Only

| Variant | Format | Size | Use Case |
|---------|--------|------|----------|
| `/merged` | Full merged weights | ~14GB | Inference, GGUF conversion |
| Adapter-only | LoRA weights | ~30MB | Further training, fine-tuning |

Both are available in this repository.

## 📚 Citation

If you use this model or findings:

### Model Card

```bibtex
@model{StudyAbroadGPT-7B-LoRa-Kaggle,
  title={StudyAbroadGPT-7B-LoRa-Kaggle: LoRA-Adapted Mistral-7B for Study Abroad Guidance},
  author={Hosen, Md Millat},
  year={2025},
  howpublished={\url{https://huggingface.co/millat/StudyAbroadGPT-7B-LoRa-Kaggle}},
  note={HuggingFace Model Hub}
}
```

### Research Paper

```bibtex
@article{hosen2025lora,
  title={A LoRA-Based Approach to Fine-Tuning LLMs for Educational Guidance in Resource-Constrained Settings},
  author={Hosen, Md Millat},
  journal={arXiv preprint arXiv:2504.15610},
  year={2025},
  doi={10.48550/arXiv.2504.15610}
}
```

## 🔐 License

[Apache 2.0](https://opensource.org/licenses/Apache-2.0)

## 📊 Performance Benchmarks

### Comparison with Base Model

On a 50-sample deterministic evaluation:
- Response length: +2.3% increase
- Generation time: +14.6% slower (acceptable trade-off for domain specialization)
- Topic relevance: Maintained domain terminology coverage

### Hardware Compatibility

| Device | Status | Notes |
|--------|--------|-------|
| NVIDIA T4 (16GB) | ✅ Tested | Kaggle |
| NVIDIA P100 (16GB) | ✅ Tested | Kaggle |
| NVIDIA A100 (40GB) | Should work | Not tested |
| CPU Only | ❌ Not recommended | Too slow |
| Mac M1/M2 | ⚠️ Requires setup | MPS acceleration possible |

## 🤝 Contributing & Feedback

- **Bug Reports**: Open an issue on [training repo](https://github.com/codermillat/StudyAbroadGPT)
- **Model Improvements**: Submit PRs with new training runs or data
- **Questions**: Check companion evaluation artifacts or GitHub discussions
- **Evaluation**: See [LoRA Paper workspace](https://github.com/codermillat/LoRA-Paper)

## 📧 Support

- 📖 **Documentation**: See [WANDB.md](https://github.com/codermillat/StudyAbroadGPT/blob/main/WANDB.md) for monitoring details
- 🔧 **Setup Help**: See [training repo README](https://github.com/codermillat/StudyAbroadGPT#quick-start)
- 📋 **Architecture**: See [architecture.md](https://github.com/codermillat/StudyAbroadGPT/blob/main/architecture.md)
- 📰 **Full Methods**: See [arXiv:2504.15610](https://arxiv.org/abs/2504.15610)

---

**Model Version**: 1.0  
**Release Date**: May 2025  
**Last Updated**: May 2025  
**Training Framework**: Unsloth + Transformers  
**Base Model Lineage**: Mistral-7B → Instruct-v0.3 → 4-bit Quantized → LoRA Fine-tuned