File size: 3,524 Bytes
e4cbca7 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 | ---
license: mit
language:
- en
tags:
- tool-calling
- function-calling
- fine-tuning
- lora
datasets:
- gorilla-llm/berkeley-function-call-leaderboard
base_model: Qwen/Qwen2.5-1.5B-Instruct
---
# Tool Calling LoRA Adapters
LoRA adapters for improving LLM tool calling, trained as part of the research paper "What Actually Improves LLM Tool Calling?"
## Model Description
These are LoRA adapters (rank 8, alpha 16) trained on top of `Qwen2.5-1.5B-Instruct` for function calling tasks using the Berkeley Function Calling Leaderboard (BFCL) dataset.
## Key Findings
Our ablation study found that:
- **SFT provides +47 points** accuracy improvement (9.7% → 57%)
- **DPO and RL add <1 point** when applied post-SFT
- **Training data diversity matters more than quantity**: 500 diverse examples outperform 500 homogeneous examples by 26 points
- **Tool generalization works well** (79% on unseen tools) but **pattern generalization is harder** (42% on unseen patterns)
## Available Adapters
| Adapter | Description | Accuracy |
|---------|-------------|----------|
| `sft/` | SFT on diverse BFCL data | 57.0% |
| `sft_dpo/` | SFT + DPO preference tuning | 57.7% |
| `sft_rl/` | SFT + reward-filtered RL | 58.0% |
| `tool_generalization/sft/` | SFT for unseen tools experiment | 79% on held-out tools |
| `category_holdout/sft/` | SFT for pattern generalization | 42% on held-out patterns |
| `diversity/high_diversity/` | Diverse training (125 x 4 categories) | 53% |
| `diversity/low_diversity/` | Homogeneous training (500 simple) | 27% |
## Usage
```python
from mlx_lm import load, generate
# Load base model with SFT adapter
model, tokenizer = load(
"mlx-community/Qwen2.5-1.5B-Instruct-4bit",
adapter_path="path/to/sft"
)
# Format your prompt with function definitions
messages = [
{"role": "system", "content": "You are a helpful assistant with access to functions..."},
{"role": "user", "content": "What's the weather in Paris?"}
]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
output = generate(model, tokenizer, prompt=prompt, max_tokens=256)
print(output)
# Output: {"name": "get_weather", "arguments": {"city": "Paris"}}
```
## Training Details
- **Base model**: Qwen2.5-1.5B-Instruct (4-bit quantization)
- **LoRA config**: rank=8, alpha=16
- **Training**: 300 iterations, learning rate 1e-5
- **Framework**: MLX on Apple Silicon
- **Data**: Berkeley Function Calling Leaderboard (BFCL)
## Call Pattern Categories
The BFCL benchmark includes four call pattern categories:
1. **Simple**: Single function call (e.g., `get_weather(city="Paris")`)
2. **Multiple**: Sequential calls where later calls depend on earlier results
3. **Parallel**: Independent concurrent calls
4. **Parallel-multiple**: Combinations requiring both parallel and sequential structure
## Recommendations
Based on our research:
1. **Use SFT with diverse training data** - this provides nearly all achievable gains
2. **Prioritize pattern diversity over tool coverage** - models generalize well to new tools but struggle with new patterns
3. **Skip complex pipelines** - DPO, RL, and scaffolding provide minimal benefit in our setting
## Citation
```bibtex
@article{ramakrishnan2024toolcalling,
title={What Actually Improves LLM Tool Calling?},
author={Ramakrishnan, Siddharth},
year={2024}
}
```
## Links
- [Paper PDF](#)
- [GitHub Repository](#)
- [BFCL Dataset](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard)
|