Upload 36 files

README.md

@@ -1,3 +1,106 @@
----
-license: mit
----
+---
+license: mit
+language:
+- en
+tags:
+- tool-calling
+- function-calling
+- fine-tuning
+- lora
+datasets:
+- gorilla-llm/berkeley-function-call-leaderboard
+base_model: Qwen/Qwen2.5-1.5B-Instruct
+---
+
+# Tool Calling LoRA Adapters
+
+LoRA adapters for improving LLM tool calling, trained as part of the research paper "What Actually Improves LLM Tool Calling?"
+
+## Model Description
+
+These are LoRA adapters (rank 8, alpha 16) trained on top of `Qwen2.5-1.5B-Instruct` for function calling tasks using the Berkeley Function Calling Leaderboard (BFCL) dataset.
+
+## Key Findings
+
+Our ablation study found that:
+
+- **SFT provides +47 points** accuracy improvement (9.7% → 57%)
+- **DPO and RL add <1 point** when applied post-SFT
+- **Training data diversity matters more than quantity**: 500 diverse examples outperform 500 homogeneous examples by 26 points
+- **Tool generalization works well** (79% on unseen tools) but **pattern generalization is harder** (42% on unseen patterns)
+
+## Available Adapters
+
+| Adapter | Description | Accuracy |
+|---------|-------------|----------|
+| `sft/` | SFT on diverse BFCL data | 57.0% |
+| `sft_dpo/` | SFT + DPO preference tuning | 57.7% |
+| `sft_rl/` | SFT + reward-filtered RL | 58.0% |
+| `tool_generalization/sft/` | SFT for unseen tools experiment | 79% on held-out tools |
+| `category_holdout/sft/` | SFT for pattern generalization | 42% on held-out patterns |
+| `diversity/high_diversity/` | Diverse training (125 x 4 categories) | 53% |
+| `diversity/low_diversity/` | Homogeneous training (500 simple) | 27% |
+
+## Usage
+
+```python
+from mlx_lm import load, generate
+
+# Load base model with SFT adapter
+model, tokenizer = load(
+    "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    adapter_path="path/to/sft"
+)
+
+# Format your prompt with function definitions
+messages = [
+    {"role": "system", "content": "You are a helpful assistant with access to functions..."},
+    {"role": "user", "content": "What's the weather in Paris?"}
+]
+
+prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
+output = generate(model, tokenizer, prompt=prompt, max_tokens=256)
+print(output)
+# Output: {"name": "get_weather", "arguments": {"city": "Paris"}}
+```
+
+## Training Details
+
+- **Base model**: Qwen2.5-1.5B-Instruct (4-bit quantization)
+- **LoRA config**: rank=8, alpha=16
+- **Training**: 300 iterations, learning rate 1e-5
+- **Framework**: MLX on Apple Silicon
+- **Data**: Berkeley Function Calling Leaderboard (BFCL)
+
+## Call Pattern Categories
+
+The BFCL benchmark includes four call pattern categories:
+
+1. **Simple**: Single function call (e.g., `get_weather(city="Paris")`)
+2. **Multiple**: Sequential calls where later calls depend on earlier results
+3. **Parallel**: Independent concurrent calls
+4. **Parallel-multiple**: Combinations requiring both parallel and sequential structure
+
+## Recommendations
+
+Based on our research:
+
+1. **Use SFT with diverse training data** - this provides nearly all achievable gains
+2. **Prioritize pattern diversity over tool coverage** - models generalize well to new tools but struggle with new patterns
+3. **Skip complex pipelines** - DPO, RL, and scaffolding provide minimal benefit in our setting
+
+## Citation
+
+```bibtex
+@article{ramakrishnan2024toolcalling,
+  title={What Actually Improves LLM Tool Calling?},
+  author={Ramakrishnan, Siddharth},
+  year={2024}
+}
+```
+
+## Links
+
+- [Paper PDF](#)
+- [GitHub Repository](#)
+- [BFCL Dataset](https://github.com/ShishirPatil/gorilla/tree/main/berkeley-function-call-leaderboard)

adapters/category_holdout/rl/0000100_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:623b3b11a7970ea1e1193f17d13a5641d6df01c89b56ee1a97cc703288a43420
+size 21126646

adapters/category_holdout/rl/adapter_config.json

@@ -0,0 +1,40 @@
+{
+    "adapter_path": "fc_benchmark/models/category_holdout/rl",
+    "batch_size": 1,
+    "config": null,
+    "data": "fc_benchmark/data/prepared/category_split/rl_filtered",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": false,
+    "iters": 100,
+    "learning_rate": 5e-06,
+    "lora_parameters": {
+        "rank": 8,
+        "dropout": 0.0,
+        "scale": 20.0
+    },
+    "lr_schedule": null,
+    "mask_prompt": false,
+    "max_seq_length": 2048,
+    "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": null,
+    "report_to": null,
+    "resume_adapter_file": "fc_benchmark/models/category_holdout/sft/adapters.safetensors",
+    "save_every": 100,
+    "seed": 0,
+    "steps_per_eval": 200,
+    "steps_per_report": 25,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

adapters/category_holdout/rl/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:623b3b11a7970ea1e1193f17d13a5641d6df01c89b56ee1a97cc703288a43420
+size 21126646

adapters/category_holdout/sft/0000100_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4342fdb99d4e5dab3afb6f1624b87d8fd32bc2c8342c321d3d662b14d0d3b896
+size 21126646

adapters/category_holdout/sft/0000200_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:754dc77e7ff3baa09ea36bee9b72c6118ef05223fc76a60b5241543c4d4434f2
+size 21126646

adapters/category_holdout/sft/0000300_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cee96b0e1c97f8e39a11cad0905f9acd330eac8eb6b09879641dedcacfac6b75
+size 21126646

adapters/category_holdout/sft/adapter_config.json

@@ -0,0 +1,40 @@
+{
+    "adapter_path": "fc_benchmark/models/category_holdout/sft",
+    "batch_size": 1,
+    "config": null,
+    "data": "fc_benchmark/data/prepared/category_split/sft",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": false,
+    "iters": 300,
+    "learning_rate": 1e-05,
+    "lora_parameters": {
+        "rank": 8,
+        "dropout": 0.0,
+        "scale": 20.0
+    },
+    "lr_schedule": null,
+    "mask_prompt": false,
+    "max_seq_length": 2048,
+    "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": null,
+    "report_to": null,
+    "resume_adapter_file": null,
+    "save_every": 100,
+    "seed": 0,
+    "steps_per_eval": 200,
+    "steps_per_report": 50,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

adapters/category_holdout/sft/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cee96b0e1c97f8e39a11cad0905f9acd330eac8eb6b09879641dedcacfac6b75
+size 21126646

adapters/diversity/high_diversity/0000100_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:acc644b355ed34aa7ca2a678193144409c81406eee0c7f6fe3d3de9e12e22ea2
+size 21126646

adapters/diversity/high_diversity/0000200_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74ae00b5c4de4788acb7085eef238c2c063a57f19aa01cb45f1c5de915c94511
+size 21126646

adapters/diversity/high_diversity/0000300_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4652a6117ba4f013b2202af19a6610c696a618b03f396ca823fe745944157c59
+size 21126646

adapters/diversity/high_diversity/adapter_config.json

@@ -0,0 +1,40 @@
+{
+    "adapter_path": "fc_benchmark/models/diversity/high_diversity",
+    "batch_size": 1,
+    "config": null,
+    "data": "fc_benchmark/data/prepared/diversity/high_diversity/sft",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": false,
+    "iters": 300,
+    "learning_rate": 1e-05,
+    "lora_parameters": {
+        "rank": 8,
+        "dropout": 0.0,
+        "scale": 20.0
+    },
+    "lr_schedule": null,
+    "mask_prompt": false,
+    "max_seq_length": 2048,
+    "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": null,
+    "report_to": null,
+    "resume_adapter_file": null,
+    "save_every": 100,
+    "seed": 0,
+    "steps_per_eval": 200,
+    "steps_per_report": 50,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

adapters/diversity/high_diversity/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4652a6117ba4f013b2202af19a6610c696a618b03f396ca823fe745944157c59
+size 21126646

adapters/diversity/low_diversity/0000100_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fccd6206210a9f473998a539f1f85c119a5bd4c292dfabdbb9b391645fc0eb43
+size 21126646

adapters/diversity/low_diversity/0000200_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b905a105509501466b607c35e0cff78607a6f7bfd5f1e09baa5a7bcfcfcff23f
+size 21126646

adapters/diversity/low_diversity/0000300_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07c2a6764d46cade0c14fca72243746e87955062f9ed8c0d871144f093178912
+size 21126646

adapters/diversity/low_diversity/adapter_config.json

@@ -0,0 +1,40 @@
+{
+    "adapter_path": "fc_benchmark/models/diversity/low_diversity",
+    "batch_size": 1,
+    "config": null,
+    "data": "fc_benchmark/data/prepared/diversity/low_diversity/sft",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": false,
+    "iters": 300,
+    "learning_rate": 1e-05,
+    "lora_parameters": {
+        "rank": 8,
+        "dropout": 0.0,
+        "scale": 20.0
+    },
+    "lr_schedule": null,
+    "mask_prompt": false,
+    "max_seq_length": 2048,
+    "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": null,
+    "report_to": null,
+    "resume_adapter_file": null,
+    "save_every": 100,
+    "seed": 0,
+    "steps_per_eval": 200,
+    "steps_per_report": 50,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

adapters/diversity/low_diversity/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:07c2a6764d46cade0c14fca72243746e87955062f9ed8c0d871144f093178912
+size 21126646

adapters/sft/0000300_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23341f026608086151e216c2bc924121b0269364820390915373794ef8e370d7
+size 21126646

adapters/sft/adapter_config.json

@@ -0,0 +1,40 @@
+{
+    "adapter_path": "fc_benchmark/models/ablation_sft",
+    "batch_size": 1,
+    "config": null,
+    "data": "fc_benchmark/data/prepared/sft",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": false,
+    "iters": 300,
+    "learning_rate": 1e-05,
+    "lora_parameters": {
+        "rank": 8,
+        "dropout": 0.0,
+        "scale": 20.0
+    },
+    "lr_schedule": null,
+    "mask_prompt": false,
+    "max_seq_length": 2048,
+    "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": null,
+    "report_to": null,
+    "resume_adapter_file": null,
+    "save_every": 300,
+    "seed": 0,
+    "steps_per_eval": 200,
+    "steps_per_report": 25,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

adapters/sft/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23341f026608086151e216c2bc924121b0269364820390915373794ef8e370d7
+size 21126646

adapters/sft_dpo/0000300_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23341f026608086151e216c2bc924121b0269364820390915373794ef8e370d7
+size 21126646

adapters/sft_dpo/adapter_config.json

@@ -0,0 +1,40 @@
+{
+    "adapter_path": "fc_benchmark/models/ablation_sft",
+    "batch_size": 1,
+    "config": null,
+    "data": "fc_benchmark/data/prepared/sft",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": false,
+    "iters": 300,
+    "learning_rate": 1e-05,
+    "lora_parameters": {
+        "rank": 8,
+        "dropout": 0.0,
+        "scale": 20.0
+    },
+    "lr_schedule": null,
+    "mask_prompt": false,
+    "max_seq_length": 2048,
+    "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": null,
+    "report_to": null,
+    "resume_adapter_file": null,
+    "save_every": 300,
+    "seed": 0,
+    "steps_per_eval": 200,
+    "steps_per_report": 25,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

adapters/sft_dpo/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23341f026608086151e216c2bc924121b0269364820390915373794ef8e370d7
+size 21126646

adapters/sft_rl/0000300_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23341f026608086151e216c2bc924121b0269364820390915373794ef8e370d7
+size 21126646

adapters/sft_rl/adapter_config.json

@@ -0,0 +1,40 @@
+{
+    "adapter_path": "fc_benchmark/models/ablation_sft",
+    "batch_size": 1,
+    "config": null,
+    "data": "fc_benchmark/data/prepared/sft",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": false,
+    "iters": 300,
+    "learning_rate": 1e-05,
+    "lora_parameters": {
+        "rank": 8,
+        "dropout": 0.0,
+        "scale": 20.0
+    },
+    "lr_schedule": null,
+    "mask_prompt": false,
+    "max_seq_length": 2048,
+    "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": null,
+    "report_to": null,
+    "resume_adapter_file": null,
+    "save_every": 300,
+    "seed": 0,
+    "steps_per_eval": 200,
+    "steps_per_report": 25,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

adapters/sft_rl/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:23341f026608086151e216c2bc924121b0269364820390915373794ef8e370d7
+size 21126646

adapters/tool_generalization/rl/0000100_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4a539c0be12fcc05ca3810a520f7740b50c0a87224a247b55bb6d637b425b3e
+size 21126646

adapters/tool_generalization/rl/adapter_config.json

@@ -0,0 +1,40 @@
+{
+    "adapter_path": "fc_benchmark/models/tool_generalization/rl",
+    "batch_size": 1,
+    "config": null,
+    "data": "fc_benchmark/data/prepared/tool_split/rl_filtered",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": false,
+    "iters": 100,
+    "learning_rate": 5e-06,
+    "lora_parameters": {
+        "rank": 8,
+        "dropout": 0.0,
+        "scale": 20.0
+    },
+    "lr_schedule": null,
+    "mask_prompt": false,
+    "max_seq_length": 2048,
+    "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": null,
+    "report_to": null,
+    "resume_adapter_file": "fc_benchmark/models/tool_generalization/sft/adapters.safetensors",
+    "save_every": 100,
+    "seed": 0,
+    "steps_per_eval": 200,
+    "steps_per_report": 25,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

adapters/tool_generalization/rl/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f4a539c0be12fcc05ca3810a520f7740b50c0a87224a247b55bb6d637b425b3e
+size 21126646

adapters/tool_generalization/sft/0000100_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:77094d2aa984db32054d83dbd561aec8899513952f7e444921ec7c92860ad977
+size 21126646

adapters/tool_generalization/sft/0000200_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4118b7d98be43e85d0226602b3a073a5d67cae60a61b741a9e0548c898f12144
+size 21126646

adapters/tool_generalization/sft/0000300_adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d033be95f7211df8194bd59f1e4264e6ea8527167465a340b6658ca0e2d069b
+size 21126646

adapters/tool_generalization/sft/adapter_config.json

@@ -0,0 +1,40 @@
+{
+    "adapter_path": "fc_benchmark/models/tool_generalization/sft",
+    "batch_size": 1,
+    "config": null,
+    "data": "fc_benchmark/data/prepared/tool_split/sft",
+    "fine_tune_type": "lora",
+    "grad_accumulation_steps": 1,
+    "grad_checkpoint": false,
+    "iters": 300,
+    "learning_rate": 1e-05,
+    "lora_parameters": {
+        "rank": 8,
+        "dropout": 0.0,
+        "scale": 20.0
+    },
+    "lr_schedule": null,
+    "mask_prompt": false,
+    "max_seq_length": 2048,
+    "model": "mlx-community/Qwen2.5-1.5B-Instruct-4bit",
+    "num_layers": 16,
+    "optimizer": "adam",
+    "optimizer_config": {
+        "adam": {},
+        "adamw": {},
+        "muon": {},
+        "sgd": {},
+        "adafactor": {}
+    },
+    "project_name": null,
+    "report_to": null,
+    "resume_adapter_file": null,
+    "save_every": 100,
+    "seed": 0,
+    "steps_per_eval": 200,
+    "steps_per_report": 50,
+    "test": false,
+    "test_batches": 500,
+    "train": true,
+    "val_batches": 25
+}

adapters/tool_generalization/sft/adapters.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5d033be95f7211df8194bd59f1e4264e6ea8527167465a340b6658ca0e2d069b
+size 21126646