Add Tool Calling Evaluation section with BFCLv4 results
Browse filesAdded Tool Calling Evaluation section to the model card with Berkeley Function-Calling Leaderboard v4 (BFCLv4) results comparing this quantized model against the base nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 model. Includes Overall, Single Turn (Non-Live and Live), Multi-Turn, and Agentic accuracy with recovery percentages.
README.md
CHANGED
|
@@ -306,3 +306,57 @@ The model was evaluated across instruct, reasoning, and coding tasks. Scores are
|
|
| 306 |
</tr>
|
| 307 |
</tbody>
|
| 308 |
</table>
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 306 |
</tr>
|
| 307 |
</tbody>
|
| 308 |
</table>
|
| 309 |
+
|
| 310 |
+
### Tool Calling Evaluation
|
| 311 |
+
|
| 312 |
+
The model was evaluated on tool calling tasks using the [Berkeley Function-Calling Leaderboard v4 (BFCLv4)](https://gorilla.cs.berkeley.edu/leaderboard.html).
|
| 313 |
+
[vLLM](https://docs.vllm.ai/en/stable/) was used as the serving backend for all evaluations.
|
| 314 |
+
|
| 315 |
+
<table>
|
| 316 |
+
<thead>
|
| 317 |
+
<tr>
|
| 318 |
+
<th>Category</th>
|
| 319 |
+
<th>Benchmark</th>
|
| 320 |
+
<th>nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16</th>
|
| 321 |
+
<th>RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16<br>(this model)</th>
|
| 322 |
+
<th>Recovery</th>
|
| 323 |
+
</tr>
|
| 324 |
+
</thead>
|
| 325 |
+
<tbody>
|
| 326 |
+
<tr>
|
| 327 |
+
<td><strong>Overall</strong></td>
|
| 328 |
+
<td><strong>BFCLv4 Overall Acc</strong></td>
|
| 329 |
+
<td><strong>55.44</strong></td>
|
| 330 |
+
<td><strong>53.95</strong></td>
|
| 331 |
+
<td><strong>97.31%</strong></td>
|
| 332 |
+
</tr>
|
| 333 |
+
<tr>
|
| 334 |
+
<td>Single Turn</td>
|
| 335 |
+
<td>Non-Live Acc</td>
|
| 336 |
+
<td>45.00</td>
|
| 337 |
+
<td>44.58</td>
|
| 338 |
+
<td>99.07%</td>
|
| 339 |
+
</tr>
|
| 340 |
+
<tr>
|
| 341 |
+
<td>Single Turn</td>
|
| 342 |
+
<td>Live Acc</td>
|
| 343 |
+
<td>71.65</td>
|
| 344 |
+
<td>71.87</td>
|
| 345 |
+
<td>100.31%</td>
|
| 346 |
+
</tr>
|
| 347 |
+
<tr>
|
| 348 |
+
<td>Multi-Turn</td>
|
| 349 |
+
<td>Multi-Turn Acc</td>
|
| 350 |
+
<td>42.12</td>
|
| 351 |
+
<td>42.12</td>
|
| 352 |
+
<td>100.00%</td>
|
| 353 |
+
</tr>
|
| 354 |
+
<tr>
|
| 355 |
+
<td>Agentic</td>
|
| 356 |
+
<td>Agentic Acc</td>
|
| 357 |
+
<td>57.64</td>
|
| 358 |
+
<td>54.24</td>
|
| 359 |
+
<td>94.10%</td>
|
| 360 |
+
</tr>
|
| 361 |
+
</tbody>
|
| 362 |
+
</table>
|