krishnateja95 commited on
Commit
7568acb
·
verified ·
1 Parent(s): 4902a43

Add Tool Calling Evaluation section with BFCLv4 results

Browse files

Added Tool Calling Evaluation section to the model card with Berkeley Function-Calling Leaderboard v4 (BFCLv4) results comparing this quantized model against the base nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16 model. Includes Overall, Single Turn (Non-Live and Live), Multi-Turn, and Agentic accuracy with recovery percentages.

Files changed (1) hide show
  1. README.md +54 -0
README.md CHANGED
@@ -306,3 +306,57 @@ The model was evaluated across instruct, reasoning, and coding tasks. Scores are
306
  </tr>
307
  </tbody>
308
  </table>
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
306
  </tr>
307
  </tbody>
308
  </table>
309
+
310
+ ### Tool Calling Evaluation
311
+
312
+ The model was evaluated on tool calling tasks using the [Berkeley Function-Calling Leaderboard v4 (BFCLv4)](https://gorilla.cs.berkeley.edu/leaderboard.html).
313
+ [vLLM](https://docs.vllm.ai/en/stable/) was used as the serving backend for all evaluations.
314
+
315
+ <table>
316
+ <thead>
317
+ <tr>
318
+ <th>Category</th>
319
+ <th>Benchmark</th>
320
+ <th>nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16</th>
321
+ <th>RedHatAI/NVIDIA-Nemotron-3-Ultra-550B-A55B-quantized.w4a16<br>(this model)</th>
322
+ <th>Recovery</th>
323
+ </tr>
324
+ </thead>
325
+ <tbody>
326
+ <tr>
327
+ <td><strong>Overall</strong></td>
328
+ <td><strong>BFCLv4 Overall Acc</strong></td>
329
+ <td><strong>55.44</strong></td>
330
+ <td><strong>53.95</strong></td>
331
+ <td><strong>97.31%</strong></td>
332
+ </tr>
333
+ <tr>
334
+ <td>Single Turn</td>
335
+ <td>Non-Live Acc</td>
336
+ <td>45.00</td>
337
+ <td>44.58</td>
338
+ <td>99.07%</td>
339
+ </tr>
340
+ <tr>
341
+ <td>Single Turn</td>
342
+ <td>Live Acc</td>
343
+ <td>71.65</td>
344
+ <td>71.87</td>
345
+ <td>100.31%</td>
346
+ </tr>
347
+ <tr>
348
+ <td>Multi-Turn</td>
349
+ <td>Multi-Turn Acc</td>
350
+ <td>42.12</td>
351
+ <td>42.12</td>
352
+ <td>100.00%</td>
353
+ </tr>
354
+ <tr>
355
+ <td>Agentic</td>
356
+ <td>Agentic Acc</td>
357
+ <td>57.64</td>
358
+ <td>54.24</td>
359
+ <td>94.10%</td>
360
+ </tr>
361
+ </tbody>
362
+ </table>