common_init_from_params: setting dry_penalty_last_n to ctx_size = 768 common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CUDA : ARCHS = 750 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | multiple_choice_score: there are 869 tasks in prompt multiple_choice_score: selecting 750 random tasks from 869 tasks available multiple_choice_score: preparing task data...done multiple_choice_score : calculating TruthfulQA score over 750 tasks. task acc_norm 1 100.00000000 2 100.00000000 3 100.00000000 4 75.00000000 5 80.00000000 6 83.33333333 7 85.71428571 8 87.50000000 9 77.77777778 10 80.00000000 11 72.72727273 12 75.00000000 13 69.23076923 14 71.42857143 15 73.33333333 16 75.00000000 17 76.47058824 18 77.77777778 19 78.94736842 20 75.00000000 21 76.19047619 22 77.27272727 23 78.26086957 24 79.16666667 25 76.00000000 26 73.07692308 27 74.07407407 28 71.42857143 29 72.41379310 30 73.33333333 31 70.96774194 32 71.87500000 33 72.72727273 34 73.52941176 35 74.28571429 36 72.22222222 37 72.97297297 38 73.68421053 39 74.35897436 40 75.00000000 41 73.17073171 42 73.80952381 43 72.09302326 44 70.45454545 45 71.11111111 46 71.73913043 47 72.34042553 48 72.91666667 49 73.46938776 50 72.00000000 51 70.58823529 52 71.15384615 53 71.69811321 54 70.37037037 55 69.09090909 56 69.64285714 57 70.17543860 58 70.68965517 59 71.18644068 60 70.00000000 61 70.49180328 62 70.96774194 63 71.42857143 64 71.87500000 65 70.76923077 66 71.21212121 67 71.64179104 68 72.05882353 69 72.46376812 70 72.85714286 71 73.23943662 72 72.22222222 73 72.60273973 74 72.97297297 75 73.33333333 76 73.68421053 77 74.02597403 78 74.35897436 79 74.68354430 80 75.00000000 81 75.30864198 82 75.60975610 83 74.69879518 84 75.00000000 85 74.11764706 86 74.41860465 87 74.71264368 88 75.00000000 89 75.28089888 90 75.55555556 91 75.82417582 92 76.08695652 93 75.26881720 94 74.46808511 95 74.73684211 96 75.00000000 97 74.22680412 98 74.48979592 99 74.74747475 100 75.00000000 101 75.24752475 102 75.49019608 103 74.75728155 104 74.03846154 105 74.28571429 106 73.58490566 107 73.83177570 108 73.14814815 109 72.47706422 110 72.72727273 111 72.97297297 112 73.21428571 113 72.56637168 114 72.80701754 115 73.04347826 116 73.27586207 117 72.64957265 118 72.88135593 119 72.26890756 120 72.50000000 121 71.90082645 122 72.13114754 123 72.35772358 124 72.58064516 125 72.80000000 126 73.01587302 127 73.22834646 128 73.43750000 129 72.86821705 130 73.07692308 131 73.28244275 132 72.72727273 133 72.93233083 134 73.13432836 135 72.59259259 136 72.79411765 137 72.99270073 138 72.46376812 139 72.66187050 140 72.14285714 141 71.63120567 142 71.83098592 143 72.02797203 144 72.22222222 145 72.41379310 146 72.60273973 147 72.78911565 148 72.97297297 149 73.15436242 150 73.33333333 151 73.50993377 152 73.68421053 153 73.85620915 154 74.02597403 155 73.54838710 156 73.71794872 157 73.88535032 158 74.05063291 159 74.21383648 160 74.37500000 161 74.53416149 162 74.69135802 163 74.23312883 164 74.39024390 165 74.54545455 166 74.69879518 167 74.85029940 168 75.00000000 169 74.55621302 170 74.70588235 171 74.85380117 172 74.41860465 173 74.56647399 174 74.71264368 175 74.28571429 176 74.43181818 177 74.57627119 178 74.71910112 179 74.86033520 180 75.00000000 181 75.13812155 182 75.27472527 183 74.86338798 184 75.00000000 185 75.13513514 186 75.26881720 187 75.40106952 188 75.00000000 189 75.13227513 190 74.73684211 191 74.86910995 192 74.47916667 193 74.09326425 194 74.22680412 195 74.35897436 196 74.48979592 197 74.61928934 198 74.74747475 199 74.37185930 200 74.50000000 201 74.62686567 202 74.75247525 203 74.87684729 204 75.00000000 205 75.12195122 206 74.75728155 207 74.87922705 208 75.00000000 209 75.11961722 210 75.23809524 211 75.35545024 212 75.00000000 213 75.11737089 214 75.23364486 215 75.34883721 216 75.00000000 217 75.11520737 218 75.22935780 219 75.34246575 220 75.00000000 221 75.11312217 222 75.22522523 223 74.88789238 224 75.00000000 225 74.66666667 226 74.33628319 227 74.44933921 228 74.56140351 229 74.67248908 230 74.78260870 231 74.45887446 232 74.13793103 233 73.81974249 234 73.50427350 235 73.19148936 236 73.30508475 237 72.99578059 238 72.68907563 239 72.80334728 240 72.91666667 241 73.02904564 242 73.14049587 243 73.25102881 244 72.95081967 245 73.06122449 246 73.17073171 247 73.27935223 248 73.38709677 249 73.49397590 250 73.60000000 251 73.70517928 252 73.80952381 253 73.91304348 254 74.01574803 255 74.11764706 256 74.21875000 257 73.92996109 258 74.03100775 259 74.13127413 260 74.23076923 261 74.32950192 262 74.42748092 263 74.52471483 264 74.62121212 265 74.71698113 266 74.43609023 267 74.53183521 268 74.62686567 269 74.72118959 270 74.81481481 271 74.90774908 272 75.00000000 273 74.72527473 274 74.45255474 275 74.18181818 276 74.27536232 277 74.36823105 278 74.46043165 279 74.55197133 280 74.64285714 281 74.73309609 282 74.82269504 283 74.55830389 284 74.64788732 285 74.73684211 286 74.82517483 287 74.56445993 288 74.65277778 289 74.39446367 290 74.48275862 291 74.57044674 292 74.65753425 293 74.74402730 294 74.48979592 295 74.57627119 296 74.66216216 297 74.74747475 298 74.83221477 299 74.91638796 300 75.00000000 301 74.75083056 302 74.83443709 303 74.91749175 304 75.00000000 305 75.08196721 306 74.83660131 307 74.91856678 308 75.00000000 309 75.08090615 310 74.83870968 311 74.91961415 312 75.00000000 313 75.07987220 314 75.15923567 315 75.23809524 316 75.31645570 317 75.39432177 318 75.47169811 319 75.23510972 320 75.00000000 321 75.07788162 322 75.15527950 323 75.23219814 324 75.30864198 325 75.38461538 326 75.46012270 327 75.22935780 328 75.30487805 329 75.37993921 330 75.15151515 331 74.92447130 332 75.00000000 333 75.07507508 334 75.14970060 335 74.92537313 336 74.70238095 337 74.77744807 338 74.55621302 339 74.63126844 340 74.70588235 341 74.78005865 342 74.85380117 343 74.63556851 344 74.70930233 345 74.49275362 346 74.56647399 347 74.63976945 348 74.42528736 349 74.49856734 350 74.57142857 351 74.64387464 352 74.71590909 353 74.78753541 354 74.57627119 355 74.36619718 356 74.43820225 357 74.50980392 358 74.58100559 359 74.37325905 360 74.44444444 361 74.51523546 362 74.58563536 363 74.65564738 364 74.72527473 365 74.79452055 366 74.59016393 367 74.65940054 368 74.72826087 369 74.79674797 370 74.86486486 371 74.66307278 372 74.46236559 373 74.53083110 374 74.59893048 375 74.40000000 376 74.46808511 377 74.27055703 378 74.33862434 379 74.14248021 380 74.21052632 381 74.01574803 382 73.82198953 383 73.62924282 384 73.69791667 385 73.76623377 386 73.57512953 387 73.64341085 388 73.71134021 389 73.77892031 390 73.58974359 391 73.40153453 392 73.46938776 393 73.53689567 394 73.35025381 395 73.41772152 396 73.23232323 397 73.29974811 398 73.11557789 399 73.18295739 400 73.25000000 401 73.31670823 402 73.38308458 403 73.44913151 404 73.51485149 405 73.58024691 406 73.64532020 407 73.71007371 408 73.77450980 409 73.59413203 410 73.65853659 411 73.72262774 412 73.78640777 413 73.84987893 414 73.91304348 415 73.73493976 416 73.79807692 417 73.62110312 418 73.68421053 419 73.50835322 420 73.57142857 421 73.63420428 422 73.69668246 423 73.75886525 424 73.82075472 425 73.88235294 426 73.70892019 427 73.53629977 428 73.36448598 429 73.42657343 430 73.25581395 431 73.31786543 432 73.37962963 433 73.21016166 434 73.27188940 435 73.33333333 436 73.39449541 437 73.45537757 438 73.28767123 439 73.34851936 440 73.40909091 441 73.46938776 442 73.30316742 443 73.36343115 444 73.42342342 445 73.48314607 446 73.54260090 447 73.60178971 448 73.66071429 449 73.71937639 450 73.55555556 451 73.61419069 452 73.67256637 453 73.73068433 454 73.56828194 455 73.62637363 456 73.68421053 457 73.74179431 458 73.79912664 459 73.85620915 460 73.69565217 461 73.75271150 462 73.80952381 463 73.65010799 464 73.70689655 465 73.76344086 466 73.81974249 467 73.87580300 468 73.93162393 469 73.77398721 470 73.82978723 471 73.88535032 472 73.94067797 473 73.78435518 474 73.62869198 475 73.47368421 476 73.52941176 477 73.58490566 478 73.64016736 479 73.69519833 480 73.75000000 481 73.80457380 482 73.85892116 483 73.91304348 484 73.96694215 485 74.02061856 486 74.07407407 487 73.92197125 488 73.97540984 489 74.02862986 490 74.08163265 491 74.13441955 492 74.18699187 493 74.03651116 494 74.08906883 495 74.14141414 496 73.99193548 497 73.84305835 498 73.89558233 499 73.94789579 500 74.00000000 501 74.05189621 502 74.10358566 503 74.15506958 504 74.20634921 505 74.25742574 506 74.11067194 507 74.16173570 508 74.21259843 509 74.06679764 510 74.11764706 511 73.97260274 512 74.02343750 513 73.87914230 514 73.92996109 515 73.98058252 516 73.83720930 517 73.88781431 518 73.93822394 519 73.98843931 520 73.84615385 521 73.89635317 522 73.94636015 523 73.80497132 524 73.66412214 525 73.52380952 526 73.38403042 527 73.24478178 528 73.10606061 529 72.96786389 530 73.01886792 531 73.06967985 532 73.12030075 533 73.17073171 534 73.03370787 535 72.89719626 536 72.94776119 537 72.99813780 538 72.86245353 539 72.91280148 540 72.77777778 541 72.82809612 542 72.87822878 543 72.74401473 544 72.79411765 545 72.84403670 546 72.71062271 547 72.76051188 548 72.81021898 549 72.85974499 550 72.90909091 551 72.77676951 552 72.64492754 553 72.69439421 554 72.74368231 555 72.79279279 556 72.66187050 557 72.71095153 558 72.58064516 559 72.62969589 560 72.67857143 561 72.72727273 562 72.77580071 563 72.82415631 564 72.87234043 565 72.74336283 566 72.79151943 567 72.83950617 568 72.88732394 569 72.75922671 570 72.63157895 571 72.67950963 572 72.55244755 573 72.60034904 574 72.64808362 575 72.69565217 576 72.74305556 577 72.79029463 578 72.83737024 579 72.88428325 580 72.75862069 581 72.63339071 582 72.68041237 583 72.55574614 584 72.60273973 585 72.64957265 586 72.69624573 587 72.74275980 588 72.78911565 589 72.83531409 590 72.88135593 591 72.75803723 592 72.80405405 593 72.84991568 594 72.89562290 595 72.94117647 596 72.81879195 597 72.86432161 598 72.90969900 599 72.78797997 600 72.83333333 601 72.87853577 602 72.75747508 603 72.63681592 604 72.68211921 605 72.72727273 606 72.77227723 607 72.81713344 608 72.86184211 609 72.74220033 610 72.78688525 611 72.66775777 612 72.71241830 613 72.59380098 614 72.63843648 615 72.52032520 616 72.40259740 617 72.44732577 618 72.49190939 619 72.37479806 620 72.41935484 621 72.46376812 622 72.34726688 623 72.39165329 624 72.43589744 625 72.48000000 626 72.52396166 627 72.56778309 628 72.45222930 629 72.49602544 630 72.53968254 631 72.58320127 632 72.46835443 633 72.35387046 634 72.39747634 635 72.44094488 636 72.48427673 637 72.52747253 638 72.57053292 639 72.61345853 640 72.65625000 641 72.69890796 642 72.58566978 643 72.47278383 644 72.51552795 645 72.40310078 646 72.44582043 647 72.33384853 648 72.37654321 649 72.26502311 650 72.30769231 651 72.35023041 652 72.23926380 653 72.28177642 654 72.17125382 655 72.06106870 656 71.95121951 657 71.99391172 658 72.03647416 659 71.92716237 660 71.96969697 661 71.86081694 662 71.90332326 663 71.94570136 664 71.98795181 665 72.03007519 666 72.07207207 667 72.11394303 668 72.00598802 669 72.04783259 670 71.94029851 671 71.98211624 672 72.02380952 673 72.06537890 674 72.10682493 675 72.14814815 676 72.18934911 677 72.23042836 678 72.27138643 679 72.31222386 680 72.35294118 681 72.24669604 682 72.28739003 683 72.18155198 684 72.22222222 685 72.26277372 686 72.30320700 687 72.34352256 688 72.38372093 689 72.27866473 690 72.31884058 691 72.21418234 692 72.10982659 693 72.15007215 694 72.19020173 695 72.08633094 696 72.12643678 697 72.16642755 698 72.06303725 699 71.95994278 700 71.85714286 701 71.75463623 702 71.79487179 703 71.83499289 704 71.87500000 705 71.91489362 706 71.95467422 707 71.99434229 708 72.03389831 709 72.07334274 710 72.11267606 711 72.15189873 712 72.19101124 713 72.23001403 714 72.12885154 715 72.02797203 716 72.06703911 717 72.10599721 718 72.00557103 719 71.90542420 720 71.94444444 721 71.84466019 722 71.88365651 723 71.92254495 724 71.82320442 725 71.86206897 726 71.76308540 727 71.80192572 728 71.84065934 729 71.87928669 730 71.91780822 731 71.95622435 732 71.99453552 733 72.03274216 734 72.07084469 735 72.10884354 736 72.01086957 737 71.91316147 738 71.95121951 739 71.85385656 740 71.75675676 741 71.79487179 742 71.83288410 743 71.87079408 744 71.77419355 745 71.81208054 746 71.84986595 747 71.75368139 748 71.65775401 749 71.56208278 750 71.60000000 Final result: 71.6000 +/- 1.6477 Random chance: 25.0083 +/- 1.5824