common_init_from_params: setting dry_penalty_last_n to ctx_size = 768 common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CUDA : ARCHS = 750 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | multiple_choice_score: there are 869 tasks in prompt multiple_choice_score: selecting 750 random tasks from 869 tasks available multiple_choice_score: preparing task data...done multiple_choice_score : calculating TruthfulQA score over 750 tasks. task acc_norm 1 100.00000000 2 100.00000000 3 100.00000000 4 75.00000000 5 80.00000000 6 83.33333333 7 85.71428571 8 87.50000000 9 77.77777778 10 80.00000000 11 72.72727273 12 75.00000000 13 69.23076923 14 71.42857143 15 73.33333333 16 75.00000000 17 76.47058824 18 77.77777778 19 78.94736842 20 75.00000000 21 76.19047619 22 77.27272727 23 73.91304348 24 75.00000000 25 72.00000000 26 69.23076923 27 70.37037037 28 67.85714286 29 68.96551724 30 70.00000000 31 67.74193548 32 68.75000000 33 69.69696970 34 70.58823529 35 71.42857143 36 69.44444444 37 70.27027027 38 71.05263158 39 71.79487179 40 72.50000000 41 70.73170732 42 71.42857143 43 69.76744186 44 68.18181818 45 68.88888889 46 69.56521739 47 70.21276596 48 70.83333333 49 71.42857143 50 70.00000000 51 68.62745098 52 69.23076923 53 69.81132075 54 68.51851852 55 67.27272727 56 67.85714286 57 68.42105263 58 68.96551724 59 69.49152542 60 68.33333333 61 68.85245902 62 69.35483871 63 69.84126984 64 70.31250000 65 69.23076923 66 69.69696970 67 70.14925373 68 70.58823529 69 71.01449275 70 71.42857143 71 71.83098592 72 70.83333333 73 71.23287671 74 71.62162162 75 72.00000000 76 72.36842105 77 72.72727273 78 73.07692308 79 73.41772152 80 73.75000000 81 74.07407407 82 74.39024390 83 74.69879518 84 75.00000000 85 75.29411765 86 75.58139535 87 75.86206897 88 76.13636364 89 76.40449438 90 76.66666667 91 76.92307692 92 77.17391304 93 76.34408602 94 75.53191489 95 75.78947368 96 76.04166667 97 75.25773196 98 75.51020408 99 75.75757576 100 76.00000000 101 76.23762376 102 76.47058824 103 75.72815534 104 75.00000000 105 75.23809524 106 74.52830189 107 74.76635514 108 74.07407407 109 73.39449541 110 73.63636364 111 73.87387387 112 74.10714286 113 73.45132743 114 73.68421053 115 73.91304348 116 74.13793103 117 73.50427350 118 73.72881356 119 73.10924370 120 73.33333333 121 72.72727273 122 72.95081967 123 73.17073171 124 73.38709677 125 73.60000000 126 73.80952381 127 74.01574803 128 74.21875000 129 73.64341085 130 73.84615385 131 74.04580153 132 73.48484848 133 73.68421053 134 73.88059701 135 73.33333333 136 73.52941176 137 73.72262774 138 73.18840580 139 73.38129496 140 72.85714286 141 72.34042553 142 72.53521127 143 72.72727273 144 72.91666667 145 73.10344828 146 73.28767123 147 73.46938776 148 72.97297297 149 73.15436242 150 73.33333333 151 73.50993377 152 73.68421053 153 73.85620915 154 74.02597403 155 73.54838710 156 73.71794872 157 73.88535032 158 74.05063291 159 74.21383648 160 74.37500000 161 74.53416149 162 74.69135802 163 74.23312883 164 74.39024390 165 74.54545455 166 74.69879518 167 74.85029940 168 75.00000000 169 74.55621302 170 74.70588235 171 74.85380117 172 74.41860465 173 74.56647399 174 74.71264368 175 74.28571429 176 74.43181818 177 74.57627119 178 74.71910112 179 74.86033520 180 75.00000000 181 75.13812155 182 75.27472527 183 74.86338798 184 75.00000000 185 75.13513514 186 75.26881720 187 75.40106952 188 75.00000000 189 75.13227513 190 74.73684211 191 74.86910995 192 74.47916667 193 74.09326425 194 74.22680412 195 74.35897436 196 74.48979592 197 74.61928934 198 74.74747475 199 74.87437186 200 75.00000000 201 75.12437811 202 75.24752475 203 75.36945813 204 75.49019608 205 75.60975610 206 75.24271845 207 75.36231884 208 75.48076923 209 75.59808612 210 75.71428571 211 75.82938389 212 75.47169811 213 75.58685446 214 75.70093458 215 75.81395349 216 75.46296296 217 75.57603687 218 75.68807339 219 75.79908676 220 75.45454545 221 75.56561086 222 75.67567568 223 75.33632287 224 75.44642857 225 75.11111111 226 74.77876106 227 74.88986784 228 75.00000000 229 75.10917031 230 75.21739130 231 74.89177489 232 74.56896552 233 74.24892704 234 73.93162393 235 73.61702128 236 73.72881356 237 73.41772152 238 73.10924370 239 73.22175732 240 73.33333333 241 73.44398340 242 73.55371901 243 73.66255144 244 73.36065574 245 73.46938776 246 73.57723577 247 73.68421053 248 73.79032258 249 73.89558233 250 74.00000000 251 74.10358566 252 74.20634921 253 74.30830040 254 74.40944882 255 74.50980392 256 74.60937500 257 74.31906615 258 74.41860465 259 74.51737452 260 74.61538462 261 74.71264368 262 74.80916031 263 74.90494297 264 75.00000000 265 75.09433962 266 74.81203008 267 74.90636704 268 75.00000000 269 75.09293680 270 75.18518519 271 75.27675277 272 75.36764706 273 75.09157509 274 74.81751825 275 74.54545455 276 74.63768116 277 74.72924188 278 74.82014388 279 74.91039427 280 75.00000000 281 75.08896797 282 75.17730496 283 74.91166078 284 75.00000000 285 75.08771930 286 75.17482517 287 74.91289199 288 75.00000000 289 74.74048443 290 74.82758621 291 74.91408935 292 75.00000000 293 75.08532423 294 74.82993197 295 74.91525424 296 75.00000000 297 75.08417508 298 75.16778523 299 75.25083612 300 75.33333333 301 75.08305648 302 75.16556291 303 75.24752475 304 75.32894737 305 75.40983607 306 75.16339869 307 75.24429967 308 75.32467532 309 75.40453074 310 75.16129032 311 75.24115756 312 75.32051282 313 75.39936102 314 75.47770701 315 75.55555556 316 75.63291139 317 75.70977918 318 75.78616352 319 75.54858934 320 75.31250000 321 75.38940810 322 75.46583851 323 75.54179567 324 75.61728395 325 75.69230769 326 75.76687117 327 75.53516820 328 75.60975610 329 75.68389058 330 75.45454545 331 75.22658610 332 75.30120482 333 75.37537538 334 75.14970060 335 74.92537313 336 74.70238095 337 74.77744807 338 74.55621302 339 74.63126844 340 74.70588235 341 74.78005865 342 74.85380117 343 74.92711370 344 75.00000000 345 74.78260870 346 74.85549133 347 74.92795389 348 74.71264368 349 74.78510029 350 74.85714286 351 74.92877493 352 75.00000000 353 75.07082153 354 74.85875706 355 74.64788732 356 74.71910112 357 74.78991597 358 74.86033520 359 74.65181058 360 74.72222222 361 74.79224377 362 74.86187845 363 74.93112948 364 75.00000000 365 75.06849315 366 74.86338798 367 74.93188011 368 75.00000000 369 75.06775068 370 75.13513514 371 74.93261456 372 75.00000000 373 75.06702413 374 75.13368984 375 74.93333333 376 75.00000000 377 74.80106101 378 74.86772487 379 74.67018470 380 74.73684211 381 74.54068241 382 74.34554974 383 74.15143603 384 74.21875000 385 74.28571429 386 74.09326425 387 74.16020672 388 74.22680412 389 74.29305913 390 74.10256410 391 73.91304348 392 73.97959184 393 74.04580153 394 73.85786802 395 73.92405063 396 73.73737374 397 73.80352645 398 73.61809045 399 73.68421053 400 73.75000000 401 73.81546135 402 73.88059701 403 73.94540943 404 74.00990099 405 74.07407407 406 74.13793103 407 74.20147420 408 74.26470588 409 74.08312958 410 74.14634146 411 74.20924574 412 74.27184466 413 74.33414044 414 74.39613527 415 74.45783133 416 74.51923077 417 74.34052758 418 74.40191388 419 74.22434368 420 74.28571429 421 74.34679335 422 74.40758294 423 74.46808511 424 74.52830189 425 74.58823529 426 74.41314554 427 74.23887588 428 74.06542056 429 74.12587413 430 73.95348837 431 74.01392111 432 74.07407407 433 73.90300231 434 73.96313364 435 74.02298851 436 74.08256881 437 74.14187643 438 73.97260274 439 74.03189066 440 74.09090909 441 74.14965986 442 73.98190045 443 74.04063205 444 74.09909910 445 74.15730337 446 74.21524664 447 74.27293065 448 74.33035714 449 74.38752784 450 74.22222222 451 74.27937916 452 74.33628319 453 74.39293598 454 74.22907489 455 74.28571429 456 74.34210526 457 74.39824945 458 74.45414847 459 74.50980392 460 74.34782609 461 74.40347072 462 74.45887446 463 74.29805616 464 74.35344828 465 74.40860215 466 74.46351931 467 74.51820128 468 74.57264957 469 74.41364606 470 74.46808511 471 74.52229299 472 74.57627119 473 74.41860465 474 74.47257384 475 74.31578947 476 74.36974790 477 74.42348008 478 74.47698745 479 74.53027140 480 74.58333333 481 74.63617464 482 74.68879668 483 74.74120083 484 74.79338843 485 74.84536082 486 74.89711934 487 74.74332649 488 74.79508197 489 74.84662577 490 74.89795918 491 74.94908350 492 75.00000000 493 74.84787018 494 74.89878543 495 74.94949495 496 74.79838710 497 74.64788732 498 74.69879518 499 74.74949900 500 74.80000000 501 74.85029940 502 74.90039841 503 74.95029821 504 75.00000000 505 75.04950495 506 74.90118577 507 74.75345168 508 74.80314961 509 74.85265226 510 74.90196078 511 74.75538160 512 74.80468750 513 74.65886940 514 74.70817121 515 74.75728155 516 74.61240310 517 74.66150870 518 74.71042471 519 74.75915222 520 74.61538462 521 74.66410749 522 74.71264368 523 74.56978967 524 74.42748092 525 74.28571429 526 74.14448669 527 74.00379507 528 73.86363636 529 73.72400756 530 73.77358491 531 73.82297552 532 73.87218045 533 73.92120075 534 73.78277154 535 73.64485981 536 73.69402985 537 73.74301676 538 73.60594796 539 73.65491651 540 73.51851852 541 73.56746765 542 73.61623616 543 73.48066298 544 73.52941176 545 73.57798165 546 73.62637363 547 73.67458867 548 73.72262774 549 73.77049180 550 73.81818182 551 73.68421053 552 73.55072464 553 73.59855335 554 73.64620939 555 73.69369369 556 73.56115108 557 73.60861759 558 73.47670251 559 73.52415027 560 73.57142857 561 73.61853832 562 73.66548043 563 73.71225577 564 73.75886525 565 73.62831858 566 73.67491166 567 73.72134039 568 73.76760563 569 73.63796134 570 73.50877193 571 73.55516637 572 73.42657343 573 73.47294939 574 73.51916376 575 73.56521739 576 73.61111111 577 73.65684575 578 73.70242215 579 73.74784111 580 73.62068966 581 73.49397590 582 73.36769759 583 73.41337907 584 73.45890411 585 73.50427350 586 73.54948805 587 73.59454855 588 73.63945578 589 73.68421053 590 73.72881356 591 73.60406091 592 73.47972973 593 73.52445194 594 73.56902357 595 73.61344538 596 73.48993289 597 73.53433836 598 73.57859532 599 73.45575960 600 73.50000000 601 73.54409318 602 73.42192691 603 73.30016584 604 73.34437086 605 73.38842975 606 73.43234323 607 73.47611203 608 73.51973684 609 73.39901478 610 73.44262295 611 73.32242226 612 73.36601307 613 73.24632953 614 73.28990228 615 73.17073171 616 73.05194805 617 73.09562399 618 73.13915858 619 73.02100162 620 73.06451613 621 73.10789050 622 72.99035370 623 73.03370787 624 73.07692308 625 73.12000000 626 73.16293930 627 73.20574163 628 73.08917197 629 73.13195548 630 73.17460317 631 73.21711569 632 73.10126582 633 72.98578199 634 73.02839117 635 73.07086614 636 73.11320755 637 73.15541601 638 73.19749216 639 73.23943662 640 73.28125000 641 73.32293292 642 73.20872274 643 73.09486781 644 73.13664596 645 73.02325581 646 73.06501548 647 72.95208655 648 72.99382716 649 72.88135593 650 72.92307692 651 72.96466974 652 72.85276074 653 72.89433384 654 72.78287462 655 72.67175573 656 72.71341463 657 72.75494673 658 72.79635258 659 72.68588771 660 72.72727273 661 72.61724660 662 72.65861027 663 72.69984917 664 72.74096386 665 72.78195489 666 72.82282282 667 72.86356822 668 72.75449102 669 72.79521674 670 72.68656716 671 72.72727273 672 72.76785714 673 72.80832095 674 72.70029674 675 72.74074074 676 72.63313609 677 72.67355982 678 72.71386431 679 72.75405007 680 72.79411765 681 72.68722467 682 72.72727273 683 72.62079063 684 72.66081871 685 72.70072993 686 72.74052478 687 72.78020378 688 72.81976744 689 72.71407837 690 72.75362319 691 72.64833575 692 72.54335260 693 72.58297258 694 72.62247839 695 72.51798561 696 72.41379310 697 72.45337159 698 72.34957020 699 72.24606581 700 72.14285714 701 72.03994294 702 72.07977208 703 72.11948791 704 72.15909091 705 72.19858156 706 72.23796034 707 72.27722772 708 72.31638418 709 72.35543018 710 72.39436620 711 72.43319269 712 72.47191011 713 72.51051893 714 72.40896359 715 72.30769231 716 72.34636872 717 72.38493724 718 72.28412256 719 72.18358832 720 72.22222222 721 72.12205270 722 72.16066482 723 72.19917012 724 72.23756906 725 72.27586207 726 72.17630854 727 72.21458047 728 72.25274725 729 72.29080933 730 72.32876712 731 72.36662107 732 72.40437158 733 72.44201910 734 72.47956403 735 72.51700680 736 72.41847826 737 72.32021710 738 72.35772358 739 72.25981055 740 72.16216216 741 72.19973009 742 72.23719677 743 72.27456258 744 72.17741935 745 72.21476510 746 72.25201072 747 72.15528782 748 72.19251337 749 72.09612817 750 72.13333333 Final result: 72.1333 +/- 1.6382 Random chance: 25.0083 +/- 1.5824