common_init_from_params: setting dry_penalty_last_n to ctx_size = 768 common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable) system_info: n_threads = 8 (n_threads_batch = 8) / 16 | CUDA : ARCHS = 750 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | AVX512_VNNI = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 | multiple_choice_score: there are 817 tasks in prompt multiple_choice_score: selecting 750 random tasks from 817 tasks available multiple_choice_score: preparing task data...done multiple_choice_score : calculating TruthfulQA score over 750 tasks. task acc_norm 1 100.00000000 2 50.00000000 3 33.33333333 4 25.00000000 5 20.00000000 6 16.66666667 7 14.28571429 8 25.00000000 9 22.22222222 10 20.00000000 11 18.18181818 12 25.00000000 13 30.76923077 14 28.57142857 15 33.33333333 16 31.25000000 17 29.41176471 18 27.77777778 19 26.31578947 20 25.00000000 21 28.57142857 22 27.27272727 23 30.43478261 24 33.33333333 25 32.00000000 26 30.76923077 27 29.62962963 28 28.57142857 29 31.03448276 30 30.00000000 31 29.03225806 32 31.25000000 33 33.33333333 34 32.35294118 35 31.42857143 36 30.55555556 37 32.43243243 38 31.57894737 39 33.33333333 40 32.50000000 41 31.70731707 42 30.95238095 43 32.55813953 44 34.09090909 45 35.55555556 46 34.78260870 47 34.04255319 48 35.41666667 49 34.69387755 50 36.00000000 51 37.25490196 52 36.53846154 53 35.84905660 54 35.18518519 55 36.36363636 56 35.71428571 57 35.08771930 58 34.48275862 59 33.89830508 60 35.00000000 61 34.42622951 62 35.48387097 63 34.92063492 64 35.93750000 65 35.38461538 66 36.36363636 67 35.82089552 68 35.29411765 69 34.78260870 70 34.28571429 71 33.80281690 72 34.72222222 73 34.24657534 74 33.78378378 75 33.33333333 76 34.21052632 77 33.76623377 78 33.33333333 79 32.91139241 80 32.50000000 81 32.09876543 82 32.92682927 83 32.53012048 84 32.14285714 85 31.76470588 86 31.39534884 87 31.03448276 88 30.68181818 89 31.46067416 90 31.11111111 91 30.76923077 92 30.43478261 93 30.10752688 94 29.78723404 95 29.47368421 96 30.20833333 97 30.92783505 98 30.61224490 99 31.31313131 100 32.00000000 101 32.67326733 102 32.35294118 103 33.00970874 104 32.69230769 105 32.38095238 106 32.07547170 107 32.71028037 108 33.33333333 109 33.94495413 110 34.54545455 111 34.23423423 112 34.82142857 113 35.39823009 114 35.96491228 115 35.65217391 116 36.20689655 117 36.75213675 118 36.44067797 119 36.13445378 120 35.83333333 121 36.36363636 122 36.06557377 123 36.58536585 124 36.29032258 125 36.80000000 126 36.50793651 127 36.22047244 128 35.93750000 129 36.43410853 130 36.92307692 131 36.64122137 132 36.36363636 133 36.09022556 134 35.82089552 135 35.55555556 136 35.29411765 137 35.76642336 138 36.23188406 139 36.69064748 140 36.42857143 141 36.87943262 142 36.61971831 143 36.36363636 144 36.80555556 145 37.24137931 146 36.98630137 147 36.73469388 148 36.48648649 149 36.24161074 150 36.00000000 151 36.42384106 152 36.18421053 153 35.94771242 154 35.71428571 155 35.48387097 156 35.89743590 157 36.30573248 158 36.70886076 159 37.10691824 160 36.87500000 161 37.26708075 162 37.03703704 163 36.80981595 164 36.58536585 165 36.36363636 166 36.74698795 167 36.52694611 168 36.90476190 169 36.68639053 170 36.47058824 171 36.25730994 172 36.04651163 173 35.83815029 174 36.20689655 175 36.00000000 176 35.79545455 177 35.59322034 178 35.39325843 179 35.19553073 180 35.00000000 181 34.80662983 182 34.61538462 183 34.42622951 184 34.23913043 185 34.59459459 186 34.94623656 187 34.75935829 188 34.57446809 189 34.39153439 190 34.21052632 191 34.03141361 192 33.85416667 193 33.67875648 194 34.02061856 195 33.84615385 196 33.67346939 197 34.01015228 198 34.34343434 199 34.17085427 200 34.50000000 201 34.32835821 202 34.65346535 203 34.48275862 204 34.31372549 205 34.63414634 206 34.46601942 207 34.78260870 208 34.61538462 209 34.44976077 210 34.28571429 211 34.12322275 212 33.96226415 213 33.80281690 214 33.64485981 215 33.48837209 216 33.33333333 217 33.17972350 218 33.02752294 219 32.87671233 220 32.72727273 221 33.03167421 222 32.88288288 223 33.18385650 224 33.48214286 225 33.77777778 226 33.62831858 227 33.92070485 228 33.77192982 229 33.62445415 230 33.47826087 231 33.33333333 232 33.62068966 233 33.47639485 234 33.33333333 235 33.19148936 236 33.47457627 237 33.33333333 238 33.61344538 239 33.47280335 240 33.33333333 241 33.19502075 242 33.05785124 243 32.92181070 244 32.78688525 245 32.65306122 246 32.52032520 247 32.79352227 248 33.06451613 249 33.33333333 250 33.20000000 251 33.46613546 252 33.33333333 253 33.20158103 254 33.46456693 255 33.72549020 256 33.59375000 257 33.46303502 258 33.33333333 259 33.20463320 260 33.07692308 261 33.33333333 262 33.20610687 263 33.07984791 264 33.33333333 265 33.58490566 266 33.45864662 267 33.33333333 268 33.20895522 269 33.45724907 270 33.33333333 271 33.21033210 272 33.08823529 273 32.96703297 274 33.21167883 275 33.09090909 276 32.97101449 277 32.85198556 278 32.73381295 279 32.61648746 280 32.85714286 281 33.09608541 282 32.97872340 283 32.86219081 284 33.09859155 285 32.98245614 286 33.21678322 287 33.44947735 288 33.68055556 289 33.56401384 290 33.44827586 291 33.67697595 292 33.56164384 293 33.78839590 294 33.67346939 295 33.89830508 296 34.12162162 297 34.00673401 298 33.89261745 299 33.77926421 300 33.66666667 301 33.55481728 302 33.44370861 303 33.66336634 304 33.55263158 305 33.44262295 306 33.33333333 307 33.22475570 308 33.44155844 309 33.33333333 310 33.22580645 311 33.11897106 312 33.33333333 313 33.22683706 314 33.12101911 315 33.01587302 316 32.91139241 317 32.80757098 318 32.70440252 319 32.60188088 320 32.50000000 321 32.39875389 322 32.60869565 323 32.50773994 324 32.40740741 325 32.30769231 326 32.20858896 327 32.41590214 328 32.62195122 329 32.52279635 330 32.72727273 331 32.62839879 332 32.83132530 333 32.73273273 334 32.63473054 335 32.83582090 336 33.03571429 337 32.93768546 338 32.84023669 339 33.03834808 340 32.94117647 341 33.13782991 342 33.33333333 343 33.23615160 344 33.43023256 345 33.62318841 346 33.52601156 347 33.71757925 348 33.62068966 349 33.81088825 350 33.71428571 351 33.61823362 352 33.52272727 353 33.42776204 354 33.33333333 355 33.52112676 356 33.42696629 357 33.61344538 358 33.79888268 359 33.70473538 360 33.61111111 361 33.79501385 362 33.70165746 363 33.60881543 364 33.51648352 365 33.42465753 366 33.33333333 367 33.24250681 368 33.15217391 369 33.06233062 370 32.97297297 371 32.88409704 372 33.06451613 373 32.97587131 374 33.15508021 375 33.06666667 376 32.97872340 377 32.89124668 378 32.80423280 379 32.71767810 380 32.89473684 381 33.07086614 382 33.24607330 383 33.15926893 384 33.07291667 385 32.98701299 386 32.90155440 387 32.81653747 388 32.98969072 389 32.90488432 390 32.82051282 391 32.73657289 392 32.65306122 393 32.56997455 394 32.74111675 395 32.65822785 396 32.57575758 397 32.74559194 398 32.66331658 399 32.58145363 400 32.50000000 401 32.66832918 402 32.83582090 403 33.00248139 404 32.92079208 405 32.83950617 406 32.75862069 407 32.67813268 408 32.59803922 409 32.51833741 410 32.43902439 411 32.36009732 412 32.52427184 413 32.68765133 414 32.85024155 415 33.01204819 416 32.93269231 417 32.85371703 418 32.77511962 419 32.69689737 420 32.85714286 421 33.01662708 422 32.93838863 423 32.86052009 424 32.78301887 425 32.70588235 426 32.62910798 427 32.55269321 428 32.47663551 429 32.40093240 430 32.55813953 431 32.48259861 432 32.63888889 433 32.79445727 434 32.94930876 435 32.87356322 436 32.79816514 437 32.72311213 438 32.87671233 439 33.02961276 440 32.95454545 441 32.87981859 442 32.80542986 443 32.73137698 444 32.88288288 445 32.80898876 446 32.73542601 447 32.66219239 448 32.58928571 449 32.73942094 450 32.88888889 451 32.81596452 452 32.74336283 453 32.89183223 454 32.81938326 455 32.74725275 456 32.67543860 457 32.60393873 458 32.53275109 459 32.46187364 460 32.39130435 461 32.32104121 462 32.46753247 463 32.61339093 464 32.75862069 465 32.90322581 466 32.83261803 467 32.76231263 468 32.69230769 469 32.62260128 470 32.55319149 471 32.69639066 472 32.83898305 473 32.76955603 474 32.91139241 475 33.05263158 476 32.98319328 477 33.12368973 478 33.05439331 479 32.98538622 480 32.91666667 481 32.84823285 482 32.98755187 483 32.91925466 484 33.05785124 485 33.19587629 486 33.12757202 487 33.05954825 488 32.99180328 489 33.12883436 490 33.26530612 491 33.40122200 492 33.53658537 493 33.46855984 494 33.40080972 495 33.53535354 496 33.46774194 497 33.40040241 498 33.33333333 499 33.46693387 500 33.40000000 501 33.33333333 502 33.26693227 503 33.20079523 504 33.33333333 505 33.26732673 506 33.20158103 507 33.33333333 508 33.46456693 509 33.59528487 510 33.52941176 511 33.46379648 512 33.39843750 513 33.33333333 514 33.26848249 515 33.39805825 516 33.33333333 517 33.46228240 518 33.39768340 519 33.33333333 520 33.46153846 521 33.39731286 522 33.33333333 523 33.26959847 524 33.20610687 525 33.33333333 526 33.26996198 527 33.39658444 528 33.33333333 529 33.45935728 530 33.39622642 531 33.33333333 532 33.45864662 533 33.58348968 534 33.52059925 535 33.45794393 536 33.39552239 537 33.51955307 538 33.45724907 539 33.39517625 540 33.33333333 541 33.45656192 542 33.39483395 543 33.33333333 544 33.45588235 545 33.57798165 546 33.51648352 547 33.63802559 548 33.75912409 549 33.69763206 550 33.63636364 551 33.57531760 552 33.69565217 553 33.81555154 554 33.75451264 555 33.87387387 556 33.81294964 557 33.75224417 558 33.69175627 559 33.63148479 560 33.75000000 561 33.86809269 562 33.80782918 563 33.74777975 564 33.86524823 565 33.98230088 566 34.09893993 567 34.03880071 568 34.15492958 569 34.27065026 570 34.21052632 571 34.32574431 572 34.26573427 573 34.20593368 574 34.14634146 575 34.26086957 576 34.20138889 577 34.31542461 578 34.42906574 579 34.54231434 580 34.65517241 581 34.59552496 582 34.70790378 583 34.64837050 584 34.76027397 585 34.70085470 586 34.81228669 587 34.75298126 588 34.69387755 589 34.63497453 590 34.57627119 591 34.51776650 592 34.45945946 593 34.56998314 594 34.68013468 595 34.62184874 596 34.56375839 597 34.67336683 598 34.61538462 599 34.55759599 600 34.50000000 601 34.60898502 602 34.71760797 603 34.82587065 604 34.76821192 605 34.87603306 606 34.98349835 607 34.92586491 608 34.86842105 609 34.97536946 610 34.91803279 611 34.86088380 612 34.96732026 613 34.91027732 614 34.85342020 615 34.79674797 616 34.90259740 617 35.00810373 618 34.95145631 619 35.05654281 620 35.00000000 621 34.94363929 622 35.04823151 623 34.99197432 624 34.93589744 625 34.88000000 626 34.82428115 627 34.92822967 628 35.03184713 629 35.13513514 630 35.07936508 631 35.18225040 632 35.12658228 633 35.22906793 634 35.33123028 635 35.27559055 636 35.22012579 637 35.16483516 638 35.10971787 639 35.05477308 640 35.15625000 641 35.10140406 642 35.20249221 643 35.14774495 644 35.09316770 645 35.19379845 646 35.13931889 647 35.23956723 648 35.18518519 649 35.13097072 650 35.23076923 651 35.33026114 652 35.42944785 653 35.52833078 654 35.47400612 655 35.41984733 656 35.36585366 657 35.46423135 658 35.41033435 659 35.35660091 660 35.30303030 661 35.24962179 662 35.19637462 663 35.29411765 664 35.24096386 665 35.18796992 666 35.28528529 667 35.23238381 668 35.17964072 669 35.27653214 670 35.37313433 671 35.32041729 672 35.26785714 673 35.21545319 674 35.16320475 675 35.25925926 676 35.35502959 677 35.30280650 678 35.25073746 679 35.19882180 680 35.29411765 681 35.38913363 682 35.33724340 683 35.28550512 684 35.38011696 685 35.32846715 686 35.27696793 687 35.22561863 688 35.31976744 689 35.41364296 690 35.36231884 691 35.45586107 692 35.54913295 693 35.49783550 694 35.44668588 695 35.39568345 696 35.48850575 697 35.43758967 698 35.38681948 699 35.33619456 700 35.42857143 701 35.37803138 702 35.32763533 703 35.27738265 704 35.22727273 705 35.17730496 706 35.26912181 707 35.36067893 708 35.31073446 709 35.26093089 710 35.21126761 711 35.30239100 712 35.25280899 713 35.20336606 714 35.29411765 715 35.38461538 716 35.33519553 717 35.42538354 718 35.37604457 719 35.46592490 720 35.41666667 721 35.36754508 722 35.31855956 723 35.26970954 724 35.22099448 725 35.17241379 726 35.26170799 727 35.21320495 728 35.16483516 729 35.25377229 730 35.34246575 731 35.29411765 732 35.38251366 733 35.33424284 734 35.28610354 735 35.23809524 736 35.19021739 737 35.27815468 738 35.23035230 739 35.31799729 740 35.27027027 741 35.35762483 742 35.30997305 743 35.26244953 744 35.34946237 745 35.30201342 746 35.25469169 747 35.34136546 748 35.29411765 749 35.38050734 750 35.46666667 Final result: 35.4667 +/- 1.7481 Random chance: 19.8992 +/- 1.4588