Add community evaluation results

#5
by SaylorTwift HF Staff - opened

Summary

This PR adds evaluation results extracted from the model card's benchmark charts and appendix tables (images, not text tables) to .eval_results/, following the Hugging Face Hub evaluation-results specification.

Benchmarks Added

Mapping Notes

  • Claw-Eval: the card reports one aggregate Pass³ score (N=3 trials) without a general/multimodal/multi_turn split; mapped to the general task as the most likely default.
  • SWE-bench suite (Verified, Multilingual, Pro): all three were run with the model's own "live-swe-agent" scaffold, temperature=1.0, top_p=0.95, max_new_tokens=32K, 384K context — noted once here rather than on each entry since it's constant across the three.
  • SkillsBench: 52.8 is the with-skills condition, not the with/without-skills lift (SkillsBench's other headline metric).

Benchmarks Skipped (Not Registered on Hub)

These appeared on the model card but have no registered eval.yaml on the Hub, so they were left out of the YAML:

IMOAnswerBench (90.9), Codeforces (3056 Elo), LiveCodeBench v6 (91.5), ARC-AGI-2 (81.4), ARC-AGI-3 arcagi3 harness (6.9), ARC-AGI-3 general harness (32.1), IFBench (80.4), IFEval (93.9), VibeLifeBench (28.1), NL2repo (49.8), Toolathlon-Verified (55.6), BrowseComp w/ CM (83.3), BrowseComp-zh (75.8), LiveBrowseComp (46.9), WideSearch (78.9), DeepSearchQA (92.1), VibeSearchBench (25.7), WorldVQA (42.7), SimpleVQA (72.5), MathVision (87.7), HiPho (74.8), ZeroBench@5 (19.0), CharxivReasoning (83.1), GDP.pdf mean-criteria (60.7), PerceptionBench (53.4), BabyVision (50.0), MMVU (79.9), VideoMMMU (86.8), LongVideoBench (76.8), VideoZeroBench (17.4), VoiceBench (92.4), AudioMC (46.9), MMAU-PRO (69.8).

These can be added once the benchmark authors register an eval.yaml on the Hub.

Also Considered

  • Terminal-Bench (card value 75.1, registered as harborframework/terminal-bench-2.0 / terminalbench_2) was left out of this PR.

Source

  • Model card: https://huggingface.co/dots-studio/dots3-note-prev
  • Benchmark images: assets/bench_en1.png, assets/bench_en2.png, assets/benchmark_appendix_en_reasoning.png, assets/benchmark_appendix_en_multimodal.png
  • Paper: not yet published ("Full Report (coming soon)" per the model card)

Verification

These results were read directly off the model card's published benchmark charts and appendix tables (visual read, not text extraction). No verified token is attached, as these were not run via HF Jobs with inspect-ai.

Ready to merge
This branch is ready to get merged automatically.

Sign up or log in to comment