Request for the evaluation protocol behind the model-card benchmark table
I reviewed the benchmark table at revision 38313f4a9aa853c15f47027e646dd84088fe7e4d. It reports ELYZA-tasks-100, Japanese MT-Bench, and English MT-Bench scores, but I could not find enough protocol detail to reproduce the numbers independently.
Could you point to, or add, the exact dataset versions/subsets, evaluator model and version, judge prompts/rubric, model prompt templates and revisions, inference/decoding settings, number of runs/seeds, aggregation method, and evaluation harness commit?
A small revision-pinned evaluation manifest would be sufficient; I am not asking for an evaluation rerun or a score change. If the original configuration is no longer available, labeling the table as historical/non-reproducible would also clarify how readers should use it.
This investigation and draft used AI assistance. I verified the listed omissions against the reviewed card revision.
Thank you very much for bringing this to our attention.
We apologize for the omission of evaluation protocol details on the Model Card.
All evaluation configs, prompt templates, judge settings, and execution scripts are available in our flexeval repository:
https://github.com/sbintuitions/flexeval/tree/main/examples/sarashina_2_2_evaluation
Here are the key protocol details addressing your questions:
- Configuration Files: Prompt templates, judge rubrics, and decoding parameters (e.g.,
temperature,top_p) are located under:./configs/instruction_models&./configs/instruction_evals(for instruction models)./configs/pretrained_models&./configs/pretrained_evals(for base models)
- Evaluator Model & Version:
gpt-4o-2024-08-06(via OpenAI API). - Execution & Aggregation: For instruction models, each reported score is the average across 9 evaluation runs (3 model generations × 3 judge evaluations per output).
We have updated the Hugging Face Model Card (README) with a direct link to this evaluation manifest.
Thank you again for your helpful feedback!