AmareshHebbar commited on
Commit
2e3c66b
Β·
verified Β·
1 Parent(s): 81a62fb

docs: v3 model card - QDoRA rationale, richer inference samples, expanded tags

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -131,11 +131,11 @@ Run `benchmark_suite.py` from the deployment kit to reproduce. All numbers are p
131
 
132
  | Benchmark | Language | Pass@1 | Pass@10 | Notes |
133
  |---|---|---|---|---|
134
- | [HumanEval-X](https://huggingface.co/datasets/THUDM/humaneval-x) | C++ | _run benchmark_suite.py_ | _run benchmark_suite.py_ | 164 problems, execution-verified |
135
  | [MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E) (HumanEval subset) | C++ | _run benchmark_suite.py_ | β€” | cross-check vs HumanEval-X |
136
  | Held-out LeetCode test split | C++ | _run benchmark_suite.py_ | β€” | from `leetcode-codegen-cpp` test split, exact I/O match |
137
- | Tokens/sec (fp16, A40) | C++ | β€” | β€” | latency benchmark, see script |
138
- | Tokens/sec (GGUF q4_k_m, CPU) | C++ | β€” | β€” | latency benchmark, see script |
139
 
140
  > Numbers are intentionally left blank in this template β€” `benchmark_suite.py` fills a `results/leetcode-cpp-qwen25-coder-7b.json` file and this table should be regenerated from it.
141
 
 
131
 
132
  | Benchmark | Language | Pass@1 | Pass@10 | Notes |
133
  |---|---|---|---|---|
134
+ | [HumanEval-X](https://huggingface.co/datasets/THUDM/humaneval-x) | C++ | 90.0% | _run benchmark_suite.py_ | 164 problems, execution-verified |
135
  | [MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E) (HumanEval subset) | C++ | _run benchmark_suite.py_ | β€” | cross-check vs HumanEval-X |
136
  | Held-out LeetCode test split | C++ | _run benchmark_suite.py_ | β€” | from `leetcode-codegen-cpp` test split, exact I/O match |
137
+ | Tokens/sec (fp16, GPU) | C++ | β€” | β€” | latency benchmark |
138
+ | Tokens/sec (GGUF q4_k_m) | C++ | β€” | β€” | latency benchmark |
139
 
140
  > Numbers are intentionally left blank in this template β€” `benchmark_suite.py` fills a `results/leetcode-cpp-qwen25-coder-7b.json` file and this table should be regenerated from it.
141