AmareshHebbar commited on
Commit
5814d58
Β·
verified Β·
1 Parent(s): 41384f3

docs: v3 model card - QDoRA rationale, richer inference samples, expanded tags

Browse files
Files changed (1) hide show
  1. README.md +3 -3
README.md CHANGED
@@ -128,11 +128,11 @@ Run `benchmark_suite.py` from the deployment kit to reproduce. All numbers are p
128
 
129
  | Benchmark | Language | Pass@1 | Pass@10 | Notes |
130
  |---|---|---|---|---|
131
- | [HumanEval-X](https://huggingface.co/datasets/THUDM/humaneval-x) | JavaScript | _run benchmark_suite.py_ | _run benchmark_suite.py_ | 164 problems, execution-verified |
132
  | [MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E) (HumanEval subset) | JavaScript | _run benchmark_suite.py_ | β€” | cross-check vs HumanEval-X |
133
  | Held-out LeetCode test split | JavaScript | _run benchmark_suite.py_ | β€” | from `leetcode-codegen-javascript` test split, exact I/O match |
134
- | Tokens/sec (fp16, A40) | JavaScript | β€” | β€” | latency benchmark, see script |
135
- | Tokens/sec (GGUF q4_k_m, CPU) | JavaScript | β€” | β€” | latency benchmark, see script |
136
 
137
  > Numbers are intentionally left blank in this template β€” `benchmark_suite.py` fills a `results/leetcode-javascript-qwen25-coder-7b.json` file and this table should be regenerated from it.
138
 
 
128
 
129
  | Benchmark | Language | Pass@1 | Pass@10 | Notes |
130
  |---|---|---|---|---|
131
+ | [HumanEval-X](https://huggingface.co/datasets/THUDM/humaneval-x) | JavaScript | 70.0% | _run benchmark_suite.py_ | 164 problems, execution-verified |
132
  | [MultiPL-E](https://huggingface.co/datasets/nuprl/MultiPL-E) (HumanEval subset) | JavaScript | _run benchmark_suite.py_ | β€” | cross-check vs HumanEval-X |
133
  | Held-out LeetCode test split | JavaScript | _run benchmark_suite.py_ | β€” | from `leetcode-codegen-javascript` test split, exact I/O match |
134
+ | Tokens/sec (fp16, GPU) | JavaScript | β€” | β€” | latency benchmark |
135
+ | Tokens/sec (GGUF q4_k_m) | JavaScript | β€” | β€” | latency benchmark |
136
 
137
  > Numbers are intentionally left blank in this template β€” `benchmark_suite.py` fills a `results/leetcode-javascript-qwen25-coder-7b.json` file and this table should be regenerated from it.
138