Update README.md

README.md

@@ -70,68 +70,72 @@ Kimi K2 Thinking is the latest, most capable version of open-source thinking mod
 | Benchmark | Setting | K2 Thinking | GPT-5 | Claude Sonnet 4.5 (Thinking) | K2 0905 | DeepSeek-V3.2 | Grok-4 |
 |:----------|:--------|------------:|------:|----------------------------:|--------:|--------------:|-------:|
 | **HLE (Text-only)** | no tools | 23.9 | 26.3 | 19.8* | 7.9 | 19.8 | 25.4 |
-| | w/ tools | 44.9 | 35.2 | 32* | 21.7 | 20.3* | 38.6 |
-| | heavy | 51 | 42 | N/A | N/A | N/A | 50.7 |
-| **AIME25** | no tools | 94.5 | 94.6 | 87 | 51 | 89.3 | 91.7 |
-| | w/ python | 99.1 | 99.6 | 100 | 75.2 | 58.1* | 98.8 |
-| | heavy | 100 | 100 | N/A | N/A | N/A | 100 |
-| **HMMT25** | no tools | 89.4 | 93.3 | 74.6* | 38.8 | 83.6 | 90 |
+| | w/ tools | 44.9 | 41.7* | 32.0* | 21.7 | 20.3* | 41.0 |
+| | heavy | 51.0 | 42.0 | N/A | N/A | N/A | 50.7 |
+| **AIME25** | no tools | 94.5 | 94.6 | 87.0 | 51 | 89.3 | 91.7 |
+| | w/ python | 99.1 | 99.6 | 100.0 | 75.2 | 58.1* | 98.8 |
+| | heavy | 100.0 | 100.0 | N/A | N/A | N/A | 100.0 |
+| **HMMT25** | no tools | 89.4 | 93.3 | 74.6* | 38.8 | 83.6 | 90.0 |
 | | w/ python | 95.1 | 96.7 | 88.8* | 70.4 | 49.5* | 93.9 |
-| | heavy | 97.5 | 100 | N/A | N/A | N/A | 96.7 |
+| | heavy | 97.5 | 100.0 | N/A | N/A | N/A | 96.7 |
 | **IMO-AnswerBench** | no tools | 78.6 | 76.0* | 65.9* | 45.8 | 76.0* | 73.1 |
 | **GPQA** | no tools | 84.5 | 85.7 | 83.4 | 74.2 | 79.9 | 87.5 |
 
 **General Tasks**
 | Benchmark | Setting | K2 Thinking | GPT-5 | Claude Sonnet 4.5 (Thinking) | K2 0905 | DeepSeek-V3.2 |
 |:----------|:--------|------------:|------:|----------------------------:|--------:|--------------:|
-| **MMLU-Pro** | no tools | 84.6 | 87.1 | 87.5 | 81.9 | 85 |
+| **MMLU-Pro** | no tools | 84.6 | 87.1 | 87.5 | 81.9 | 85.0 |
 | **MMLU-Redux** | no tools | 94.4 | 95.3 | 95.6 | 92.7 | 93.7 |
 | **Longform Writing** | no tools | 73.8 | 71.4 | 79.8 | 62.8 | 72.5 |
-| **HealthBench** | no tools | 58 | 67.2 | 44.2 | 43.8 | 46.9 |
+| **HealthBench** | no tools | 58.0 | 67.2 | 44.2 | 43.8 | 46.9 |
 
 **Agentic Search Tasks**
 | Benchmark | Setting | K2 Thinking | GPT-5 | Claude Sonnet 4.5 (Thinking) | K2 0905 | DeepSeek-V3.2 |
 |:----------|:--------|------------:|------:|----------------------------:|--------:|--------------:|
 | **BrowseComp** | w/ tools | 60.2 | 54.9 | 24.1 | 7.4 | 40.1 |
-| **BrowseComp-ZH** | w/ tools | 62.3 | 63* | 42.4* | 22.2 | 47.9 |
+| **BrowseComp-ZH** | w/ tools | 62.3 | 63.0* | 42.4* | 22.2 | 47.9 |
 | **Seal-0** | w/ tools | 56.3 | 51.4* | 53.4* | 25.2 | 38.5* |
-| **Frames** | w/ tools | 87 | 86* | 85* | 58.1 | 80.2* |
+| **Frames** | w/ tools | 87.0 | 86.0* | 85.0* | 58.1 | 80.2* |
 
 **Coding Tasks**
 | Benchmark | Setting | K2 Thinking | GPT-5 | Claude Sonnet 4.5 (Thinking) | K2 0905 | DeepSeek-V3.2 |
 |:----------|:--------|------------:|------:|----------------------------:|--------:|--------------:|
 | **SWE-bench Verified** | w/ tools | 71.3 | 74.9 | 77.2 | 69.2 | 67.8 |
-| **SWE-bench Multilingual** | w/ tools | 61.1 | 55.3* | 68 | 55.9 | 57.9 |
+| **SWE-bench Multilingual** | w/ tools | 61.1 | 55.3* | 68.0 | 55.9 | 57.9 |
 | **Multi-SWE-bench** | w/ tools | 41.9 | 39.3* | 44.3 | 33.5 | 30.6 |
 | **SciCode** | no tools | 44.8 | 42.9 | 44.7 | 30.7 | 37.7 |
-| **LiveCodeBenchV6** | no tools | 83.1 | 87* | 64* | 56.1* | 74.1 |
+| **LiveCodeBenchV6** | no tools | 83.1 | 87.0* | 64.0* | 56.1* | 74.1 |
 | **OJ-Bench (cpp)** | no tools | 48.7 | 56.2* | 30.4* | 25.5* | 38.2* |
-| **Terminal-Bench** | w/ simulated tools (JSON) | 47.1 | 43.8 | 51 | 44.5 | 37.7 |
+| **Terminal-Bench** | w/ simulated tools (JSON) | 47.1 | 43.8 | 51.0 | 44.5 | 37.7 |
 <details>
 <summary><b>Footnotes</b></summary>
 
-1. All benchmarks were evaluated at temperature = 1.0 and 256 k context length for k2-thinking, except for SciCode, for which we followed the official temperature setting of 0.0.
+1. To ensure a fast, lightweight experience, we selectively employ a subset of tools and reduce the number of tool call turns under the chat mode on kimi.com. As a result, chatting on kimi.com may not reproduce our benchmark scores. Our agentic mode will be updated soon to reflect the full capabilities of K2 Thinking.
 
-2. For HLE (w/ tools) and the agentic-search benchmarks (BrowseComp, BrowseComp-zh, Frames, and Seal-0), k2-thinking was equipped with search, code-interpreter, and web-browsing tools. Any benchmark containing fewer than 300 questions was run 4 times independently and the average is reported (avg@4). The evaluation used o3-mini as judge, configured identically to the official HLE setting; judge prompts were taken verbatim from the official repository.
+2. **Testing Details**:  
+ 2.1. All benchmarks were evaluated at temperature = 1.0 and 256 k context length for K2 Thinking, except for SciCode, for which we followed the official temperature setting of 0.0.  
+ 2.2. HLE (no tools), AIME25, HMMT25, and GPQA were capped at a 96k thinking-token budget, while IMO-Answer Bench, LiveCodeBench and OJ-Bench were capped at a 128k thinking-token budget. Longform Writing was capped at a 32k completion-token budget.  
+ 2.3. For AIME and HMMT (no tools), we report the average of 32 runs (avg@32). For AIME and HMMT (with Python), we report the average of 16 runs (avg@16). For IMO-AnswerBench, we report the average of 8 runs (avg@8).
 
-3. We report the official scores for GPT-5 and Grok-4 on the HLE full set with tools. In our internal evaluation on the HLE text-only subset, GPT-5 scores 41.7 and Grok-4 scores 38.6（Grok-4's launch cited 41.0 on the text-only subset）.For GPT-5's HLE w/o tool, we use the score from <a href="https://scale.com/leaderboard/humanitys_last_exam_text_only" target="_blank">Scale.ai</a>. The official GPT-5 HLE full set w/o tool is 24.8.
+3. **Baselines**:  
+ 3.1 GPT-5, Claude-4.5-sonnet, Grok-4 results and DeepSeek-V3.2 results are quoted from the [GPT-5 post](https://openai.com/index/introducing-gpt-5/), [GPT-5 for Developers post](https://openai.com/index/introducing-gpt-5-for-developers/), [GPT-5 system card](https://openai.com/index/gpt-5-system-card/), [claude-sonnet-4-5 post](https://www.anthropic.com/news/claude-sonnet-4-5), [grok-4 post](https://x.ai/news/grok-4), [deepseek-v3.2 post](https://api-docs.deepseek.com/news/news250929), the [public Terminal-Bench leaderboard](https://www.tbench.ai/leaderboard) (Terminus-2), the [public Vals AI leaderboard](https://vals.ai/) and [artificialanalysis](https://artificialanalysis.ai/). Benchmarks for which no available public scores were re-tested under the same conditions used for k2 thinking and are marked with an asterisk(*).  
+ 3.2 The GPT-5 and Grok-4 on the HLE full set with tools are 35.2 and 38.6 from the official posts. In our internal evaluation on the HLE text-only subset, GPT-5 scores 41.7 and Grok-4 scores 38.6 (Grok-4’s launch cited 41.0 on the text-only subset). For GPT-5's HLE text-only w/o tool, we use score from <a href="https://scale.com/leaderboard/humanitys_last_exam_text_only" target="_blank">Scale.ai</a>. The official GPT5 HLE full set w/o tool is 24.8.  
+ 3.3 For <a href="https://aclanthology.org/2025.emnlp-main.1794.pdf" target="_blank">IMO-AnswerBench</a>: GPT-5 scored 65.6 in the benchmark paper. We re-evaluated GPT-5 with official API and obtained a score of 76.
 
-4. On HLE, the maximum turn limit was 120, with a 48 k-token thinking budget per turn; on agentic-search tasks, the limit was 300 turns with a 24 k-token thinking budget per turn. When tool execution results cause the accumulated input to exceed the model's context limit (256k), we employ a simple context management strategy that hides all previous tool outputs.
+4. **For HLE (w/ tools) and the agentic-search benchmarks**:  
+ 4.1. K2 Thinking was equipped with search, code-interpreter, and web-browsing tools.  
+ 4.2. BrowseComp-ZH, Seal-0 were run 4 times independently and the average is reported (avg@4).  
+ 4.3. The evaluation used o3-mini as judge, configured identically to the official HLE setting; judge prompts were taken verbatim from the official repository.  
+ 4.4. On HLE, the maximum turn limit was 120, with a 48 k-token reasoning budget per turn; on agentic-search tasks, the limit was 300 turns with a 24 k-token reasoning budget per turn.  
+ 4.5. When tool execution results cause the accumulated input to exceed the model's context limit (256k), we employ a simple context management strategy that hides all previous tool outputs.  
+ 4.6. The web access to Hugging Face may lead to data leakage in certain benchmark tests, such as HLE. K2 Thinking can achieve a score of 51.3 on HLE without blocking Hugging Face. To ensure a fair and rigorous comparison, we blocked access to Hugging Face during testing.
 
-5. The web access to Hugging Face may lead to data leakage in certain benchmark tests, such as HLE. K2 Thinking can achieve a score of 51.3 on HLE without blocking Hugging Face. To ensure a fair and rigorous comparison, we blocked access to Hugging Face through search or web-browsing tools during testing to prevent potential data leakage.
-
-6. HLE (no tools), AIME25, HMMT25, and GPQA were capped at a 96k thinking-token budget, while IMO-Answer Bench, LiveCodeBench and OJ-Bench were capped at a 128k thinking-token budget. Longform Writing was capped at a 32k completion-token budget.
-
-7. For AIME and HMMT (no tools), we test 32 times and report the average over 32 runs (avg@32). For AIME and HMMT (with Python), we test 16 times and report the average over 16 runs (avg@16). For IMO-AnswerBench, we test 8 times and report the average over 8 runs (avg@8).
-
-8. IMO-AnswerBench: In the <a href="https://aclanthology.org/2025.emnlp-main.1794.pdf" target="_blank">benchmark paper</a>, GPT-5 scored 65.6. We re-evaluated GPT-5 with official API and obtained a score of 76.
-
-9. Terminal-Bench scores were obtained with the default agent framework (Terminus-2) and the provided JSON parser. For other coding tasks, the result was produced with our in-house evaluation harness. The harness is derived from SWE-agent, but we clamp the context windows of the Bash and Edit tools and rewrite the system prompt to match the task semantics. All reported scores of coding tasks are averaged over 5 independent runs.
-
-10. GPT-5, Claude-4.5-sonnet, Grok-4 results and DeepSeek-V3.2 results are quoted from the [GPT-5 post](https://openai.com/index/introducing-gpt-5/), [GPT-5 for Developers post](https://openai.com/index/introducing-gpt-5-for-developers/), [GPT-5 system card](https://openai.com/index/gpt-5-system-card/), [claude-sonnet-4-5 post](https://www.anthropic.com/news/claude-sonnet-4-5), [grok-4 post](https://x.ai/news/grok-4), [deepseek-v3.2 post](https://api-docs.deepseek.com/news/news250929), the [public Terminal-Bench leaderboard](https://www.tbench.ai/leaderboard) (Terminus-2), the [public Vals AI leaderboard](https://vals.ai/) and [artificialanalysis](https://artificialanalysis.ai/). Benchmarks for which no available public scores were re-tested under the same conditions used for k2 thinking and are marked with an asterisk(*).
-
-11. K2 Thinking Heavy Mode employs an efficient parallel strategy: it first rolls out eight trajectories simultaneously, then reflectively aggregates all outputs to generate the final result. Heavy mode for GPT-5 denotes the official GPT-5 Pro score.
+5. **For Coding Tasks**:  
+ 5.1. Terminal-Bench scores were obtained with the default agent framework (Terminus-2) and the provided JSON parser.  
+ 5.2. For other coding tasks, the result was produced with our in-house evaluation harness. The harness is derived from SWE-agent, but we clamp the context windows of the Bash and Edit tools and rewrite the system prompt to match the task semantics.  
+ 5.3. All reported scores of coding tasks are averaged over 5 independent runs.
 
+6. **Heavy Mode**: K2 Thinking Heavy Mode employs an efficient parallel strategy: it first rolls out eight trajectories simultaneously, then reflectively aggregates all outputs to generate the final result. Heavy mode for GPT-5 denotes the official GPT-5 Pro score.
 </details>
 
 ## 4. Native INT4 Quantization