zechen-nlp commited on
Commit
84449d4
·
verified ·
1 Parent(s): 3bb04e9

Update Automated MNLP evaluation report (2026-05-28)

Browse files
Files changed (1) hide show
  1. EVAL_REPORT.md +5 -75
EVAL_REPORT.md CHANGED
@@ -2,7 +2,7 @@
2
 
3
  - **Model repo:** [`cs-552-2026-vibe-trainers/general_knowledge_model`](https://huggingface.co/cs-552-2026-vibe-trainers/general_knowledge_model)
4
  - **Owner(s):** group **vibe-trainers**
5
- - **Generated at:** 2026-05-27T13:02:22+00:00 (UTC)
6
  - **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
7
 
8
  _This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
@@ -12,7 +12,7 @@ _This PR is opened automatically by the course CI. It is **non-blocking** — yo
12
  | Benchmark | Accuracy | Status |
13
  |---|---:|---|
14
  | Math | — | not run |
15
- | Knowledge | 0.2700 | ok |
16
  | Multilingual | — | not run |
17
  | Safety | — | not run |
18
 
@@ -35,81 +35,11 @@ _Prompts are intentionally omitted to avoid revealing benchmark contents. For mu
35
 
36
  **Incorrect** (1 shown)
37
 
38
- - **reference**: `I`
39
  - **overall** (0/1 completions correct)
40
- - **extracted** (✗): `A`
41
  - **completion**:
42
 
43
  ```text
44
- To solve this problem, we are given a **simple random walk** $ S_n $ starting at $ S_0 = 0 $, with the following transition probabilities:
45
-
46
- - $ P(S_n - S_{n-1} = 1) = \frac{1}{4} $
47
- - $ P(S_n - S_{n-1} = -1) = \frac{3}{4} $
48
-
49
- We are to find the probability that the **maximum** of the walk $ M = \max\{S_n : n \geq 0\} $ is **at least 5**, i.e., $ P(M \geq 5) $.
50
-
51
- ---
52
-
53
- ### Step 1: Understand the proces
54
-
55
- ... [truncated 822 chars] ...
56
-
57
- :
58
-
59
- $$
60
- P_k = \frac{1}{4} P_{k+1} + \frac{3}{4} P_{k-1}
61
- $$
62
-
63
- We can compute this using a **forward pass** from $ k = 5 $ to $ k = 0 $, with boundary conditions:
64
-
65
- - $ P_5 = 1 $ (if the walk is already at 5, it has reached the maximum)
66
- - $ P_0 = 0 $ (if the walk starts at 0, it has not yet reached 5)
67
-
68
- We can compute the values step-by-step.
69
-
70
- ---
71
-
72
- ### Step 5: Compute the values
73
-
74
- Let’s compute the values from $ k = 5 $ down to $ k = 0 $, using the recurrence:
75
-
76
- $$
77
- P_k = \frac{1}{4} P_{k+1} + \frac{3}{4} P_{k-1}
78
- $$
79
-
80
- We can compute this using a **forward pass** from $ k = 5 $ to $ k = 0 $, with the boundary conditions:
81
-
82
- - $ P_5 = 1 $
83
- - $ P_0 = 0 $
84
-
85
- We can compute the values for $ k = 4, 3, 2, 1, 0 $.
86
-
87
- Let’s compute them:
88
-
89
- - $ P_5 = 1 $
90
- - $ P_4 = \frac{1}{4} P_5 + \frac{3}{4} P_3 = \frac{1}{4} \cdot 1 + \frac{3}{4} P_3 $
91
- - $ P_3 = \frac{1}{4} P_4 + \frac{3}{4} P_2 $
92
- - $ P_2 = \frac{1}{4} P_3 + \frac{3}{4} P_1 $
93
- - $ P_1 = \frac{1}{4} P_2 + \frac{3}{4} P_0 = \frac{1}{4} P_2 + 0 $
94
- - $ P_0 = 0 $
95
-
96
- We can solve this system of equations step-by-step.
97
-
98
- ---
99
-
100
- ### Step 6: Final computation
101
-
102
- After solving the system, we find that:
103
-
104
- $$
105
- P_0 = \boxed{0.00123456}
106
- $$
107
-
108
- ---
109
-
110
- ### ✅ Final Answer:
111
-
112
- $$
113
- \boxed{A}
114
- $$
115
  ```
 
2
 
3
  - **Model repo:** [`cs-552-2026-vibe-trainers/general_knowledge_model`](https://huggingface.co/cs-552-2026-vibe-trainers/general_knowledge_model)
4
  - **Owner(s):** group **vibe-trainers**
5
+ - **Generated at:** 2026-05-28T08:03:33+00:00 (UTC)
6
  - **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
7
 
8
  _This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
 
12
  | Benchmark | Accuracy | Status |
13
  |---|---:|---|
14
  | Math | — | not run |
15
+ | Knowledge | 0.3300 | ok |
16
  | Multilingual | — | not run |
17
  | Safety | — | not run |
18
 
 
35
 
36
  **Incorrect** (1 shown)
37
 
38
+ - **reference**: `F`
39
  - **overall** (0/1 completions correct)
40
+ - **extracted** (✗): `C`
41
  - **completion**:
42
 
43
  ```text
44
+ \boxed{C}
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
45
  ```