Update Automated MNLP evaluation report (2026-05-28)
Browse files- EVAL_REPORT.md +5 -75
EVAL_REPORT.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
| 2 |
|
| 3 |
- **Model repo:** [`cs-552-2026-vibe-trainers/general_knowledge_model`](https://huggingface.co/cs-552-2026-vibe-trainers/general_knowledge_model)
|
| 4 |
- **Owner(s):** group **vibe-trainers**
|
| 5 |
-
- **Generated at:** 2026-05-
|
| 6 |
- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
|
| 7 |
|
| 8 |
_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
|
|
@@ -12,7 +12,7 @@ _This PR is opened automatically by the course CI. It is **non-blocking** — yo
|
|
| 12 |
| Benchmark | Accuracy | Status |
|
| 13 |
|---|---:|---|
|
| 14 |
| Math | — | not run |
|
| 15 |
-
| Knowledge | 0.
|
| 16 |
| Multilingual | — | not run |
|
| 17 |
| Safety | — | not run |
|
| 18 |
|
|
@@ -35,81 +35,11 @@ _Prompts are intentionally omitted to avoid revealing benchmark contents. For mu
|
|
| 35 |
|
| 36 |
**Incorrect** (1 shown)
|
| 37 |
|
| 38 |
-
- **reference**: `
|
| 39 |
- **overall** (0/1 completions correct)
|
| 40 |
-
- **extracted** (✗): `
|
| 41 |
- **completion**:
|
| 42 |
|
| 43 |
```text
|
| 44 |
-
|
| 45 |
-
|
| 46 |
-
- $ P(S_n - S_{n-1} = 1) = \frac{1}{4} $
|
| 47 |
-
- $ P(S_n - S_{n-1} = -1) = \frac{3}{4} $
|
| 48 |
-
|
| 49 |
-
We are to find the probability that the **maximum** of the walk $ M = \max\{S_n : n \geq 0\} $ is **at least 5**, i.e., $ P(M \geq 5) $.
|
| 50 |
-
|
| 51 |
-
---
|
| 52 |
-
|
| 53 |
-
### Step 1: Understand the proces
|
| 54 |
-
|
| 55 |
-
... [truncated 822 chars] ...
|
| 56 |
-
|
| 57 |
-
:
|
| 58 |
-
|
| 59 |
-
$$
|
| 60 |
-
P_k = \frac{1}{4} P_{k+1} + \frac{3}{4} P_{k-1}
|
| 61 |
-
$$
|
| 62 |
-
|
| 63 |
-
We can compute this using a **forward pass** from $ k = 5 $ to $ k = 0 $, with boundary conditions:
|
| 64 |
-
|
| 65 |
-
- $ P_5 = 1 $ (if the walk is already at 5, it has reached the maximum)
|
| 66 |
-
- $ P_0 = 0 $ (if the walk starts at 0, it has not yet reached 5)
|
| 67 |
-
|
| 68 |
-
We can compute the values step-by-step.
|
| 69 |
-
|
| 70 |
-
---
|
| 71 |
-
|
| 72 |
-
### Step 5: Compute the values
|
| 73 |
-
|
| 74 |
-
Let’s compute the values from $ k = 5 $ down to $ k = 0 $, using the recurrence:
|
| 75 |
-
|
| 76 |
-
$$
|
| 77 |
-
P_k = \frac{1}{4} P_{k+1} + \frac{3}{4} P_{k-1}
|
| 78 |
-
$$
|
| 79 |
-
|
| 80 |
-
We can compute this using a **forward pass** from $ k = 5 $ to $ k = 0 $, with the boundary conditions:
|
| 81 |
-
|
| 82 |
-
- $ P_5 = 1 $
|
| 83 |
-
- $ P_0 = 0 $
|
| 84 |
-
|
| 85 |
-
We can compute the values for $ k = 4, 3, 2, 1, 0 $.
|
| 86 |
-
|
| 87 |
-
Let’s compute them:
|
| 88 |
-
|
| 89 |
-
- $ P_5 = 1 $
|
| 90 |
-
- $ P_4 = \frac{1}{4} P_5 + \frac{3}{4} P_3 = \frac{1}{4} \cdot 1 + \frac{3}{4} P_3 $
|
| 91 |
-
- $ P_3 = \frac{1}{4} P_4 + \frac{3}{4} P_2 $
|
| 92 |
-
- $ P_2 = \frac{1}{4} P_3 + \frac{3}{4} P_1 $
|
| 93 |
-
- $ P_1 = \frac{1}{4} P_2 + \frac{3}{4} P_0 = \frac{1}{4} P_2 + 0 $
|
| 94 |
-
- $ P_0 = 0 $
|
| 95 |
-
|
| 96 |
-
We can solve this system of equations step-by-step.
|
| 97 |
-
|
| 98 |
-
---
|
| 99 |
-
|
| 100 |
-
### Step 6: Final computation
|
| 101 |
-
|
| 102 |
-
After solving the system, we find that:
|
| 103 |
-
|
| 104 |
-
$$
|
| 105 |
-
P_0 = \boxed{0.00123456}
|
| 106 |
-
$$
|
| 107 |
-
|
| 108 |
-
---
|
| 109 |
-
|
| 110 |
-
### ✅ Final Answer:
|
| 111 |
-
|
| 112 |
-
$$
|
| 113 |
-
\boxed{A}
|
| 114 |
-
$$
|
| 115 |
```
|
|
|
|
| 2 |
|
| 3 |
- **Model repo:** [`cs-552-2026-vibe-trainers/general_knowledge_model`](https://huggingface.co/cs-552-2026-vibe-trainers/general_knowledge_model)
|
| 4 |
- **Owner(s):** group **vibe-trainers**
|
| 5 |
+
- **Generated at:** 2026-05-28T08:03:33+00:00 (UTC)
|
| 6 |
- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)
|
| 7 |
|
| 8 |
_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._
|
|
|
|
| 12 |
| Benchmark | Accuracy | Status |
|
| 13 |
|---|---:|---|
|
| 14 |
| Math | — | not run |
|
| 15 |
+
| Knowledge | 0.3300 | ok |
|
| 16 |
| Multilingual | — | not run |
|
| 17 |
| Safety | — | not run |
|
| 18 |
|
|
|
|
| 35 |
|
| 36 |
**Incorrect** (1 shown)
|
| 37 |
|
| 38 |
+
- **reference**: `F`
|
| 39 |
- **overall** (0/1 completions correct)
|
| 40 |
+
- **extracted** (✗): `C`
|
| 41 |
- **completion**:
|
| 42 |
|
| 43 |
```text
|
| 44 |
+
\boxed{C}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 45 |
```
|