# Automated MNLP evaluation report

- **Model repo:** [`cs-552-2026-thinking-tokens/general_knowledge_model`](https://huggingface.co/cs-552-2026-thinking-tokens/general_knowledge_model)
- **Owner(s):** group **thinking-tokens**
- **Generated at:** 2026-06-08T04:40:54+00:00 (UTC)
- **Pipeline:** [mnlp-project-ci](https://github.com/eric11eca/mnlp-project-ci)

_This PR is opened automatically by the course CI. It is **non-blocking** — you do not need to merge it. The next nightly run will refresh this file._

## Evaluated checkpoint

- **Commit:** [`3dade21`](https://huggingface.co/cs-552-2026-thinking-tokens/general_knowledge_model/commit/3dade21d4454015942e40dab74360bb2fa82960e)
- **Message:** Update generation_config.json
- **Committed:** 2026-06-07T21:40:33+00:00

## Summary

| Benchmark | Accuracy | Status |
|---|---:|---|
| Math | — | not run |
| Knowledge | 0.3000 | ok |
| Multilingual | — | not run |
| Safety | — | not run |

## Sample completions

_Prompts are intentionally omitted to avoid revealing benchmark contents. For multi-completion problems, only one completion is shown per sample._

### Knowledge

**Correct** (1 shown)

- **reference**: `D`
- **overall** (1/1 completions correct)
- **extracted** (✓): `D`
- **completion**:

    ```text
    <think>
    We need to determine how much insurance will pay for Frank Hartman's car after a $2,350 damage. ThomasSiden has property-damage insurance on his car with a $5,000 limit. This means that the insurance company will cover up to the $5,000 limit, but not more than that. Since the damage is $2,350, which is less than $5,000, the insurance company will pay the full amount of the damage, not less. Therefore, the correct answer is the amount of the damage itself. The options include $2,350 as an option, so that is the correct choice.
    
    \boxed{D}
    ```

**Incorrect** (1 shown)

- **reference**: `C`
- **overall** (0/1 completions correct)
- **extracted** (✗): `H`
- **completion**:

    ```text
    <think>
    The patient's symptoms of urinary frequency and incontinence that begin a few days before flights and resolve the day after are classic signs of psychosomatic urinary incontinence. These symptoms are often triggered by stress, anxiety, or other psychological factors, particularly when the patient is about to travel. The only medication she takes is an oral contraceptive, which is not indicated for this condition. The other options are either unnecessary (diet change, behavioral therapy, physical therapy, psychoanalytic psychotherapy) or not indicated (neurological examination, sexual history review, stopping contraceptives). Therefore, the most appropriate next step is to advise her to stop taking oral contraceptives, as they are likely the cause of the symptoms.
    </think>
    
    \boxed{H}
    ```