# Project Report Skeleton for MVM²

This document serves as a structured template for the final project report. It combines Markdown structure with standard LaTeX placeholders where applicable for formal formatting.

---

# 1. Introduction

## 1.1 Problem Statement
The verification of mathematical reasoning generated by Large Language Models (LLMs) faces two distinct challenges:
1.  **Hallucination:** LLMs often produce "step-by-step" reasoning that appears plausible but contains logical gaps or contradictions.
2.  **Multimodal Noise:** When the input source is an image (handwritten or printed), Optical Character Recognition (OCR) errors introduce uncertainty (e.g., misinterpreting symbols like $\int$ vs $S$). Existing pipelines typically treat transcribed text as ground truth, leading to catastrophic error propagation.

**Objective:** To develop *MVM²*, a multimodal verification system that integrates OCR confidence scores, symbolic execution, and multi-agent consensus to robustly verify mathematical solutions.

## 1.2 Motivation
- Requirement for trusted AI in education (automated grading, tutoring).
- Limitations of "black box" verifiers (GPT-4) which lack explainability.
- Need for formally verifiable metrics (SymPy) combined with semantic understanding (LLMs).

---

# 2. Literature Review

| Paper / Tool | Key Contribution | Limitation Addressed by MVM² |
|---|---|---|
| **Math-Verify (HuggingFace)** | Rule-based answer extraction & equivalency. | Lacks semantic logic checking; purely symbolic. |
| **MathVerse (ECCV 2024)** | Multimodal benchmark for visual math. | Focuses on evaluation, not the *verification algorithm* itself. |
| **Self-Consistency (Wang et al.)** | Majority voting for LLMs. | Computationally expensive; doesn't handle visual uncertainty. |

---

# 3. Methodology & System Architecture

## 3.1 Overview
The system adopts a modular microservice architecture consisting of seven key components:
1.  **Input Receiver:** Validates multimodal inputs.
2.  **Preprocessing:** Image binarization and noise reduction (OpenCV).
3.  **OCR Service:** Hybrid extraction (Tesseract + Handwritten CNN).
4.  **Representation:** Canonicalization to Intermediate Representation (IR).
5.  **Verification Engine:** SymPy (Symbolic) + Multi-Agent LLMs (Logical).
6.  **Classifier:** Weighted Consensus Scoring.
7.  **Reporting:** Explainable feedback generation.

## 3.2 Formal Methods

### 3.2.1 OCR-Aware Confidence Propagation
We propose a novel method to discount verification confidence based on visual uncertainty. Let $C_{ocr}$ be the OCR confidence score. The final confidence $C_{final}$ is calibrated as:

$$
C_{final} = S_{weighted} \times (\lambda + (1-\lambda)C_{ocr})
$$

Where $\lambda=0.9$ ensures a high floor for legibility but penalizes ambiguity.

### 3.2.2 Hybrid Scoring Function
The validity score $S_{weighted}$ is computed from three independent signals:

$$
S_{weighted} = \alpha S_{sym} + \beta S_{log} + \gamma S_{clf}
$$

**Parameters:**
- $\alpha = 0.40$ (Symbolic Accuracy)
- $\beta = 0.35$ (Logical Consistency)
- $\gamma = 0.25$ (Classifier Consensus)

## 3.3 Multi-Agent Consensus
We utilize three agents with distinct prompts:
1.  **Solver:** Independently solves the problem.
2.  **Critic:** Reviews the provided steps for logical fallacies.
3.  **Verifier:** Compares the Solver and User steps.

**Hallucination Rate ($H$)** is defined as the fraction of steps where agents fail to reach consensus (similarity threshold $< 0.7$).

---

# 4. Experiments

## 4.1 Dataset
- **Sources:** Hand-curated samples, modified GSM8K subset.
- **Types:** Text-only, Clean Images, Noisy Images (Gaussian noise added).

## 4.2 Ablation Studies
We evaluated four configurations to quantify the contribution of each component:

| Mode | Description |
|---|---|
| **M1: Baseline** | Single LLM (Gemini-Pro) without SymPy or OCR weighting. |
| **M2: Hybrid** | Single LLM + SymPy verification. |
| **M3: Consensus** | Multi-Agent LLM + SymPy (No OCR calibration). |
| **M4: Full MVM²** | Full pipeline with OCR-aware confidence. |

---

# 5. Results & Analysis

## 5.1 Quantitative Results via `run_evaluation.py`

*(Insert table from `evaluation_results.csv` here)*

| Mode | Accuracy | Hallucination Rate | Avg Latency |
|---|---|---|---|
| M1 (Baseline) | Low | High | Low |
| M4 (Full) | **High** | **Low** | Moderate |

**Key Finding:** The multi-agent approach reduced the hallucination rate by **X%** compared to the baseline.

## 5.2 Case Studies
- **Case A (Ambiguous Handwriting):** M4 correctly flagged "Low Confidence" due to OCR uncertainty, whereas M1 confidently marked it incorrect based on bad transcription.
- **Case B (Algebraic Error):** SymPy component ($S_{sym}$) detected a subtle sign error that the LLM ($S_{log}$) missed.

---

# 6. Limitations & Future Work

## 6.1 Limitations
- **Latency:** Multi-agent calls increase response time (~3-4s).
- **OCR Dependency:** Extremely poor handwriting still fails early in the pipeline.

## 6.2 Future Work
- **Fine-tuning:** Train a dedicated small model (SLM) for the Critic role to reduce latency.
- **Visual-LLM Integration:** Feed images directly to Gemini 1.5 Pro to bypass OCR for complex geometry problems.

---

# 7. Conclusion
MVM² successfully demonstrates that integrating formal symbolic methods with probabilistic LLM reasoning—calibrated by visual uncertainty—significantly improves the reliability of mathematical verification systems.