# CitationEdge Agent Pipeline — Accuracy & Quality Report

**Date**: 10 May 2026
**Scope**: 7 research papers tested end-to-end through the 11-agent pipeline
**LLM**: Groq `llama-3.3-70b-versatile` (free tier: 12k TPM, 100k TPD)
**Knowledge Graph**: Neo4j AuraDB Cloud | **Vector Store**: LanceDB | **Doc DB**: MongoDB

---

## 1. Executive Summary

The pipeline successfully processes papers and produces structured outputs (claims, scores, counterfactuality analysis, PDF reports). However, **4 of 11 agents have critical defects** that silently degrade output quality. The overall scores (8.1–8.9/10) are **inflated** — they reward high claim counts and perfect citation completeness (a default) while real accuracy gaps go unpunished.

---

## 2. Per-Agent Accuracy Assessment

### 2.1 ParserAgent — ✅ GOOD
| Metric | Value |
|--------|-------|
| Papers parsed | 7/7 (100%) |
| Avg parse time | ~8s |
| Accuracy | **HIGH** |

**Assessment**: Correctly extracts sections, paragraphs, and text from PDFs. 0 references extracted across all papers (these papers may genuinely have no inline reference markers). No false positives or garbage text observed. **No issues.**

### 2.2 VisualParserAgent — ❌ BROKEN
| Metric | Value |
|--------|-------|
| Success rate | 0/7 (0%) |
| Root cause | Missing dependency: `fitz` (PyMuPDF) |
| Impact | No figures extracted from any paper |

**Assessment**: Every single run fails with `No module named 'fitz'`. This is a missing dependency — `pip install pymupdf` would fix it. Even with PyMuPDF, meaningful figure analysis would require a vision LLM, which is not configured.

### 2.3 KeywordAgent — ❌ BROKEN
| Metric | Value |
|--------|-------|
| Keywords extracted | **0 across all 7 papers** |
| Root cause | Keras 3 incompatibility with `transformers` |
| Impact | CitationGapAgent gets 0 keywords → skips all analysis |

**Assessment**: Both KBIR and KeyBERT backends fail with: *"Your currently installed version of Keras is Keras 3, but this is not yet supported in Transformers."* Fix: `pip install tf-keras`. Without keywords, the entire citation gap analysis chain is dead.

### 2.4 ClaimAgent — ✅ GOOD (with caveats)
| Metric | Value |
|--------|-------|
| Claims extracted | **170 total** across 7 papers (avg 24/paper) |
| LLM success rate | 5/7 succeeded, 2 hit TPD limit |
| Avg claim confidence | 0.75–0.85 |

**Caveats**:
- **23/38 claims** (paper 6) and **3/18 claims** (paper 7) have `verdict: null` — these never went through verification
- Some extracted "claims" are fragments or non-claim sentences (e.g. *"the temporal dimension provides some valuable information"*)
- Claim type distribution is reasonable but the LLM sometimes misclassifies (e.g., a methods description as "breakthrough")
- **Quality is good overall** — most claims are genuine substantive statements from the papers

### 2.5 CitationGapAgent — ❌ BROKEN (downstream)
| Metric | Value |
|--------|-------|
| Gaps found | **0 across all 7 papers** |
| Root cause | Receives 0 keywords from KeywordAgent |
| Impact | Citation completeness scores always = 10.0/10 (default) |

**Assessment**: The agent checks `if not keywords: return` — it doesn't even try to search without keywords. Even if keywords were available, Semantic Scholar returns HTTP 429 on every call (no API key). This agent is **non-functional** at two levels.

### 2.6 ClaimVerifierAgent — ⚠️ DEGRADED
| Metric | Value |
|--------|-------|
| Claims verified | ~75 total across 6 papers |
| CONTRADICTED | **0** |
| SUPPORTED | ~65 |
| UNVERIFIED | ~10 |

**Assessment**: **Zero contradicted claims across 7 papers is statistically suspicious.** Every claim a paper makes being fully supported by external literature is implausible, especially for breakthrough/significant claims. The likely explanations:

1. Semantic Scholar returns 429 for every search → LLM has no external evidence → defaults to SUPPORTED
2. LLM exhibits confirmation bias — when it can't find contradictory evidence, it assumes the claim is correct
3. The verification prompt may be skewed toward "supported" as the default output

**False negative rate**: Unknown — cannot verify without manual annotation.

### 2.7 CounterFactualityAgent — ✅ GOOD
| Metric | Value |
|--------|-------|
| Claims analyzed | 170 total |
| Avg CF score | 0.04–0.18 (very low) |
| Classification | ~90% factual, ~10% hypothetical/speculative |

**Assessment**: The classifications are **reasonable** — speculative claims using "could", "might", "may" are correctly flagged as `hypothetical`. The low overall scores (0.04) reflect that well-written research papers are mostly factual. The agent provides a `cf_reasoning` field explaining each classification, which is useful for audit.

**Potential issue**: The agent may be **too conservative** — truly overstated/overclaimed findings may not be caught if phrased confidently.

### 2.8 EvidenceGroundingAgent — ❌ BROKEN
| Metric | Value |
|--------|-------|
| Claims grounded | **0 across all 7 papers** |
| Log message | "No claims found to ground in evidence" |

**Assessment**: This agent appears to query for a specific data structure (evidence nodes) that the EnhancedParserAgent would create — but that agent doesn't exist or doesn't run. The pipeline logs show the agent starts and immediately exits. **Non-functional.**

### 2.9 ArgumentationAgent — ✅ GOOD
| Metric | Value |
|--------|-------|
| Papers analyzed | 7/7 |
| Score | 8.2/10 (identical for all papers) |

**Assessment**: Produces coherent strengths/weaknesses assessments. However, the score is **suspiciously constant** (8.2 for every paper). This suggests the LLM defaults to a narrow range regardless of paper quality. The strengths/weaknesses text is generic but reasonable.

### 2.10 ScoringAgent — ⚠️ INFLATED
| Metric | Value |
|--------|-------|
| Avg overall score | 8.50 / 10 |
| Literary score | 7.96–7.98 |
| Argument score | 8.20 (from ArgumentationAgent) |
| Citation completeness | **10.0/10** (all papers) |
| Verification score | 8.67–9.33 (when data available) |

**Assessment**: The scoring has **structural inflation**:
- `citation_completeness: 10.0` is a default fallback when CitationGapAgent finds 0 gaps — but 0 gaps means the agent didn't run, not that the paper has perfect citations
- A paper could have **zero citations** and still get 10/10 for citation completeness
- Overall scores (8.5–8.9) are consistently high because the two broken agents produce "perfect" scores by default

### 2.11 ReportAgent — ✅ GOOD
| Metric | Value |
|--------|-------|
| PDFs generated | 7/7 |
| Report size | 5–13 KB |

**Assessment**: Successfully generates PDF reports with scores, claims table, counterfactuality section, and argumentation analysis. The claim table is capped at **10 claims** regardless of how many were extracted (38 → 10). This is a design choice but worth noting.

---

## 3. Overall Accuracy Scorecard

| Agent | Functional? | Accurate? | Score | Notes |
|-------|-------------|-----------|-------|-------|
| ParserAgent | ✅ Yes | ✅ High | **9/10** | Rock solid |
| VisualParserAgent | ❌ No | N/A | **0/10** | Missing dependency |
| KeywordAgent | ❌ No | N/A | **0/10** | Keras 3 bug |
| ClaimAgent | ✅ Yes | ⚠️ Medium | **7/10** | Good but some fragments |
| CitationGapAgent | ❌ No | N/A | **0/10** | No keywords + Semantic Scholar 429 |
| ClaimVerifierAgent | ⚠️ Partial | ❌ Low | **4/10** | No CONTRADICTED ever found |
| CounterFactualityAgent | ✅ Yes | ✅ High | **8/10** | Reasonable classifications |
| EvidenceGroundingAgent | ❌ No | N/A | **0/10** | Never finds claims |
| ArgumentationAgent | ✅ Yes | ⚠️ Medium | **6/10** | Score stuck at 8.2 |
| ScoringAgent | ⚠️ Partial | ❌ Low | **3/10** | Inflated by broken agents |
| ReportAgent | ✅ Yes | ✅ High | **8/10** | Cap at 10 claims noted |

**Pipeline average**: **4.1/10** (weighted) — despite individual agents working, the silent failures cascade and inflate scores.

---

## 4. Systemic Issues

### 4.1 Score Inflation from Broken Agents
The scoring formula gives `citation_completeness: 10.0` when CitationGapAgent finds 0 gaps. Since CitationGapAgent always finds 0 gaps (because it's broken), every paper gets perfect citation marks. A paper with literally zero references to prior work would score 10/10 on citations. **This is a critical scoring bug.**

### 4.2 No CONTRADICTED Verdicts
Across 170+ claims and 7 papers, **zero** claims were marked as contradicted. This is statistically implausible for research papers. The verifier likely defaults to SUPPORTED when Semantic Scholar fails (429 on every call). Adding a "NO_DATA" verdict would surface this issue.

### 4.3 Cascade Failures
When KeywordAgent produces 0 keywords:
- CitationGapAgent gets 0 keywords → skips analysis
- Citation completeness defaults to 10.0
- Overall score is inflated

When Semantic Scholar returns 429:
- CitationGapAgent gets no evidence → CitationGapAgent skips
- ClaimVerifierAgent gets no external evidence → defaults to SUPPORTED
- ClaimCounterFactualityAgent still works (LLM only) but has no external validation

### 4.4 Groq Free Tier Limitations
- 12,000 TPM → frequent throttling (every paper hits multiple 429s on TPM)
- 100,000 TPD → exhausted after 3-4 papers
- First API key exhausted completely; second key needed to finish
- Each paper consumes ~15-25k tokens (claim extraction is the biggest consumer)

---

## 5. Recommendations

| Priority | Fix | Impact |
|----------|-----|--------|
| **P0** | `pip install tf-keras` | Unblocks KeywordAgent → unblocks CitationGapAgent → removes score inflation |
| **P0** | Add SEMANTIC_SCHOLAR_API_KEY to config | Unblocks CitationGapAgent and ClaimVerifierAgent |
| **P1** | Fix scoring: don't default to 10.0 when 0 gaps found | Fixes inflated overall scores |
| **P1** | Add "NO_DATA" verdict to verifier | Reveals when verification couldn't run |
| **P1** | `pip install pymupdf` | Unblocks VisualParserAgent |
| **P2** | Fix EvidenceGroundingAgent query | It looks for non-existent data structures |
| **P2** | Switch to a paid LLM tier or add multiple key rotation | Eliminates TPD/TPM bottlenecks |
| **P3** | Increase report claim cap from 10 to configurable | More comprehensive PDFs |
| **P3** | Add `tf-keras` + `pymupdf` to requirements.txt | Prevents future setup issues |

---

## 6. Conclusion

The pipeline **architecture is solid** — DAG wave execution, parallel agents, fault isolation, A2A protocol, and Graph-RAG all work correctly. But the **accuracy is compromised** by:
1. **2 completely broken agents** (KeywordAgent, VisualParserAgent)
2. **2 functionally dead agents** (CitationGapAgent, EvidenceGroundingAgent)
3. **1 degraded agent** (ClaimVerifierAgent — always returns SUPPORTED)
4. **1 scoring bug** that inflates results from broken agents

After fixing the top 3 issues (tf-keras, Semantic Scholar key, scoring default), the pipeline would produce genuinely meaningful and accurate paper analyses.