# Implementation Plan — Corpus Similarity Report ("Originality Report")

**Feature:** Turnitin-style similarity/originality reporting for uploaded papers.
**Status:** Phases 0–4 implemented; merged with remote `report-ui-updated` (merge commit
`506ad88`). Phases 5–8 outstanding.
**Date:** 2026-08-02 · revised after the merge

> ### Post-merge amendments (read before implementing Phases 5–8)
>
> The remote branch landed 15 commits while Phases 0–4 were being built. Four of them
> change assumptions in this document. The sections below have been corrected in place;
> this box is the summary.
>
> 1. **Every API route now lives under `/api`** (`d58fb43`). Each router carries its own
>    prefix and the SPA is mounted at `/` with a client-side fallback, so any route
>    *not* under `/api` is swallowed by the frontend. The similarity router was moved to
>    `prefix="/api/similarity"`; §9 reflects the new paths.
> 2. **Wave 3 now runs eight agents**, not five — `citation_relevance` and `novelty`
>    joined (`b54aebf`). The latency argument in §8.1 still holds (the wave is bounded by
>    its slowest member and these are also network-bound), but the wave is busier than
>    §8.1 originally described.
> 3. **`PdfHighlightMapper` is called from `agents/report_agent.py:327`, not the
>    orchestrator.** §8.4 said the orchestrator should make the single combined call.
>    That is now wrong and, happily, unnecessary: `report` is wave 5 and `similarity` is
>    wave 3, so ReportAgent already has both span sets in hand. The "call it once"
>    requirement is satisfied structurally — Phase 5 only has to pass similarity spans
>    into the existing call site.
> 4. **`report_agent.py` is now 979 lines** (was 775) after the line-level AI-badge work
>    (`7a1c4bd`, `f5f5aee`). It is well past the 800-line cap, so the Phase 7 extraction
>    is no longer merely advisable — nothing new can go in that file until it is split.

---

## Table of Contents

1. [What we are building](#1-what-we-are-building)
2. [Decisions already locked](#2-decisions-already-locked)
3. [Why this is hard (and how we get around it)](#3-why-this-is-hard-and-how-we-get-around-it)
4. [How the number is computed](#4-how-the-number-is-computed)
5. [Architecture — where every piece lives](#5-architecture--where-every-piece-lives)
6. [Data model](#6-data-model)
7. [Corpus providers](#7-corpus-providers)
8. [Integration into the existing pipeline](#8-integration-into-the-existing-pipeline)
9. [API surface](#9-api-surface)
10. [Frontend](#10-frontend)
11. [PDF report](#11-pdf-report)
12. [Configuration](#12-configuration)
13. [Testing strategy](#13-testing-strategy)
14. [Evaluation and calibration](#14-evaluation-and-calibration)
15. [Build order — phases with exit criteria](#15-build-order--phases-with-exit-criteria)
16. [Risks and mitigations](#16-risks-and-mitigations)
17. [What we are deliberately NOT building](#17-what-we-are-deliberately-not-building)
18. [Open questions](#18-open-questions)

---

## 1. What we are building

A report, attached to every analysis job, that answers: **which passages of this paper
already exist somewhere else, and where.**

Output shape, mirroring the reference screenshot:

```
Research_Proposal.pdf
CORPUS SIMILARITY REPORT

  11%              10%              8%               8%
  CORPUS           OPEN ACCESS      PUBLICATIONS     CITATIONEDGE
  SIMILARITY       SOURCES                           CORPUS

PRIMARY SOURCES
  [1] arxiv.org · Open Access                                    5%
  [2] www.isca-archive.org · Open Access                         2%
  [3] "Text, Speech and Dialogue", Springer 2022 · Publication   1%
  [4] CitationEdge Corpus · submitted 2026-03-14                 1%
  ...

Checked against 412 candidate sources across 4 providers · 6,180 words compared
· quoted text and bibliography excluded · not a Turnitin-equivalent index
```

Plus a **separate, non-headline** "Paraphrase Risk" signal, and **on-page highlighting**
of matched spans over the rasterized PDF (the viewer already exists — see §8.4).

### Why this feature belongs in CitationEdge

Per `PRODUCT.md`, the product promise is "depth over search" and "grounded output". A
similarity report is the natural sibling of the existing `citation_gap` agent: that one
says *"you should have cited something here"*, this one says *"this passage already
exists there"*. Both are claims about the paper's relationship to the literature, and
both must trace back to a real, linkable source. The two features share the same
provider layer, the same ranking intuition, and the same UI grammar.

---

## 2. Decisions already locked

| # | Decision | Chosen | Why |
|---|---|---|---|
| 1 | Corpus | Hybrid: internal upload corpus + open academic full text | Free, no vendor lock-in, defensible provenance. Internal corpus gives duplicate-submission and self-plagiarism detection that no external API can. |
| 2 | Match method | Verbatim fingerprinting for the headline %, semantic embeddings as a separate signal | A percentage must mean "this fraction of words is copied". Cosine similarity cannot carry that meaning — every paper scores non-zero against its own field. |
| 3 | Framing | "Corpus Similarity" with mandatory coverage disclosure | We cannot index what Turnitin indexes. Shipping the number without the denominator would violate `PRODUCT.md` principle 2 (grounded output) and invites misconduct decisions on incomplete evidence. |
| 4 | Integration | Pipeline agent (wave 3, non-critical) + on-demand recheck endpoint | Fits the existing DAG exactly; recheck lets a report be refreshed as the corpus grows without re-parsing the PDF. |

---

## 3. Why this is hard (and how we get around it)

### The core problem

Turnitin works because it owns a crawled index of ~100 billion web pages plus licensed
publisher content plus ~2 billion student papers. We have none of that, and building it
is not a feature — it is a company.

**We cannot download the corpus. So we invert the problem: we let someone else's
inverted index do the retrieval, and we do the verification locally.**

### Two-stage retrieval

```
Stage 1 — RETRIEVAL (remote, cheap, recall-oriented)
  Pick ~20-30 distinctive exact phrases from the paper.
  Send each as a phrase query to CORE's Solr index:
      GET https://api.core.ac.uk/v3/search/works?q=fullText:"<phrase>"
  CORE searches 40M+ open-access full texts and returns the works
  containing that literal phrase, WITH their full text in the response.
  → Yields a candidate set of maybe 50-400 documents.

Stage 2 — VERIFICATION (local, exact, precision-oriented)
  For each candidate: winnowing fingerprint intersection → seed matches
  → greedy span extension → exact character offsets in both documents.
  → Yields defensible spans with real word counts.
```

This is the whole trick. Stage 1 costs a few HTTP calls and gives us recall against a
corpus we could never host. Stage 2 costs milliseconds of CPU and gives us the precision
that makes a percentage honest. **A phrase that appears in no candidate contributes 0% —
we never estimate, interpolate, or ask an LLM to guess.**

### Why phrase queries and not keyword queries

A keyword query (`fullText:"transformer attention"`) returns tens of thousands of
topically-related papers, none of which necessarily share a single sentence. A phrase
query for a distinctive 8-12 word run returns only documents that literally contain those
words. Recall is lower, but every hit is already close to a confirmed match, so the
verification stage does very little wasted work. This is the same reason
`paper_recommender.py` strips `FILLER_TOKENS` before querying — generic words return
generic results.

### Choosing which phrases to query

We cannot query every phrase in the paper (a 6,000-word paper has ~6,000 overlapping
8-grams). We select the most *distinctive* ones:

1. Drop phrases that are wholly inside excluded regions (bibliography, quotes, headers).
2. Drop phrases whose tokens are all in `STOPWORDS ∪ FILLER_TOKENS`
   (reuse the lists already curated in `services/paper_recommender.py` — do not fork them).
3. Drop phrases matching the **boilerplate stoplist** (§16, risk 3):
   "in this paper we propose", "the rest of this paper is organized as follows", etc.
4. Score remaining phrases by mean inverse token frequency *within the document* —
   rare-in-this-paper words are usually rare globally too.
5. Spread the selection across sections so we don't query 30 phrases from the
   introduction and none from the methods.
6. Take the top `SIM_MAX_QUERIES` (default 24).

Reasoning: a paper that copies will almost always copy a *contiguous run*. A stratified
sample of 24 distinctive phrases across the document has high probability of landing
inside any copied run longer than ~150 words. Copied runs shorter than that are caught
only if a sampled phrase happens to land in them — an acknowledged recall limit, and one
more reason the coverage disclosure is mandatory.

---

## 4. How the number is computed

This section is the spec. Everything else is plumbing.

### 4.1 The denominator

```
total_words = words in the document AFTER exclusions
```

Excluded by default (each independently toggleable, mirroring Turnitin's options):

| Region | Excluded? | Why |
|---|---|---|
| Bibliography / references section | Yes | Reference strings are *supposed* to be identical to the original. Including them inflates every paper by 3-8%. `ParserAgent` already isolates this section. |
| Quoted text (`"..."`, block quotes) | Yes | Properly attributed quotation is not plagiarism. |
| Title / author / affiliation block | Yes | Institutional addresses match thousands of papers. |
| Equations, tables, figure captions | Yes | Notation collides across the whole field. |
| Everything else | No | This is the paper. |

### 4.2 The coverage bitmap

The single data structure the whole score rests on:

```python
covered: list[int | None]   # one slot per word in the post-exclusion document
                            # value = index of the source that owns this word, or None
```

Every confirmed match writes its word range into the bitmap. **A word is counted once no
matter how many sources contain it** — this is what stops the percentages from exceeding
100% when a boilerplate sentence appears in forty papers.

```
overall_similarity_pct = round(100 * count(covered[i] is not None) / total_words)
```

### 4.3 Attribution — who gets credit for a word

When two sources both contain a span, the word is attributed to **the source with the
longest confirmed match overlapping that word**; ties break on higher provider trust
(internal corpus > CORE full text > publisher abstract). This is why per-source
percentages in the primary-sources list *approximately* sum to the headline index rather
than wildly exceeding it.

```
per_source_pct[s] = round(100 * count(covered[i] == s) / total_words)
```

### 4.4 Bucket percentages

The three bucket numbers (Open Access / Publications / CitationEdge Corpus) are **each
computed independently**, as "the similarity index you would get if this were the only
bucket". They deliberately do **not** sum to the headline number — in the reference
screenshot, 10 + 8 + 8 = 26 against an index of 11, for exactly this reason. Turnitin
behaves the same way; replicating it avoids surprising anyone who has read a real report.

```python
bucket_pct[b] = round(100 * count(word covered by ANY source in bucket b) / total_words)
```

Implementation: run the bitmap fill three more times, once per bucket, over the same
confirmed match list. Cheap (it is a pass over a list of integers), and much clearer than
trying to derive bucket numbers from the attributed bitmap.

### 4.5 Minimum match length

A match must be at least `SIM_MIN_MATCH_WORDS` (default **8**) consecutive words to enter
the bitmap. Below that, shared runs are linguistic coincidence, not reuse. This threshold
is the single biggest lever on the false-positive rate and must be exposed in config and
recorded in the report output so a result is reproducible.

### 4.6 The paraphrase signal (separate, never in the %)

Parallel pass using the existing `VectorStoreService` (SciBERT):

- Embed each body paragraph of the uploaded paper.
- Embed each candidate document's paragraphs (candidates are already in hand from stage 1
  — no extra network cost).
- Flag pairs with cosine ≥ `SIM_PARAPHRASE_THRESHOLD` (default 0.92) **that are not
  already verbatim-matched**.

Reported as a count and a list — "4 passages closely track a source without verbatim
overlap" — with the passage pairs shown side by side. It is a *reading prompt for a
human*, not a score, and it never touches the headline number. Reasoning: SciBERT cosine
between two paragraphs from the same subfield routinely hits 0.85 with zero copying; any
percentage built on that would be noise wearing a lab coat.

---

## 5. Architecture — where every piece lives

The new code goes in a **self-contained `similarity/` package**, deliberately mirroring
the existing `text_detection/` package (config / schema / pipeline / pluggable backends,
everything env-toggled, degrades instead of blocking). That package is the house style
for "a bounded analytical capability the pipeline calls into", and copying its shape
means anyone who has read one can read the other.

```
similarity/
├── __init__.py
├── config.py               # pydantic-settings, all SIM_* env vars          (~80 lines)
├── schema.py               # Pydantic models — the contract                 (~150)
├── normalize.py            # canonicalization + offset map                  (~120)
├── exclusions.py           # bibliography / quotes / boilerplate regions    (~180)
├── fingerprint.py          # k-gram hashing + winnowing                     (~130)
├── selector.py             # pick distinctive query phrases                 (~140)
├── matcher.py              # fingerprint seeds → verified spans             (~200)
├── aggregate.py            # coverage bitmap → percentages + buckets        (~180)
├── semantic.py             # SciBERT paraphrase pass                        (~120)
├── pipeline.py             # orchestrates all of the above                  (~220)
├── index_writer.py         # write this doc into the internal corpus        (~110)
└── corpus/
    ├── __init__.py
    ├── base.py             # CorpusProvider protocol + Candidate dataclass  (~90)
    ├── core_api.py         # CORE v3 — the full-text workhorse              (~170)
    ├── openalex.py         # abstract-level matching                        (~90)
    ├── arxiv.py            # abstract + OA PDF fetch                        (~110)
    ├── crossref.py         # publisher metadata / abstracts                 (~90)
    └── internal.py         # prior CitationEdge uploads                     (~190)
```

Every file lands well inside the 800-line cap and most inside the 200-400 "typical" band
from the coding-style rules.

### Files modified (not created)

| File | Current | Change | Note |
|---|---|---|---|
| `agents/similarity_agent.py` | — | **new**, ~90 lines | Thin adapter: pull text from Neo4j → call pipeline → write Neo4j + return dict. All logic lives in `similarity/`. |
| `agents/__init__.py` | | export `SimilarityAgent` | |
| `orchestrators/custom_orchestrator.py` | 359 | +~40 | Register agent; fetch similarity rows; add to `final_result` / `save_result`. |
| `configs/pipeline_config.yaml` | 39 | +4 | Declare `similarity` at wave 3, `depends_on: [parser, keyword]`. |
| `services/pdf_highlight_mapper.py` | 307 | ~+30 | Generalize hardcoded `"ai"` / `"human"` typing — see §8.4. |
| `backend/routers/reports.py` | 478 | +~45 | `GET /reports/{id}/similarity`. |
| `backend/routers/similarity.py` | — | **new**, ~110 | `POST /{id}/recheck`, `GET /status`. |
| `backend/schemas/similarity.py` | — | **new**, ~40 | Request + status models; reuse `similarity.schema` for the response, exactly as `text_detection` does. |
| `backend/main.py` | 288 | +2 | Register router (done). |
| `agents/report_agent.py` | **979** | **extract, then +2** | Grew from 775 to 979 in the merge; far past the cap and cannot absorb a new section. Extract the similarity section into `agents/report_sections/similarity_section.py`. |
| `frontend-app/src/pages/ReportPage.jsx` | **869** | **−~300, +~8** | **Already over the 800 cap.** Extract `buildReport()` and `DEMO` into `pages/reportAdapter.js` / `pages/reportDemo.js` first, then add the similarity block as 8 lines calling new components. |
| `frontend-app/src/components/SimilarityIndex.jsx` + `.css` | — | **new**, ~160 | Header block: big %, three buckets. |
| `frontend-app/src/components/SimilaritySources.jsx` | — | **new**, ~140 | Numbered, colour-coded primary sources. |
| `frontend-app/src/components/CoverageNote.jsx` | — | **new**, ~60 | The disclosure. |
| `frontend-app/src/api.js` | 103 | +~15 | `getSimilarity`, `recheckSimilarity`. |
| `requirements.txt` | 56 | +0 | **No new dependencies.** `httpx`, `numpy`, `pymupdf`, `transformers` are all present. |

**Zero new runtime dependencies** is a deliberate design constraint: the fingerprinting is
~130 lines of standard-library Python, and adding a plagiarism SDK would undo the
"no vendor lock-in" decision from §2.

---

## 6. Data model

### 6.1 Pydantic schema (`similarity/schema.py`)

```python
class Bucket(str, Enum):
    OPEN_ACCESS   = "open_access"     # CORE, arXiv — the "Internet Sources" analogue
    PUBLICATION   = "publication"     # Crossref/DOI-bearing, publisher-hosted
    INTERNAL      = "internal"        # prior CitationEdge uploads

class MatchSpan(BaseModel):
    doc_start_word: int               # inclusive, index into post-exclusion word list
    doc_end_word: int                 # exclusive
    doc_char_start: int               # offset into ORIGINAL text — drives highlighting
    doc_char_end: int
    word_count: int
    excerpt: str                      # <= 300 chars, for display
    source_excerpt: str               # the matching text in the source
    source_index: int                 # 1-based, matches the numbered UI chips

class SourceMatch(BaseModel):
    source_index: int
    bucket: Bucket
    title: str
    url: str                          # ALWAYS resolvable — a source you cannot open
                                      # is not evidence. Candidates without a URL are
                                      # dropped, same rule as _score_candidates().
    doi: str | None
    authors: list[str] = []
    year: int | None
    provider: str                     # "CORE" | "arXiv" | "Crossref" | "OpenAlex" | "CitationEdge Corpus"
    display_label: str                # "arxiv.org" / '"Text, Speech and Dialogue", Springer, 2022'
    matched_words: int                # attributed words only (§4.3)
    percent: int                      # matched_words / total_words * 100, rounded
    spans: list[MatchSpan]

class ParaphraseFlag(BaseModel):
    doc_excerpt: str
    source_excerpt: str
    source_index: int
    cosine: float

class Coverage(BaseModel):
    """Mandatory disclosure — the denominator behind the number."""
    total_words_compared: int
    words_excluded: int
    exclusions_applied: list[str]       # ["bibliography", "quotes", "title_block", ...]
    providers_queried: list[str]
    providers_failed: list[str]         # a 429 from CORE must be visible, never silent
    phrases_queried: int
    candidates_retrieved: int
    candidates_verified: int
    internal_corpus_size: int
    min_match_words: int
    budget_exhausted: bool              # True if we stopped early on the time budget
    checked_at: datetime

class SimilarityReport(BaseModel):
    overall_percent: int
    bucket_percents: dict[Bucket, int]
    sources: list[SourceMatch]          # sorted by percent desc; the "primary sources" list
    paraphrase_flags: list[ParaphraseFlag]
    coverage: Coverage
    status: Literal["complete", "partial", "unavailable"]
    notes: list[str]                    # human-readable caveats surfaced in the UI
    processing_time_ms: float
```

`status` semantics, following the `text_detection` precedent of degrading rather than
blocking:

- `complete` — all configured providers answered.
- `partial` — at least one provider failed or the time budget was exhausted. The number
  is a **lower bound**; the UI must say so.
- `unavailable` — no provider answered, or the document was too short
  (`< SIM_MIN_DOCUMENT_WORDS`, default 300). **No number is shown.** We never render a
  0% that means "we didn't look".

### 6.2 Persistence

| Store | Collection / label | Contents | Why there |
|---|---|---|---|
| MongoDB | `results.similarity` | Full `SimilarityReport` dict | Same place every other agent output lives; `GET /reports/{id}/json` picks it up for free via the existing `$set` merge in `save_result`. |
| MongoDB | `corpus_fingerprints` | `{doc_id, job_id, title, owner_email, word_count, fingerprints: [int], created_at}` | The internal corpus inverted index. Multikey index on `fingerprints` makes candidate lookup a single indexed query. |
| MongoDB | `corpus_texts` | `{doc_id, normalized_text, offset_map}` | Needed to verify and excerpt an internal match. Separate collection so the fingerprint index stays small and hot. |
| Neo4j | `(:Document)-[:HAS_SIMILARITY]->(:SimilarityReport)` and `(:SimilarityReport)-[:MATCHES]->(:SimilaritySource)` | Summary properties only | Keeps the graph the primary thinking surface (`PRODUCT.md` principle 4) — you can traverse from a paper to the works it overlaps. Full spans stay in Mongo; the graph is not a document store. |
| LanceDB | table `corpus_paragraphs` | Paragraph embeddings for the paraphrase pass | `LanceDBService` already exists and is already wired into `AgentContext.vector_db`. |

---

## 7. Corpus providers

All providers implement one protocol so the pipeline is provider-agnostic and each can be
switched off independently:

```python
class CorpusProvider(Protocol):
    name: str
    bucket: Bucket
    async def search_phrase(self, phrase: str, limit: int) -> list[Candidate]: ...
    async def fetch_text(self, candidate: Candidate) -> str | None: ...
```

### 7.1 CORE — the workhorse

- Endpoint: `GET https://api.core.ac.uk/v3/search/works?q=fullText:"<phrase>"`
- Coverage: ~300M metadata records, **40M+ full texts** — the largest open-access
  aggregation available.
- Auth: free; works unauthenticated but **register for an API key** — the unregistered
  limit is roughly 5 single requests / 10 seconds, which 24 phrase queries would blow
  through in the first two seconds.
- Crucially, the response can carry `fullText`, so retrieval and text-fetch collapse into
  one round trip.
- Handling: shared `asyncio.Semaphore(2)` plus a stagger, exactly like `_PROVIDER_SEM` in
  `paper_recommender.py`; `@async_retry` on 429/5xx; on persistent failure record it in
  `coverage.providers_failed` and continue.

### 7.2 arXiv

- Reuse `_search_arxiv()` from `services/paper_recommender.py` verbatim for metadata.
- For candidates that survive abstract-level screening, fetch the OA PDF and extract text
  with the existing `utils.pdf.extract_text_from_pdf`. Cap at `SIM_MAX_PDF_FETCHES`
  (default 5) — PDF fetch and parse is the slowest thing in the pipeline.
- Bucket: `OPEN_ACCESS`.

### 7.3 OpenAlex and Crossref

- Reuse `_search_openalex()` and `_search_crossref()` unchanged.
- **Abstracts only** — neither serves full text. An abstract-level match is real but
  small, so these mostly populate the `PUBLICATION` bucket with 1% entries, which is
  exactly what the reference screenshot shows for Springer.
- `source_for()` and the `DOI_PUBLISHER` prefix map already turn a DOI into
  "Elsevier" / "Springer" / "IEEE" — reuse directly for `display_label`.

**DRY note:** these three provider functions are imported from `paper_recommender.py`,
never copied. If that means lifting them into a shared `services/academic_search.py` that
both modules import, do that refactor as part of Phase 2 — it is a targeted improvement
to code we are already working in, not unrelated cleanup.

### 7.4 Internal corpus

The only bucket where we own the index, and the only one that can catch a paper
submitted twice.

**Write path** (`similarity/index_writer.py`, called at the end of a successful job when
`SIM_INDEX_UPLOADS=true`): normalize → fingerprint → insert into `corpus_fingerprints` +
`corpus_texts` → upsert paragraph embeddings into LanceDB.

**Read path:** fingerprint the new document → query
`corpus_fingerprints.find({fingerprints: {$in: doc_fingerprints}})` → the multikey index
returns only documents sharing at least one fingerprint → fetch their normalized text →
verify locally. No embeddings needed for the verbatim path; this is exact and fast.

**Privacy — this is not optional.** Another researcher's uploaded paper is confidential.
The report may show:

- that a match exists,
- the matched **excerpt from the user's own document**,
- a neutral label: `CitationEdge Corpus · submitted 2026-03-14`,
- the percentage.

The report must **never** expose the other document's title, authors, owner, full text, or
a link to it. This mirrors how Turnitin handles student-paper matches, and it is the
behaviour that lets us keep the bucket at all. Enforcement: `SourceMatch.title` and
`.url` are set to the neutral label for `Bucket.INTERNAL`, in `internal.py`, at
construction — not filtered later in the UI where a future refactor could drop the filter.

Self-match guard: a document must never match itself. Exclude by `doc_id`, and on
recheck, by `job_id` too.

---

## 8. Integration into the existing pipeline

### 8.1 The agent

`agents/similarity_agent.py`:

```python
class SimilarityAgent(BaseAgent):
    name = "similarity"
    wave = 3          # after parser (w1) and keyword (w2)
    critical = False  # a similarity failure must never fail an analysis
```

**Why wave 3.** It needs paragraphs from `ParserAgent` (wave 1) and benefits from
`KeywordAgent` output (wave 2) for query construction and domain-token scoring. Wave 3
already runs five agents in parallel — `citation_gap`, `claim_verifier`,
`counter_factuality`, `evidence_grounding`, `argumentation` — and all but the last are
already blocking on external HTTP. `HybridWorkflow` runs a wave via `ParallelWorkflow`,
so the added wall-clock cost is `max(0, similarity_time − current_wave3_time)`, which for
a 45-second budget against agents that already take that long is close to zero.

**Why `critical = False`.** Look at `HybridWorkflow.execute`: a failed critical agent
breaks the wave loop and aborts the pipeline. A CORE outage must not cost the user their
claim verification, citation gaps, and PDF report. Non-critical failure degrades the job
to `partial_failure` and the rest of the analysis completes — the same choice
`citation_gap`, `claim_verifier`, `counter_factuality`, and `evidence_grounding` already
make.

The agent itself stays thin (~90 lines): read paragraphs from Neo4j (the same query the
orchestrator already uses for AI-text detection), call
`SimilarityPipeline.run(text, doc_id, keywords)`, write summary nodes to Neo4j, return
the report dict. All algorithmic work lives in `similarity/` where it can be unit-tested
without a database.

### 8.2 Time budget

A hard `asyncio.wait_for` wrapping the whole retrieval stage at `SIM_BUDGET_SECONDS`
(default 45). On timeout the pipeline returns whatever it has verified so far with
`status="partial"` and `coverage.budget_exhausted=True`. Partial evidence, honestly
labelled, beats a hung job — and this is what makes it safe to put a network-bound step
inside a user-facing pipeline at all.

### 8.3 Orchestrator wiring

In `custom_orchestrator.py`, following the exact pattern already used for
`counterfactuality` and `ai_text_detection`:

1. Add `SimilarityAgent()` to `get_agents()`.
2. After the workflow returns, read the similarity result off the `AgentResult` list
   (cleaner than a Neo4j round trip, and the report is a document not a graph).
3. Add `similarity` to `final_result`, to `upsert_job`, and to `save_result`.

Note that `save_result` merges with `$set`, so adding a key is non-breaking for existing
consumers.

### 8.4 Highlighting — reuse, with one required change

`services/pdf_highlight_mapper.py` already does the hard part: PyMuPDF word extraction,
normalized substring location, multi-line box grouping, 0-1 coordinate normalization, 2x
page rasterization, and graceful handling of unmatched spans and missing text layers.
`ReportPage.jsx` already renders those boxes over page images served by
`GET /jobs/{id}/pages/{n}.png`. **We should not build a second highlighting system.**

Two changes are required:

1. **Generalize the span type.** `_match_span()` hardcodes
   `"type": "ai" if span.get("verdict") == "AI_GENERATED" else "human"`, and `run()`
   filters to `verdict in ("AI_GENERATED", "REAL")`. Add an optional `kind` field on the
   input span that passes through when present, plus an optional `meta` dict (carrying
   `source_index` so the frontend can colour-code by source the way the reference
   screenshot numbers and colours each entry). Default behaviour unchanged — existing
   AI-detection callers and their tests keep working.

2. **Call the mapper once, not twice.** It rasterizes into `pages_dir` and returns a full
   `pages` array. Two independent callers would either race on the PNGs or produce two
   `page_highlights` structures that the frontend cannot merge.

   **Corrected after the merge:** the sole caller is `agents/report_agent.py:327`, not the
   orchestrator as this section originally assumed. That is the easier situation —
   `report` is wave 5 and `similarity` is wave 3, so ReportAgent already has both the AI
   spans and the similarity report available when it builds the call. Phase 5 therefore
   does not introduce a second call site; it extends the existing one with the similarity
   spans, each tagged with its `kind`. The frontend then filters by `kind` to toggle
   "AI detection" and "Similarity" overlay layers independently.

This is a real integration cost and it is the reason to do highlighting in its own phase
(Phase 5) rather than smuggling it into the agent phase.

---

## 9. API surface

Following the conventions in `backend/backend.md`. **All routes sit under `/api`** — the
SPA is mounted at `/` with a catch-all fallback, so a route outside `/api` returns the
frontend's HTML instead of JSON. Each router declares its own prefix; there is no
app-level prefix to inherit.

| Method | Path | Response | Notes |
|---|---|---|---|
| `GET` | `/api/reports/{job_id}/similarity` | `200 SimilarityReport` · `202` not ready · `404` no job | Lives in `reports.py` beside `/json`, `/pdf`, `/graph`. |
| `POST` | `/api/similarity/{job_id}/recheck` | `202 {job_id, status: "rechecking"}` | Re-runs against the current corpus using the stored normalized text — **no PDF re-parse, no re-analysis**. Runs as a `BackgroundTask`, same as `POST /analyze`. |
| `GET` | `/api/similarity/status` | `200 SimilarityStatus` | Which providers are configured, corpus size, thresholds. Directly modelled on `GET /text-detection/status`, and for the same reason: it is how an operator distinguishes "nothing configured" from "configured but failing". |

`recheck` needs a guard: reject with `409` if a recheck for that job is already in flight,
tracked with a simple in-process set keyed by `job_id`. Without it, a user clicking the
button four times launches four concurrent 24-query provider storms and earns a rate-limit
ban for everyone.

Auth: the existing routers are unauthenticated and jobs are filtered by `user_email` as a
query parameter. Match the existing pattern — do not invent a new auth story inside this
feature. (Flagged in §18 as a pre-existing gap worth its own work.)

---

## 10. Frontend

### 10.1 Prerequisite refactor

`ReportPage.jsx` is **869 lines — already over the 800-line cap** in the coding-style
rules (826 before the merge; the AI-badge work added to it). Before adding anything:

- Move `buildReport()` and its helpers → `pages/reportAdapter.js` (~180 lines).
- Move the `DEMO` constant → `pages/reportDemo.js` (~90 lines).

That brings `ReportPage.jsx` to roughly 590 lines and makes room. This is exactly the
"targeted improvement to code you are working in" case — it is required to add the
feature cleanly, not opportunistic refactoring.

### 10.2 New components

**`SimilarityIndex.jsx`** — the header block from the screenshot: one large percentage
plus three bucket figures. Follows `DESIGN.md` / `PRODUCT.md` brand rules: obsidian
background, glass card, Instrument Serif for the big number, DM Mono for the bucket
figures. **Restraint over alarm** — the number is set in the standard white display
treatment, not red. Red is reserved for errors in this brand, and colouring a similarity
score red makes an accusation the data does not support. A single accent colour appears
only on the per-source chips, to tie a source to its highlight colour on the page.

**`SimilaritySources.jsx`** — the numbered list. Each row: colour chip with index, title
as a link (internal-corpus rows are deliberately not links), provider label, percentage.
Clicking a row scrolls the page viewer to that source's first highlight and dims the
others.

**`CoverageNote.jsx`** — always rendered, never behind a disclosure triangle:

> Checked 6,180 words against 412 candidate sources from CORE, arXiv, OpenAlex, Crossref
> and 1,204 papers in the CitationEdge corpus. Quoted text and bibliography excluded.
> Matches under 8 consecutive words are not counted. This is not a Turnitin-equivalent
> index — coverage is limited to open-access literature and papers analysed here.

When `status === "partial"`, prepend: *"One or more sources could not be reached; this
figure is a lower bound."* When `unavailable`, the component replaces the whole block —
**no percentage is rendered at all.**

### 10.3 Wiring

`ReportPage.jsx` gains ~8 lines: read `result.similarity`, render the three components,
pass `kind === 'similarity'` highlights to the existing page viewer. `api.js` gains
`getSimilarity(jobId)` and `recheckSimilarity(jobId)` following the existing helper style.

---

## 11. PDF report

`agents/report_agent.py` is **979 lines** after the merge — already past the cap. Nothing
new goes in inline; the extraction below is now a precondition, not a preference.

- Create `agents/report_sections/similarity_section.py` (~140 lines) exporting
  `build_similarity_section(report: dict) -> list[Flowable]` returning ReportLab
  flowables.
- `report_agent.py` imports it and appends the result — a two-line change. Consider
  extracting the existing AI-badge section the same way while you are in there; at 979
  lines the file is the worst offender in the repo.
- Section contents: the index block, the primary-sources table, and the coverage note
  **verbatim from the UI**. The disclosure must survive the transition to PDF, because
  the PDF is the artefact that gets emailed, forwarded, and attached to decisions.
- Ordering: place it after AI-text detection and before citation gaps — the two
  provenance signals belong together.

`report` is wave 5 and `similarity` is wave 3, so the data is guaranteed present. When
similarity is `unavailable`, the section renders a one-line "not available for this
document" note rather than being omitted, so a reader can tell the check ran.

---

## 12. Configuration

`similarity/config.py`, pydantic-settings, mirroring `text_detection/config.py`.

| Env var | Default | Purpose |
|---|---|---|
| `SIM_ENABLED` | `true` | Master switch. |
| `SIM_CORE_API_KEY` | `""` | CORE key. Empty ⇒ unauthenticated (heavily rate-limited) — logged as a warning at startup. |
| `SIM_ENABLE_CORE` | `true` | Per-provider toggles… |
| `SIM_ENABLE_ARXIV` | `true` | |
| `SIM_ENABLE_OPENALEX` | `true` | |
| `SIM_ENABLE_CROSSREF` | `true` | |
| `SIM_ENABLE_INTERNAL` | `true` | |
| `SIM_INDEX_UPLOADS` | `false` | **Ships off.** Indexing users' papers into a shared corpus is a consent decision, not a default. See §16 risk 6. |
| `SIM_KGRAM_SIZE` | `5` | Winnowing k, in words. |
| `SIM_WINDOW_SIZE` | `4` | Winnowing w. Guarantees detection of any shared run ≥ `w + k − 1` = 8 words. |
| `SIM_MIN_MATCH_WORDS` | `8` | Minimum reportable match. |
| `SIM_MIN_DOCUMENT_WORDS` | `300` | Below this ⇒ `unavailable`. |
| `SIM_MAX_QUERIES` | `24` | Phrase queries per document. |
| `SIM_MAX_CANDIDATES` | `400` | Candidates verified per document. |
| `SIM_MAX_PDF_FETCHES` | `5` | OA PDFs downloaded per document. |
| `SIM_BUDGET_SECONDS` | `45` | Hard wall-clock cap on retrieval. |
| `SIM_EXCLUDE_QUOTES` | `true` | |
| `SIM_EXCLUDE_BIBLIOGRAPHY` | `true` | |
| `SIM_PARAPHRASE_THRESHOLD` | `0.92` | SciBERT cosine for the paraphrase flag. |
| `SIM_ENABLE_PARAPHRASE` | `true` | |

Every threshold that affects a number is echoed into `Coverage` so any report is
reproducible from its own output.

---

## 13. Testing strategy

TDD per the project rules: test first (RED), minimal implementation (GREEN), refactor,
80% minimum coverage. The architecture is built around making this possible — the entire
scoring core is pure functions over strings and lists, with zero I/O.

### Unit — `tests/similarity/`

| File | What it pins down |
|---|---|
| `test_normalize.py` | Offset map round-trips: for every word in the normalized text, the recorded char offsets slice the *original* text back to the same word. This is what makes highlighting land on the right pixels. |
| `test_fingerprint.py` | **Property test:** for random text pairs sharing a run of ≥ `w+k−1` words, the fingerprint sets always intersect. This is the winnowing correctness guarantee and it is the foundation of every recall claim in this document. |
| `test_exclusions.py` | Bibliography, quotes, and title block are removed from the denominator; a paper that is 100% bibliography yields `unavailable`, not 100%. |
| `test_selector.py` | Phrases are stratified across sections; boilerplate and all-stopword phrases are never selected. |
| `test_matcher.py` | Seed → span extension produces exact word and char boundaries; overlapping seeds merge into one span, not two. |
| `test_aggregate.py` | The scoring spec in §4, case by case: identical text ⇒ 100%; disjoint text ⇒ 0%; **one span claimed by three sources counts once** (the double-count regression); attribution goes to the longest match; bucket percentages are independent and may exceed the headline; matches below `min_match_words` are dropped. |
| `test_semantic.py` | Paraphrase flags never alter `overall_percent` — asserted directly, because this is the invariant most likely to be broken by a future "improvement". |
| `test_privacy.py` | An internal-corpus `SourceMatch` never carries the other document's title, URL, authors, or text. Asserted at the boundary so a UI change cannot leak it. |

### Provider tests — no network

`httpx.MockTransport` with recorded fixtures for CORE / arXiv / OpenAlex / Crossref,
covering: happy path, 429, 5xx, malformed JSON, empty results, and a candidate with no
resolvable URL. Matches the existing "no real DB or LLM required" testing posture in
`backend/tests/conftest.py`.

### Integration — `backend/tests/test_similarity.py`

Router behaviour with a mocked pipeline, via `app.dependency_overrides`: report retrieval,
202 while pending, 404 unknown job, recheck accepted, recheck 409 when already in flight,
status endpoint shape.

### Pipeline test

One end-to-end test with a stub provider returning a known document: assert the pipeline
produces exactly the expected percentage. This is the test that catches integration drift
between the seven modules.

---

## 14. Evaluation and calibration

A similarity number nobody has measured is a guess with a decimal point.
`scripts/eval_similarity.py`, modelled on the existing `scripts/eval_text_detection.py`:

**Construction.** Take N open-access papers with known full text from CORE. Build
synthetic documents with *known* copy rates: 0%, 5%, 10%, 25%, 50% verbatim insertion from
other OA papers, at varying run lengths (10 / 50 / 200 words). Also build a paraphrased
set (LLM-reworded insertions) to measure how much verbatim matching misses by design.

**Metrics.**

| Metric | Target | Why |
|---|---|---|
| Mean absolute error vs. true copy rate | ≤ 3 points at 200-word runs | The headline claim. |
| False-positive rate on the 0% set | **< 1%** | The number that matters most. A clean paper reading 6% destroys trust permanently. |
| Recall by run length | reported, not gated | Honest characterisation of the sampling limit from §3. |
| Paraphrase recall | reported | Quantifies what the verbatim index cannot see, and sizes the paraphrase signal. |
| p50 / p95 latency | p95 ≤ `SIM_BUDGET_SECONDS` | Confirms the budget is real. |

**Calibration.** Tune `SIM_MIN_MATCH_WORDS` and the boilerplate stoplist against the 0%
set until FPR < 1%, *then* measure MAE. Precision first: an inflated score on an honest
paper is a much worse failure than a missed match on a dishonest one. Commit the results
to `similarity_eval_report.md` alongside the existing `agent_accuracy_report.md`.

---

## 15. Build order — phases with exit criteria

Each phase is independently shippable and independently testable. Phases 0-1 deliver a
working, provable feature with no external dependency at all — which is what makes the
external-provider work in Phase 2 low-risk.

| Phase | Work | Exit criteria | Rough size |
|---|---|---|---|
| ✅ **0. Scoring core** | `schema`, `normalize`, `exclusions`, `fingerprint`, `selector`, `matcher`, `aggregate`. Pure functions, no I/O. | All unit tests green incl. the winnowing property test. Feeding a document against itself yields exactly 100%; against unrelated text, 0%. | ~1,000 lines + tests |
| ✅ **1. Internal corpus** | `corpus/internal.py`, `index_writer.py`, Mongo collections + indexes, privacy boundary, corpus purge on job delete. | Upload paper A, then upload A again ⇒ ~100% against the internal corpus, with the other document's identity **not** in the response. Privacy test green. | ~300 |
| ✅ **2. External providers** | `corpus/{base,core_api,arxiv,openalex,crossref}.py`. Lift shared search into `services/academic_search.py`. | Mocked-transport tests green. Manual run against a paper with a known arXiv preprint finds it. | ~550 |
| ✅ **3. Pipeline + agent** | `pipeline.py`, `semantic.py`, `SimilarityAgent`, orchestrator wiring, budget, persistence. | Full analysis job produces `results.similarity`; killing CORE mid-run yields `status="partial"` and a completed job. | ~450 |
| ✅ **4. API** | `reports.py` extension, `routers/similarity.py`, schemas, recheck guard. | Integration tests green; `/docs` shows the endpoints. | ~200 |
| ⬜ **5. Highlighting** | Generalize `PdfHighlightMapper`; single combined mapper invocation. | Similarity spans render as boxes on real page images; existing AI-detection highlights unchanged and their tests still pass. | ~120 |
| ⬜ **6. Frontend** | `ReportPage.jsx` extraction refactor, then the three new components + `api.js`. | Report page renders the block; `ReportPage.jsx` back under 800 lines; `unavailable` renders no percentage. | ~450 |
| ⬜ **7. PDF section** | `report_sections/similarity_section.py`. | Generated PDF contains the block and the coverage note. | ~150 |
| ⬜ **8. Eval + docs** | `scripts/eval_similarity.py`, calibration, `backend.md` + `README.md` updates. | FPR < 1% on the 0% set; `similarity_eval_report.md` committed. | ~300 |

**Phases 0–4 are done and merged** (commits `12cfcb6`, `69c39cd`, `658a727`, `ecc8207`,
`328d8c2`, merged at `506ad88`). Post-merge suite: **175 passed, 2 failed, 1 skipped** —
both failures are `frontend-app/dist` missing in the test environment
(`test_root`, `test_spa_served_on_client_side_routes`), unrelated to similarity.

**Suggested checkpoints:** review after Phase 1 (the scoring is provable and needs no
network — this is where a wrong percentage formula is cheapest to fix), and again after
Phase 3 (the first time it touches a real job).

---

## 16. Risks and mitigations

**1. The number is read as a misconduct verdict.**
*Impact: severe — someone's career.* This is the dominant risk and it is not technical.
Mitigations: the "Corpus Similarity" label instead of "Similarity Index"; mandatory
coverage disclosure in UI *and* PDF; no red/alarm styling; no threshold that produces a
pass/fail; language throughout is "passages to review", never "plagiarism detected".
`status="unavailable"` renders no number at all.

**2. Preprint self-match.** A published paper will match its own arXiv preprint at ~100%
and produce a terrifying, meaningless score. **This is the single most likely first bug
report.** Mitigation: after retrieval and before scoring, drop any candidate whose
normalized title has ≥ 0.9 token overlap with the uploaded paper's title, or whose DOI
matches, or whose author set overlaps by ≥ 50% while similarity is > 60%. Surface dropped
candidates in `coverage.notes` as "1 probable preprint/self-citation excluded" so the
exclusion is visible rather than mysterious. Needs its own test fixture.

**3. Boilerplate false positives.** "The rest of this paper is organized as follows",
standard method descriptions, dataset names, common equations. Mitigation: the
`SIM_MIN_MATCH_WORDS=8` floor; a curated boilerplate stoplist seeded from the eval corpus
(any 8-gram appearing in > 0.5% of sampled papers); exclusion of equations and captions.
Measured directly by the FPR metric in §14.

**4. CORE rate limiting.** Unregistered access is ~5 requests / 10s. Mitigations: register
for a key; shared semaphore + stagger (the `_PROVIDER_SEM` pattern already in the
codebase); `@async_retry` with backoff; phrase-query result caching keyed by phrase hash;
the hard time budget; failures recorded in `coverage.providers_failed`, never swallowed.

**5. Pipeline latency.** Mitigations: wave 3 parallelism, `critical=False`, the 45s hard
budget, and partial results. Worst case the wave takes 45s instead of ~40s.

**6. Corpus consent.** Indexing a user's paper so it can be matched against other users'
papers is a meaningful commitment about their unpublished work. Mitigations:
`SIM_INDEX_UPLOADS` **ships `false`**; enabling it is a deployment decision that requires
a corresponding change to user-facing terms; fingerprints and normalized text are stored,
and the privacy boundary in §7.4 means a match never reveals whose paper it hit. A
deletion path (`DELETE /jobs/{id}` must also purge `corpus_fingerprints` and
`corpus_texts`) is required before this is switched on — **built in Phase 1**.

**7. Scanned PDFs with no text layer.** No text ⇒ no fingerprints ⇒ 0%, which reads as
"clean" and is actually "we couldn't look". Mitigation: `SIM_MIN_DOCUMENT_WORDS` forces
`unavailable`. `PdfHighlightMapper` already flags `hasTextLayer=false` per page; surface
that in `coverage.notes`. `VisualParserAgent` exists but is figure-oriented — OCR fallback
is out of scope here.

**8. Non-English papers.** Normalization is Latin-script-oriented and the corpus skews
English. `PRODUCT.md` already lists non-English support as undecided. Mitigation: detect
and note in `coverage.notes`; do not silently return a meaningless low score.

---

## 17. What we are deliberately NOT building

Per YAGNI, and to keep the first version reviewable:

- **A web crawler.** The "Internet Sources" bucket is open-access academic literature, not
  the web. Anything else is a dishonest label.
- **A commercial API fallback.** Explicitly rejected in §2. The `CorpusProvider` protocol
  means one could be added later as one more provider — but only as a deliberate decision,
  not a hedge.
- **Cross-language similarity.** Requires translation of the whole corpus.
- **Source-code similarity.** Different tokenization, different product.
- **An instructor/reviewer dashboard, class management, submission windows.** Turnitin is
  a workflow product; this is an analysis feature.
- **Auto-generated misconduct reports.** We surface evidence. A human decides.
- **An LLM in the scoring path.** The percentage must be reproducible and explainable. An
  LLM may later *summarize* a report; it must never *compute* one.

---

## 18. Open questions

Not blocking — each has a stated default so implementation can proceed.

1. **CORE API key** — needs registering at `core.ac.uk/services/api`. *Default: build and
   test unauthenticated with a low `SIM_MAX_QUERIES`; register before Phase 8 eval, which
   will otherwise take hours.*
2. **Corpus consent copy** — if `SIM_INDEX_UPLOADS` is ever switched on, the upload page
   needs a line about it. *Default: ships off; revisit when enabling.*
3. **Recheck rate limiting per user** — the in-process guard prevents concurrent rechecks
   of the same job, not a user rechecking forty jobs. *Default: ship the per-job guard;
   add per-user throttling if it becomes a problem.*
4. **Auth on the new endpoints** — the existing routers are unauthenticated and filter by
   `user_email` as a query param, which means anyone with a `job_id` can read a report.
   Similarity data is more sensitive than a keyword list. *Default: match the existing
   pattern for consistency; flag it as a pre-existing gap deserving its own piece of work
   rather than a divergent auth story invented inside this feature.*

---

## Appendix — key source references

- Schleimer, Wilkerson & Aiken (2003), *Winnowing: Local Algorithms for Document
  Fingerprinting* — the algorithm behind MOSS, and behind §4's detection guarantee that
  any shared run of ≥ `w + k − 1` tokens is found.
- [CORE API v3](https://core.ac.uk/services/api) — ~300M metadata records, 40M+ full
  texts, free, `fullText:"..."` phrase query support.
  [Docs](https://api.core.ac.uk/docs/v3).
- [OpenAlex works search](https://docs.openalex.org/api-entities/works/search-works) —
  already integrated in `services/paper_recommender.py`.
- Existing code this plan builds on, and must not duplicate:
  `services/paper_recommender.py` (providers, tokenization, ranking),
  `services/pdf_highlight_mapper.py` (PDF coordinate mapping),
  `services/vector_store_service.py` (SciBERT),
  `services/lancedb_service.py` (vector store),
  `text_detection/` (package shape, degradation posture, status endpoint),
  `workflows/hybrid_workflow.py` (wave semantics and critical-agent behaviour).