# OpenCode Implementation Prompt — Corpus Similarity Report > Paste everything below the line into OpenCode as the opening prompt. > Run it from the repo root (`IIT-Patna/`). --- ## Your task > **State as of merge `7b89d44`:** Phases 0–4 are implemented, merged with the remote, > and pushed. **Start at Phase 5.** All API routes now live under `/api/*`; read the > post-merge amendments box at the top of `plan.md` before doing anything. Implement the Corpus Similarity Report feature specified in `plan.md` in this repository. Work phase by phase, test-first, committing after each phase. `plan.md` is the **specification and the source of truth**. This prompt tells you how to execute it. Where they conflict, `plan.md` wins on *what* and this prompt wins on *how*. **Read these before writing any code, in this order:** 1. `plan.md` — the full spec. Read all 18 sections. Do not skim §4 (the scoring math) or §7.4 (the privacy boundary); those are the two places a plausible-looking implementation is most likely to be wrong. 2. `text_detection/` — the package you are copying the shape of. Read `config.py`, `schema.py`, `pipeline.py`, `detectors/base.py`. Note: env-prefixed settings, `extra="ignore"`, everything ships disabled, pipeline degrades instead of raising. 3. `services/paper_recommender.py` — the provider clients, rate-limit semaphore (`_PROVIDER_SEM`), `STOPWORDS` / `FILLER_TOKENS`, `source_for()`, `DOI_PUBLISHER`. You will import from this file. Do not copy from it. 4. `services/pdf_highlight_mapper.py` — the existing PDF span→coordinate mapper you will extend in Phase 5. 5. `orchestrators/custom_orchestrator.py` and `workflows/hybrid_workflow.py` — how agents are registered, how waves execute, what `critical=False` actually does. 6. `agents/base_agent.py` — the `BaseAgent` / `AgentContext` / `AgentResult` contract. 7. `backend/backend.md` — the API conventions you must match. --- ## Stack facts you need - Python 3.13, FastAPI, `pydantic` v2, `pydantic-settings`, `motor` (async Mongo), `neo4j` (sync driver), `httpx`, `pytest` + `pytest-asyncio`. - Frontend: React 18 + Vite, plain JSX (no TypeScript), one CSS file per page/component. - Run the backend: `uvicorn backend.main:app --reload --port 8000` - Run tests: `python -m pytest tests/similarity/ -v` and `python -m pytest backend/tests/ -v` - Build the frontend: `cd frontend-app && npm run build` - Windows dev machine. Use `pathlib`, never string concatenation, for paths. --- ## Non-negotiable constraints Violating any of these means the work gets rejected regardless of whether it runs. 1. **No new runtime dependencies.** `requirements.txt` must not grow. Winnowing is stdlib. If you believe you need a package, stop and ask — you almost certainly do not. 2. **TDD, genuinely.** Write the failing test, run it, watch it fail for the right reason, then implement. A test written after the code is not a test, it is a description. Minimum 80% coverage on the `similarity/` package. 3. **File size:** 200–400 lines typical, 800 hard maximum. Two files are already at the edge — see "Traps" below. 4. **No mutation.** Build new objects; do not modify inputs in place. The scoring functions in `similarity/aggregate.py` must be pure. 5. **Never swallow an error.** A provider that 429s goes into `coverage.providers_failed`. A span that cannot be located gets logged and omitted, never guessed. There is no bare `except: pass` anywhere in this feature. 6. **No LLM in the scoring path.** The percentage is computed arithmetically from verified spans. `LLMService` must not be imported anywhere under `similarity/`. 7. **The privacy boundary in `plan.md` §7.4 is a hard requirement.** An internal-corpus match must never carry the other document's title, URL, authors, owner, or text. This is enforced at construction in `corpus/internal.py`, not filtered downstream. There is a test for it; do not weaken the test. 8. **Never fabricate a source.** Every `SourceMatch.url` must be a real, resolvable URL returned by a provider. A candidate without a URL is dropped, exactly as `_score_candidates()` in `paper_recommender.py` already does. --- ## Patterns to follow | Need | Copy from | Not from | |---|---|---| | Package layout, settings, degradation | `text_detection/` | anywhere else | | Env-prefixed settings class | `text_detection/config.py` (`TEXTDET_` → yours is `SIM_`) | `backend/config.py` | | Async provider client + retry + throttle | `services/paper_recommender.py` (`@async_retry`, `_PROVIDER_SEM`) | write your own | | Agent class shape | `agents/citation_gap_agent.py` (176 lines, `critical=False`, wave 3) | `agents/report_agent.py` (775 lines) | | Router shape | `backend/routers/text_detection.py` | `backend/routers/reports.py` | | Test fixtures / dependency overrides | `backend/tests/conftest.py` | write your own | | Status endpoint | `GET /text-detection/status` | invent a new shape | --- ## Execution order Do these in order. **Commit at the end of each phase**, with a conventional-commit message (`feat:`, `test:`, `refactor:`). Do not start a phase until the previous phase's exit criteria pass. ### Phase 0 — Scoring core (no I/O) Create `similarity/{__init__,config,schema,normalize,exclusions,fingerprint,selector,matcher,aggregate}.py`. Everything here is a pure function over strings and lists. No network, no database, no async. This is deliberate: it is the part that must be provably correct, so it must be testable in milliseconds. Implement in this order, each with its tests first: 1. `schema.py` — the Pydantic models exactly as written in `plan.md` §6.1. Copy the field names verbatim; later phases and the frontend depend on them. 2. `config.py` — `SimilaritySettings(BaseSettings)` with `env_prefix="SIM_"`, `extra="ignore"`, every field and default from `plan.md` §12. Module-level `settings = SimilaritySettings()`, matching `text_detection/config.py`. 3. `normalize.py` — `normalize(text) -> NormalizedDoc` carrying the word list plus an offset map back into the original string. **The offset map is what makes highlighting land on the right pixels.** Test it by round-tripping every word. 4. `exclusions.py` — find bibliography, quoted text, title block, equations, captions; return index ranges to exclude. Reuse `utils.pdf.split_into_sections` for the bibliography — `ParserAgent` already isolates it. 5. `fingerprint.py` — k-gram hashing + winnowing. `k = settings.kgram_size` (5 words), `w = settings.window_size` (4). The correctness test is a **property test**: generate random text pairs sharing a run of ≥ `w+k-1` words, assert the fingerprint sets always intersect. That guarantee is the foundation of every recall claim in the spec. 6. `selector.py` — pick the query phrases per `plan.md` §3 ("Choosing which phrases"). Import `STOPWORDS` and `FILLER_TOKENS` from `services.paper_recommender`. 7. `matcher.py` — fingerprint seeds → verified spans with exact word and char offsets. Overlapping seeds merge into one span. 8. `aggregate.py` — the coverage bitmap and all four percentage calculations from `plan.md` §4. This is the most important file in the feature. **Exit criteria:** - `python -m pytest tests/similarity/ -v` — all green, including the winnowing property test. - A document scored against itself returns exactly `100`. - A document scored against unrelated text returns exactly `0`. - `test_aggregate.py` covers every case listed in `plan.md` §13, especially: one span claimed by three sources counts **once**. ### Phase 1 — Internal corpus Create `similarity/corpus/{__init__,base}.py`, `similarity/corpus/internal.py`, and `similarity/index_writer.py`. - `base.py` — the `CorpusProvider` protocol and `Candidate` dataclass from `plan.md` §7. - Mongo collections `corpus_fingerprints` and `corpus_texts` per `plan.md` §6.2. Create the multikey index on `fingerprints` at startup — add it beside the existing index creation in the orchestrator, not on every query. - `index_writer.py` — gated on `settings.index_uploads`, which defaults to `False`. - **Also in this phase:** extend `DELETE /jobs/{job_id}` in `backend/routers/jobs.py` to purge that document's `corpus_fingerprints` and `corpus_texts` rows. `plan.md` §16 risk 6 requires this before the corpus can ever be enabled; build it now while the write path is fresh, not later. - Self-match guard: exclude by `doc_id`, and on recheck by `job_id`. **Exit criteria:** - Index document A, then score A against the corpus ⇒ ~100%. - `tests/similarity/test_privacy.py` green: the returned `SourceMatch` for an internal hit carries no title, URL, author, or text from the other document — only the neutral label `CitationEdge Corpus · submitted YYYY-MM-DD`. - Deleting a job removes its corpus rows. ### Phase 2 — External providers Create `similarity/corpus/{core_api,arxiv,openalex,crossref}.py`. - **CORE is the workhorse.** `GET https://api.core.ac.uk/v3/search/works` with `q=fullText:""`. The response can carry `fullText`, so retrieval and text fetch are one round trip. Send `Authorization: Bearer ` when the key is set; log a startup warning when it is not (unregistered is ~5 requests / 10 seconds and will rate-limit you immediately). - arXiv / OpenAlex / Crossref: **import** `_search_arxiv`, `_search_openalex`, `_search_crossref`, `source_for`, and `DOI_PUBLISHER` from `services.paper_recommender`. If clean importing requires lifting them into a shared `services/academic_search.py` that both modules import, do that refactor — it is in-scope. Do not duplicate the functions. - Every provider: shared `asyncio.Semaphore(2)`, stagger between calls, `@async_retry` from `utils.retry`, and on persistent failure append to `coverage.providers_failed` and return `[]`. **Tests: no network.** Use `httpx.MockTransport` with fixtures covering happy path, 429, 5xx, malformed JSON, empty results, and a candidate with no resolvable URL. **Exit criteria:** provider tests green; one manual run against a paper with a known arXiv preprint locates it. ### Phase 3 — Pipeline and agent Create `similarity/{pipeline,semantic}.py` and `agents/similarity_agent.py`. - `pipeline.py` — orchestrates normalize → exclude → select → retrieve → verify → match → aggregate → report. Wrap the whole retrieval stage in `asyncio.wait_for(..., timeout=settings.budget_seconds)`; on timeout return what is verified so far with `status="partial"` and `coverage.budget_exhausted=True`. - **Implement the preprint self-match guard here** (`plan.md` §16 risk 2) — drop candidates whose normalized title overlaps the uploaded title by ≥ 0.9 tokens, or whose DOI matches, or whose authors overlap ≥ 50% while similarity > 60%. Record the drop in `coverage.notes`. This needs its own test fixture. Skipping it produces a ~100% score on every already-published paper. - `semantic.py` — the paraphrase pass using the existing `VectorStoreService`. Assert in a test that it **cannot** change `overall_percent`. - `agents/similarity_agent.py` — `name = "similarity"`, `wave = 3`, `critical = False`. Keep it under 100 lines: read paragraphs from Neo4j (reuse the query the orchestrator already runs for AI-text detection), call the pipeline, write summary nodes, return the dict. - Wire into `orchestrators/custom_orchestrator.py`: add to `get_agents()`, read the result off the `AgentResult` list, add `similarity` to `final_result`, `upsert_job`, and `save_result`. `save_result` merges with `$set`, so adding a key is non-breaking. - Add the `similarity` entry to `configs/pipeline_config.yaml` (wave 3, `depends_on: [parser, keyword]`). **Exit criteria:** a full analysis job writes `results.similarity`; simulating a CORE outage yields `status="partial"` **and a completed job** — not a failed one. ### Phase 4 — API - `GET /reports/{job_id}/similarity` in `backend/routers/reports.py`. - New `backend/routers/similarity.py`: `POST /{job_id}/recheck` (202, `BackgroundTask`, reuses stored normalized text — **no PDF re-parse**) and `GET /status`. - New `backend/schemas/similarity.py` for the request/status models. Reuse `similarity.schema.SimilarityReport` directly as the response model, the way `text_detection` reuses `EnsembleResult` — do not mirror it. - Register the router in `backend/main.py`. - Recheck needs a `409` guard: an in-process set keyed by `job_id`, so four clicks do not launch four concurrent 24-query provider storms. **Exit criteria:** `backend/tests/test_similarity.py` green (report, 202 pending, 404 unknown, recheck accepted, recheck 409, status shape); endpoints visible at `/docs`. ### Phase 5 — Highlighting Modify `services/pdf_highlight_mapper.py`. - Generalize the hardcoded span typing. Today `_match_span()` does `"type": "ai" if span.get("verdict") == "AI_GENERATED" else "human"` and `run()` filters to `verdict in ("AI_GENERATED", "REAL")`. Add an optional `kind` on the input span that passes through when present, plus an optional `meta` dict carrying `source_index` for per-source colouring. **Default behaviour must not change** — the existing AI-detection tests must pass untouched. - Call the mapper **once**, not twice. It rasterizes into `pages_dir` and returns the full `pages` array; two callers would race on the PNGs and produce two `page_highlights` structures the frontend cannot merge. **The sole call site is `agents/report_agent.py:327`** — not the orchestrator. `report` is wave 5 and `similarity` is wave 3, so ReportAgent already has both span sets. Do not add a second call site; extend the existing one with the similarity spans, each tagged with `kind`. **Exit criteria:** similarity spans render as boxes on real page images; every existing `pdf_highlight_mapper` test still passes with no edits. ### Phase 6 — Frontend **Refactor first.** `frontend-app/src/pages/ReportPage.jsx` is **826 lines — already over the 800-line cap**. Before adding anything: - Move `buildReport()` and its helpers → `pages/reportAdapter.js` - Move the `DEMO` constant → `pages/reportDemo.js` - Verify the page still renders identically before continuing. Then create: - `components/SimilarityIndex.jsx` + `.css` — the header block: one large percentage, three bucket figures. Brand rules from `DESIGN.md`: obsidian background, glass card, Instrument Serif for the number, DM Mono for the buckets. **Not red.** Red is the error colour in this brand; colouring a similarity score red makes an accusation the data does not support. - `components/SimilaritySources.jsx` — the numbered, colour-chipped source list. Internal corpus rows render as plain text, never links. - `components/CoverageNote.jsx` — the disclosure from `plan.md` §10.2, always visible, never behind a toggle. - `api.js` — `getSimilarity(jobId)`, `recheckSimilarity(jobId)`. Wire into `ReportPage.jsx` in ~8 lines. Pass `kind === 'similarity'` highlights to the existing page viewer. **Exit criteria:** page renders the block; `ReportPage.jsx` back under 800 lines; `status: "unavailable"` renders **no percentage at all**; `npm run build` clean. ### Phase 7 — PDF report section - Create `agents/report_sections/similarity_section.py` exporting `build_similarity_section(report: dict) -> list[Flowable]`. - `agents/report_agent.py` is 775 lines — it gets a two-line import-and-append, nothing more. - Place the section after AI-text detection, before citation gaps. - The coverage note must appear **verbatim** in the PDF. The PDF is the artefact that gets emailed and attached to decisions; the disclosure has to travel with it. - When similarity is `unavailable`, render a one-line "not available for this document" note rather than omitting the section, so a reader can tell the check ran. ### Phase 8 — Evaluation - `scripts/eval_similarity.py`, modelled on `scripts/eval_text_detection.py`. - Build synthetic documents with known copy rates (0 / 5 / 10 / 25 / 50%) at run lengths 10 / 50 / 200 words, from CORE open-access papers. - Tune `SIM_MIN_MATCH_WORDS` and the boilerplate stoplist against the 0% set until the false-positive rate is **< 1%**, *then* measure mean absolute error. Precision first: an inflated score on an honest paper is far worse than a missed match on a dishonest one. - Write `similarity_eval_report.md` with the measured numbers. Do not write target numbers as if they were results. - Update `backend/backend.md` (new endpoints) and `README.md` (feature list). --- ## Traps — read before you start These are real conditions in this repo that will bite you. 1. **`ReportPage.jsx` is 869 lines, already over the cap.** Adding to it without the Phase 6 extraction makes a bad file worse. 2. **`report_agent.py` is 979 lines.** Well past the cap. The Phase 7 extraction is a precondition for Phase 5 touching that file, not a later cleanup. 3. **`PdfHighlightMapper` rasterizes pages as a side effect.** Two callers race. One call with a combined span list. 4. **`save_result()` merges with `$set`.** Adding `similarity` is safe; replacing the document is not. 5. **`critical = True` aborts the pipeline.** Read `HybridWorkflow.execute` — a failed critical agent breaks the wave loop. Similarity must be `critical = False` or a CORE outage costs users their entire analysis. 6. **A published paper matches its own preprint at ~100%.** The Phase 3 guard is not optional polish; without it the feature is unusable on real papers. 7. **CORE unauthenticated is ~5 requests / 10 seconds.** 24 phrase queries will 429 instantly. Throttle from the first line of code, not after you see the error. 8. **Bucket percentages do not sum to the headline number** (`plan.md` §4.4). This is correct and matches Turnitin. Do not "fix" it. 9. **Scanned PDFs have no text layer** ⇒ no fingerprints ⇒ 0%, which reads as "clean" but means "we could not look". `SIM_MIN_DOCUMENT_WORDS` must force `unavailable`. --- ## Stop and ask if - A phase's exit criteria cannot be met without changing the spec in `plan.md`. - You believe a new dependency is required. - The privacy boundary in §7.4 seems to conflict with a UI requirement. - CORE's API shape differs from what `plan.md` §7.1 describes (the spec was written from their docs; verify against the live API in Phase 2 and report any mismatch rather than silently adapting). - Measured false-positive rate in Phase 8 stays above 1% after calibration — that is a product decision, not a tuning problem. Do **not** stop to ask whether to continue between phases. Finish the phase, commit, report the exit criteria you verified, and start the next one. --- ## Reporting After each phase, report: - Which exit criteria you verified, and the **actual command output** that proves it. - Any file that grew past 400 lines, and why. - Anything in `plan.md` that turned out to be wrong once you hit the real code. Do not report a phase complete on the basis of code that looks right. Run the tests and paste what they said. If something fails and you cannot fix it, say so plainly and move to what you can finish.