# Similarity Report Feature — Work Done (Phases 0–4) This document records everything implemented for the **Corpus Similarity Report** feature on branch `report-ui-updated`, per `plan.md` + `opencode-prompt.md`. The feature is complete through Phase 4 (API). Phase 5 (PDF highlight mapper integration) was **not** started. > **Superseded 2026-08-02:** the remote merge described under "Remaining" is DONE > (merge commit `506ad88`). All 9 conflicts resolved; suite is 175 passed / 2 failed > (both `frontend-app/dist` env failures). See the post-merge box at the top of > `plan.md` for what the merge changed. Resume at Phase 5. All 5 commits are local-only (based on `33cb4a2`) and have **not** been pushed. The remote `report-ui-updated` branch has diverged (20+ commits of Groq-key UI, citation-relevance/novelty agents, frontend work, LF line endings) and contains **zero** similarity code. --- ## Commits | Commit | Description | |---|---| | `12cfcb6` | `feat(similarity): Phase 0 scoring core — normalize, exclusions, winnowing, matcher, aggregate` | | `69c39cd` | `feat(similarity): Phase 1 internal corpus — write path, read path, privacy` | | `658a727` | `feat(similarity): Phase 2 external providers — CORE, arXiv, OpenAlex, Crossref` | | `ecc8207` | `feat(similarity): Phase 3 pipeline + agent` | | `328d8c2` | `feat(similarity): Phase 4 API — report, recheck, status` | Working tree currently contains 212 files of pure CRLF-vs-LF noise (`git diff -w` collapses them to 0) plus one dirty LFS binary `models/kbir_semeval2017/training_args.bin`. Plan agreed with user: normalize line endings to LF in a housekeeping commit, then merge the remote. --- ## Files added ### Similarity package (`similarity/`) - `__init__.py` — package export - `config.py` — settings + environment toggles - `schema.py` — `SimilarityReport`, `SourceMatch`, `Coverage`, providers - `normalize.py` — text normalization - `exclusions.py` — stopword/boilerplate exclusion - `fingerprint.py` — winnowing fingerprints (k=5, w=4) - `matcher.py` — matching + attribution - `aggregate.py` — aggregation into report - `index_writer.py` — corpus fingerprint write path - `selector.py` — internal-corpus read path - `pipeline.py` — `SimilarityPipeline` orchestration - `semantic.py` — semantic paraphrase pass - `corpus/base.py` — provider protocol - `corpus/internal.py` — internal corpus provider - `corpus/core_api.py` — CORE provider - `corpus/arxiv.py` — arXiv provider (https base) - `corpus/openalex.py` — OpenAlex provider - `corpus/crossref.py` — Crossref provider ### Agent (`agents/similarity_agent.py`) `SimilarityAgent` (`name="similarity"`, `wave=3`, `critical=False`) — reads paragraph/section text from Neo4j, builds providers per `SIM_ENABLE_*` toggles, persists `SimilaritySource` nodes via `ctx.neo4j.run_write`. ### Phase 4 API - `backend/routers/similarity.py` — `POST /similarity/{job_id}/recheck` (202, 409 in-flight guard, `BackgroundTasks`), `GET /similarity/status` - `backend/schemas/similarity.py` — `SimilarityStatus`, `RecheckAccepted` - `backend/tests/test_similarity.py` — 7 tests (report 200/202/404, recheck accepted/409/404, status shape) ## Files modified - `backend/routers/reports.py` — added `GET /reports/{job_id}/similarity` (200 report / 202 not-ready / 404 unknown), `response_model=SimilarityReport` - `backend/main.py` — router registration - `orchestrators/custom_orchestrator.py` — `get_agents()` + similarity read-off, `upsert_job`, `save_result` (3 spots) - `agents/__init__.py` — export - `configs/pipeline_config.yaml` — `similarity: wave 3, depends_on: [parser, keyword]` - `tests/test_agents.py` — +1 SimilarityAgent short-text/`unavailable` test - New tests: `tests/similarity/test_pipeline.py` (8), `test_self_match_guard.py` (9), `test_semantic.py` (3) --- ## Key algorithm facts (stable) - Winnowing k=5, w=4; coverage bitmap `covered[i] = source_index|None`. - Attribution longest-match, ties by trust (internal > CORE > publisher). - Bucket percents computed independently; status `complete|partial|unavailable` (< `SIM_MIN_DOCUMENT_WORDS`=300 → unavailable). - Headline is verbatim-only; semantic pass never touches `overall_percent`. - Preprint self-match guard: title token-overlap ≥ 0.9, DOI match, author overlap ≥ 50% at similarity > 60%. - Budget timeout → `status="partial"` + `coverage.budget_exhausted=True`. - Provider exceptions caught → `last_error` + `coverage.providers_failed`. - Privacy boundary (plan §7.4): internal matches via `build_source_match()` (neutral label, no URL/authors, stripped `source_excerpt`). - Recheck reuses stored Neo4j text (no PDF re-parse); `save_result` uses `$set` merge so refresh is non-breaking. --- ## Test status (last full run) `tests/ backend/tests/ tests/similarity/` → **212 passed, 7 failed**. All 7 failures are pre-existing and environment-related (not caused by the similarity work) — verified they also fail at Phase 3 commit `ecc8207`: - `tests/test_agents.py::test_scoring_agent_returns_result` - `tests/test_text_pipeline.py::test_no_detectors_enabled_returns_uncertain` - `backend/tests/test_health.py::test_root` (404 — no `frontend-app/dist`) - `backend/tests/test_reports.py::test_download_pdf` (500) - `backend/tests/test_text_detection.py` (3 tests — model returns REAL not AI_GENERATED) `tests/similarity/` alone: 112/112 green. `backend/tests/test_similarity.py`: 7/7. Test command: ``` cd /mnt/c/citation_edge/IIT-Patna && timeout N /tmp/opencode/venv/bin/python -m pytest tests backend/tests tests/similarity -q ``` --- ## Environment notes - venv `/tmp/opencode/venv` (Py 3.12); missing torch/transformers installed (CPU torch). Also installed python-dotenv, aiofiles, pdfplumber, reportlab, pyyaml, tenacity, structlog, motor, pymongo, uvicorn, pymupdf, matplotlib, scikit-learn, keybert, scipy, fastapi/alice httpx, pydantic 2.13.4. - Git auth: pull via askpass script `/tmp/opencode/git-askpass.sh` (Username `x-access-token`), token supplied by user; never written to repo. - git user.name/user.email (per-repo): `Vansh Patil` / `vanshpatil@Vansh.localdomain`. - Commits carry a "Git LFS" hook warning (noise only). --- ## Remaining (Phase 5+) - Phase 5: PDF highlight mapper integration — combined AI+similarity span list, `kind` field on spans, single mapper call (plan §8). - ~~Merge remote `report-ui-updated`~~ **DONE** (`506ad88`) — resolved 9 overlapping files (`backend/main.py`, `backend/routers/reports.py`, `orchestrators/custom_orchestrator.py`, `configs/pipeline_config.yaml`, `agents/__init__.py`, `backend/routers/jobs.py`, `backend/tests/conftest.py`, `backend/tests/test_jobs.py`, `tests/test_agents.py`). - ~~Push local commits~~ **DONE**.