Cleanup pass 3: Delete debug_verdict.py, rewrite tasks/todo.md and src/README.md for v2.0

src/README.md

@@ -1,55 +1,58 @@
-# src/ — source code
+# src/ — Source Code
 
-Purpose
-
-All core project code should live in `src/`: data loaders, preprocessing, training scripts, model definitions, and inference API.
-
-Suggested layout
-
-- `src/app.py` — lightweight Flask/FastAPI app for inference and testing
-- `src/train.py` — training orchestration (argument parsing, checkpointing)
-- `src/evaluate.py` — evaluation scripts and metrics
-- `src/data_loader.py` — dataset loaders and helpers
-- `src/preprocessing.py` — normalization, tokenization, and text cleaning
-- `src/models/` — model classes or adapters
-- `src/utils/` — utility functions (logging, metrics, etc.)
-- `src/tests/` — unit and integration tests for core modules
-
-File responsibilities and guidelines
-
-- Keep functions small and well-documented.
-- Expose a stable API for inference (e.g., `predict(text: str) -> dict`). Document the input/output shapes.
-- Add type hints where practical.
-
-Coding conventions
-
-- Use the project style (PEP8). Add a linter config if needed (e.g., `pyproject.toml` or `.flake8`).
-- Write tests for data loaders and small utilities. Place tests under `src/tests/` and use pytest.
-
-What to add when working in `src/`
-
-- Small, focused commits that update a single feature or test.
-- If adding a new script, update top-level `README.md` with a one-line summary and usage example.
-- If you modify input/output shapes for the inference API, update the API docs and communicate in the PR description.
-
-Quick example: inference contract
-
-- Input: Arabic text string (utf-8)
-- Output: JSON object with fields such as `text`, `suggestions` (list), `confidence` (0-1), and optional `edits` with positions
-
-Example output shape
+## Structure
 
 ```
-{
-  "text": "...",
-  "suggestions": [
-    {"start": 5, "end": 12, "replacement": "...", "explanation": "grammar: agreement"}
-  ],
-  "confidence": 0.87
-}
+src/
+├── app.py                  # Flask application — routes + pipeline orchestration
+├── model_loader.py         # Model loading (Spelling, Grammar, Summarization, Autocomplete)
+├── hf_inference.py         # HuggingFace API inference wrappers
+├── index.html              # Main web UI (single-page app)
+├── favicon.svg             # Site icon
+├── css/                    # Stylesheets
+│   ├── tokens.css          # Design tokens (colors, spacing)
+│   ├── base.css            # Base styles
+│   └── components.css      # Component styles
+├── js/                     # Frontend JavaScript modules
+│   ├── editor.js           # Editor logic, events, debouncing
+│   ├── renderer.js         # Offset-based highlight rendering
+│   ├── selection.js        # Cursor/selection save & restore
+│   ├── ui.js               # Tooltips, suggestion lists, scores
+│   ├── api.js              # Backend API fetch wrappers
+│   ├── format.js           # Text formatting
+│   ├── theme.js            # Theme switching
+│   ├── autocomplete.js     # Autocomplete UI
+│   ├── auth/               # Authentication (Supabase)
+│   ├── documents/          # Local document management
+│   ├── documents-cloud/    # Cloud document sync
+│   ├── summaries/          # Text summarization UI
+│   ├── settings-sync/      # Settings sync
+│   ├── sync/               # Real-time sync engine
+│   └── vendor/             # Third-party libraries
+└── nlp/                    # NLP pipeline modules
+    ├── spelling/            # AraSpell spelling correction
+    │   ├── araspell_rules.py
+    │   └── araspell_service.py
+    ├── grammar/             # Grammar correction
+    │   ├── grammar_rules.py
+    │   └── grammar_service.py
+    ├── punctuation/         # Punctuation restoration
+    │   ├── punctuation_rules.py
+    │   └── punctuation_service.py
+    ├── autocomplete/        # Text autocomplete
+    ├── dialect/             # Dialect detection
+    ├── pipeline_context.py  # Shared pipeline state
+    ├── stage_locker.py      # Cross-stage text locking
+    └── correction_patch.py  # Correction patch utilities
 ```
 
-Testing
+## API Contract
+
+- **Input**: Arabic text string (UTF-8)
+- **Output**: JSON with `corrected`, `suggestions[]` (each with `start`, `end`, `replacement`, `explanation`), and `timing_ms`
+
+## Running
 
-- Add unit tests for `preprocessing.py` and `data_loader.py` (happy path + one edge case).
-- Add a small integration test for `app.py` that exercises the prediction endpoint with a short sample input.
+```bash
+cd src && gunicorn app:app --bind 0.0.0.0:7860 --timeout 120 --workers 1
+```

tasks/todo.md

@@ -1,41 +1,52 @@
-# Task List: Phase 1 Editor Stabilization Refactor
+# BAYAN v2.0 — Task List
 
-## Milestone 1: Modularize Editor Logic
-- [ ] Create folder `src/js/` if it doesn't exist
-- [ ] Create `src/js/api.js` to handle all API communications
-- [ ] Create `src/js/editor.js` to manage editor elements, events, and debouncing
-- [ ] Create `src/js/renderer.js` to handle HTML escape and offset-based highlighting
-- [ ] Create `src/js/selection.js` to save and restore range/cursor selections
-- [ ] Create `src/js/ui.js` to handle tooltips, lists, scores, and loading spinners
-- [ ] Update `src/index.html` to load modular JS files instead of inline scripts
-- [ ] Verify that the frontend loads correctly and contains no console errors
+## Phase A: Test Infrastructure ✅
+- [x] Create `tests/v2/test_level1_raw.py` — Raw model tests with TP/FP/FN/TN verdicts
+- [x] Create `tests/v2/test_level2_solo.py` — Solo API endpoint tests
+- [x] Create `tests/v2/test_level3_integrated.py` — Full pipeline tests
+- [x] Create `tests/v2/benchmark_matrix.py` — Master comparison runner
+- [x] Fix verdict logic (strip terminal punctuation before comparison)
+- [x] Run baseline on entities + spelling datasets
+- [ ] Run full 320-test baseline across all 3 levels
 
-## Milestone 2: Selection & Caret Preservation
-- [ ] Implement `saveSelection` and `restoreSelection` based on character offset in `src/js/selection.js`
-- [ ] Implement `getCaretCharacterOffsetWithin` and `setCaretCharacterOffsetWithin`
-- [ ] Integrate selection restore before and after highlight updates
-- [ ] Verify that typing does not cause cursor jumps or selection losses
+## Phase A.1: Project Cleanup ✅
+- [x] Archive legacy scripts (AraSpell.py, Grammer_Rules.py, PuncAra.py)
+- [x] Archive 36 old phase/verification reports
+- [x] Archive 23 old test files + 8 phase10 helpers
+- [x] Delete 35 orphaned debug/temp files
+- [x] Fix .gitignore corruption (binary null bytes)
+- [x] Fix PROJECT_DESCRIPTION.md stale reference
+- [x] Archive docs/audit + docs/audits
 
-## Milestone 3: Backend Offset Support
-- [x] Implement `get_word_positions` in `src/app.py`
-- [x] Implement `OffsetMapper` coordinate transform class in `src/app.py`
-- [x] Rewrite `/api/analyze` in `src/app.py` to calculate exact character offsets (`start`, `end`) for all suggestions
-- [ ] Verify using the test script `reproduce_issue.py` or `test_analyze_api.py` that the backend returns `start` and `end` offsets for all suggestions
+## Phase B: Extract Stages (NOT STARTED)
+- [ ] Create `src/nlp/stages/spelling_stage.py`
+- [ ] Create `src/nlp/stages/grammar_stage.py`
+- [ ] Create `src/nlp/stages/punctuation_stage.py`
+- [ ] Each stage wraps: model call → filter → verdict
+- [ ] Hash (comment out) old inline stage code in `app.py`
+- [ ] Re-run v2 benchmark → must match Phase A baseline
 
-## Milestone 4: Offset-Based Rendering
-- [ ] Write offset-based rendering parser in `src/js/renderer.js`
-- [ ] Ensure the renderer splits the text into highlighted spans and normal text nodes based on sorted, non-overlapping suggestions
-- [ ] Support multiple occurrences of duplicate words by matching exact start/end offsets
+## Phase C: Extract Filters (NOT STARTED)
+- [ ] Create `src/nlp/filters/` module
+- [ ] Extract overlap resolution, religious guard, entity guard
+- [ ] Hash old filter code in `app.py`
+- [ ] Re-run v2 benchmark → must match baseline
 
-## Milestone 5: Secure Rendering
-- [ ] Implement `escapeHtml` utility in `src/js/renderer.js`
-- [ ] Ensure all user inputs, suggestion corrections, and text are escaped before DOM insertion to prevent XSS
+## Phase D: Extract Preprocessors (NOT STARTED)
+- [ ] Create `src/nlp/preprocessors/` module
+- [ ] Extract text normalization, diacritic handling, chunk splitting
+- [ ] Hash old preprocessor code in `app.py`
+- [ ] Re-run v2 benchmark → must match baseline
 
-## Milestone 6: Tooltip Mapping
-- [ ] Assign unique `data-suggestion-id` (index in suggestions list) to each highlight span
-- [ ] Wire click event listeners on the editor container to detect suggestion spans, retrieve the suggestion, and open the tooltip dynamically
+## Phase E: Create Pipeline Orchestrator (NOT STARTED)
+- [ ] Create `src/nlp/pipeline.py` — orchestrates stages via PipelineContext
+- [ ] Wire `app.py` /api/analyze to use `pipeline.run(text)`
+- [ ] Hash old monolithic analyze code in `app.py`
+- [ ] Re-run v2 benchmark → must match baseline
 
-## Milestone 7: Integration & Validation
-- [ ] Launch backend local server using `run_app.py`
-- [ ] Run end-to-end browser check to verify that highlights, suggestions, cursor, and metrics work perfectly
-- [ ] Document final results in `tasks/todo.md` and lessons in `tasks/lessons.md`
+## Phase F: Clean app.py (NOT STARTED)
+- [ ] Move helpers (get_word_positions, OffsetMapper, etc.) to utility modules
+- [ ] Remove all hashed (commented) code blocks
+- [ ] app.py should only contain: Flask routes + pipeline.run() calls
+- [ ] Final v2 benchmark → must match baseline
+- [ ] Target: app.py < 500 lines

tests/v2/debug_verdict.py

@@ -1,10 +0,0 @@
-import json
-
-d = json.load(open('tests/v2/reports/level3_integrated_results.json', 'r', encoding='utf-8'))
-for r in d['results'][:5]:
-    print(f"ID: {r['id']}")
-    print(f"  IN:  {r['input_text']}")
-    print(f"  EXP: {r['expected']}")
-    print(f"  OUT: {r['pipeline_corrected']}")
-    print(f"  V:   {r['verdict']} | {r['detail']}")
-    print()