handoff doc

HANDOFF.md

@@ -0,0 +1,494 @@
+# CourtMitra MissionOS — Handoff Document
+
+## 1. What This Is
+
+CourtMitra is a citizen-side grievance/legal-action autopilot for India. A user describes a problem once (WhatsApp / Telegram / browser); the system turns it into a **mission**, builds the evidence file, grounds it in Indian law/procedure, picks an official route, drafts + sends approved actions, follows up, and records the outcome.
+
+### Architecture
+
+```
+apps/
+  web/         → Next.js App Router (port 3000)
+  api/         → NestJS + Fastify (port 3001)
+  worker/      → Inngest functions (port 3002)
+packages/
+  domain/      → Pure business logic (no framework imports)
+  ports/       → Interfaces only (hexagonal boundary)
+  adapters/    → Concrete implementations (LLM, email, channels, etc.)
+  db/          → Drizzle schema, migrations, seeds
+  contracts/   → Zod schemas (single source of truth)
+  prompts/     → Versioned prompt templates
+  ui/          → Shared React components
+infra/
+  docker-compose.yml  → Postgres, Inngest dev server, API, Worker, Web
+  Dockerfile.*        → Multi-stage builds
+```
+
+### How It Works
+
+1. **Intake**: User sends a message via WhatsApp/Telegram/browser → system extracts structured facts via LLM
+2. **Evidence**: User uploads screenshots/PDFs/audio → vision-LLM or ASR extracts data
+3. **Facts**: User confirms extracted facts
+4. **Legal grounding**: pgvector + FTS retrieves relevant Indian law citations
+5. **Route plan**: LLM generates a step-by-step plan grounded in law
+6. **Action preview + consent**: System drafts an action, user approves with hash-bound consent
+7. **Execution**: Resend sends email, portal fills forms, etc.
+8. **Follow-up**: System tracks replies, schedules reminders
+9. **Resolution**: Outcome recorded, anonymized ledger entry
+
+---
+
+## 2. Environment Variables — Complete Reference
+
+Copy `infra/.env.example` to `infra/.env` and fill in the values below.
+
+### Required (system won't start without these)
+
+| Variable | Where to Get | Notes |
+|----------|-------------|-------|
+| `DATABASE_URL` | Auto-set by Docker compose | `postgres://courtmitra:courtmitra@localhost:5432/courtmitra` for local dev |
+| `DIRECT_DATABASE_URL` | Same as DATABASE_URL | Used by Drizzle for direct connections |
+| `AI_PROVIDER` | Choose one | `openrouter` (default) or `opencode` |
+| `OPENROUTER_API_KEY` | https://openrouter.ai/keys | Free tier available. Used for LLM calls. |
+| `OPENAI_API_KEY` | https://platform.openai.com/api-keys | Required for Whisper ASR (voice notes). Free tier: $5 credit. |
+
+### Required for Email Sending
+
+| Variable | Where to Get | Notes |
+|----------|-------------|-------|
+| `RESEND_API_KEY` | https://resend.com/api-keys | Free tier: 100 emails/day, 3000/month |
+| `RESEND_FROM` | Set in Resend dashboard | e.g. `noreply@yourdomain.com` |
+| `RESEND_REPLY_DOMAIN` | Your domain | e.g. `reply.yourdomain.com` — used for reply-token threading |
+
+### Required for Vector Search
+
+| Variable | Where to Get | Notes |
+|----------|-------------|-------|
+| `QDRANT_URL` | https://cloud.qdrant.io | Free tier: 1GB storage, 1 cluster |
+| `QDRANT_API_KEY` | Qdrant Cloud dashboard | Created when you set up the cluster |
+| `QDRANT_COLLECTION` | Just set | `courtmitra_legal` (default) |
+
+### Optional — WhatsApp
+
+| Variable | Where to Get | Notes |
+|----------|-------------|-------|
+| `WHATSAPP_VERIFY_TOKEN` | You create this | Any random string for webhook verification |
+| `WHATSAPP_APP_SECRET` | Meta Developer Portal | For HMAC signature verification |
+| `WHATSAPP_ACCESS_TOKEN` | Meta Developer Portal | Permanent token from App Dashboard |
+| `WHATSAPP_PHONE_NUMBER_ID` | Meta Developer Portal | Found in WhatsApp > API Setup |
+| `WHATSAPP_BUSINESS_ACCOUNT_ID` | Meta Developer Portal | Found in WhatsApp > API Setup |
+
+### Optional — Telegram
+
+| Variable | Where to Get | Notes |
+|----------|-------------|-------|
+| `TELEGRAM_BOT_TOKEN` | @BotFather on Telegram | `/newbot` command |
+
+### Optional — Portal Automation
+
+| Variable | Where to Get | Notes |
+|----------|-------------|-------|
+| `BROWSERBASE_API_KEY` | https://www.browserbase.com | Free tier: 5 hours/month |
+| `BROWSERBASE_PROJECT_ID` | Browserbase dashboard | Created when you sign up |
+| `ENABLE_PORTAL_AUTOMATION` | Set to `1` | Without this, portal assist degrades to manual packet |
+
+### Optional — eCourts CNR Lookup
+
+| Variable | Where to Get | Notes |
+|----------|-------------|-------|
+| `SUREPASS_API_KEY` | https://surepass.io | Paid API for Indian court case data |
+
+### Optional — Observability
+
+| Variable | Where to Get | Notes |
+|----------|-------------|-------|
+| `LANGFUSE_PUBLIC_KEY` | https://langfuse.com | Free tier available |
+| `LANGFUSE_SECRET_KEY` | Langfuse dashboard | |
+
+### Defaults (usually fine as-is)
+
+| Variable | Default | Notes |
+|----------|---------|-------|
+| `APP_BASE_URL` | `http://localhost:3000` | Web app URL |
+| `API_BASE_URL` | `http://localhost:3001` | API URL |
+| `WEBHOOK_BASE_URL` | `http://localhost:3001` | For Resend/WhatsApp webhook callbacks |
+| `LOCAL_STORAGE_DIR` | `.storage` | File storage path (relative to repo root) |
+| `INNGEST_BASE_URL` | `http://localhost:8288` | Inngest dev server |
+| `INNGEST_DEV` | `1` | Enable Inngest dev mode |
+| `PII_HASH_SALT` | `dev-salt-change-me` | For PII hashing |
+| `JWT_SECRET` | `dev-jwt-secret-change-me` | For session tokens |
+| `AI_MODEL` | `google/gemini-2.0-flash-exp:free` | LLM model (free on OpenRouter) |
+| `AI_VISION_MODEL` | `google/gemini-2.0-flash-exp:free` | Vision model for evidence parsing |
+
+---
+
+## 3. Step-by-Step Setup
+
+### Prerequisites
+
+- Node.js >= 20
+- pnpm 10.20.0+ (`corepack enable && corepack prepare pnpm@10 --activate`)
+- Docker + Docker Compose
+- Git
+
+### Step 1: Clone and Install
+
+```bash
+git clone <repo-url> hackathon
+cd hackathon
+pnpm install
+```
+
+### Step 2: Start Infrastructure
+
+```bash
+# Start Postgres + Inngest dev server
+pnpm infra:up
+
+# Wait for Postgres to be healthy (check with:)
+docker ps
+# Should show courtmitra_pg as "healthy"
+```
+
+### Step 3: Set Up Environment
+
+```bash
+cp infra/.env.example infra/.env
+# Edit infra/.env with your API keys (see table above)
+```
+
+**Minimum viable .env for local dev:**
+```bash
+DATABASE_URL=postgres://courtmitra:courtmitra@localhost:5432/courtmitra
+DIRECT_DATABASE_URL=postgres://courtmitra:courtmitra@localhost:5432/courtmitra
+AI_PROVIDER=openrouter
+OPENROUTER_API_KEY=<your-key>
+OPENAI_API_KEY=<your-key>
+RESEND_API_KEY=<your-key>
+RESEND_FROM=noreply@yourdomain.com
+RESEND_REPLY_DOMAIN=reply.yourdomain.com
+QDRANT_URL=<your-qdrant-cloud-url>
+QDRANT_API_KEY=<your-qdrant-key>
+QDRANT_COLLECTION=courtmitra_legal
+LOCAL_STORAGE_DIR=.storage
+INNGEST_BASE_URL=http://localhost:8288
+INNGEST_DEV=1
+PII_HASH_SALT=dev-salt-change-me
+JWT_SECRET=dev-jwt-secret-change-me
+AI_MODEL=google/gemini-2.0-flash-exp:free
+AI_VISION_MODEL=google/gemini-2.0-flash-exp:free
+```
+
+### Step 4: Run Database Migrations
+
+```bash
+pnpm db:migrate
+```
+
+This creates all 30+ tables including: users, missions, actions, evidence, legal_chunks, followups, social_posts, case_lookups, portal_assist_sessions, etc.
+
+### Step 5: Seed Legal Data + Qdrant
+
+```bash
+# Seed legal sources, chunks, and route rules into Postgres
+pnpm db:seed
+
+# Seed Qdrant with embeddings (requires OPENROUTER_API_KEY for embeddings)
+npx tsx scripts/seed-qdrant.ts
+```
+
+### Step 6: Start the Application
+
+```bash
+# Start all three services (API, Worker, Web)
+pnpm dev
+```
+
+This starts:
+- **Web** on http://localhost:3000
+- **API** on http://localhost:3001
+- **Worker** on http://localhost:3002
+- **Inngest Dev Server** on http://localhost:8288 (from Docker)
+
+### Step 7: Verify Everything Works
+
+```bash
+# Run all checks
+pnpm typecheck && pnpm lint && pnpm test && pnpm test:contracts
+
+# Check API health
+curl http://localhost:3001/api/admin/health
+
+# Check capabilities (shows what's available/blocked)
+curl http://localhost:3001/api/capabilities
+```
+
+---
+
+## 4. How to Test the Product End-to-End
+
+### Test 1: Create a Mission from Browser
+
+1. Open http://localhost:3000
+2. Click "Create Mission"
+3. Select issue domain (consumer/payment/cyber_fraud)
+4. Enter title and description
+5. Click create → mission appears on the board
+
+### Test 2: Send a Telegram Message (Easiest Channel Test)
+
+1. Set `TELEGRAM_BOT_TOKEN` in `.env`
+2. Restart the API
+3. Send a message to your bot on Telegram
+4. Check http://localhost:3000 — mission should appear
+5. Check Inngest dev server at http://localhost:8288 — events should flow
+
+### Test 3: Full Loop (Email Send)
+
+1. Create a mission from browser
+2. Confirm facts (click "Confirm Facts" in the mission room)
+3. Wait for route plan generation (check Inngest)
+4. Preview an action → approve with consent
+5. Execute → email sent via Resend
+6. Check proof_state = `sent_by_email`
+7. Reply to the email → reply ingested → mission advances
+
+### Test 4: WhatsApp (Requires Meta App)
+
+1. Set WhatsApp env vars
+2. Configure webhook URL in Meta Developer Portal: `https://your-domain/webhooks/whatsapp`
+3. Set verify token to match `WHATSAPP_VERIFY_TOKEN`
+4. Send a voice note or text message
+5. Mission created, evidence parsed, facts extracted
+
+### Test 5: Voice Notes (Requires OPENAI_API_KEY)
+
+1. Send a voice note via WhatsApp or Telegram
+2. System downloads audio → transcribes via Whisper → includes transcript in intake
+3. Check evidence_parsed_outputs for parserName = "whisper-asr"
+
+---
+
+## 5. What's Built (Phase 1 + Phase 2 Modules 1-8)
+
+### Phase 1 — Complete Automation Loop ✅
+
+| Component | Status |
+|-----------|--------|
+| Mission CRUD from browser | ✅ |
+| WhatsApp webhook + media download | ✅ |
+| Telegram webhook + voice | ✅ |
+| Intake (LLM extracts structured facts) | ✅ |
+| Vision-LLM evidence parsing | ✅ |
+| Whisper ASR transcription | ✅ |
+| Fact confirmation UI | ✅ |
+| Legal grounding (pgvector + FTS + RRF) | ✅ |
+| Route plan generation with citations | ✅ |
+| Claim verification (literal substring) | ✅ |
+| Action preview + consent hash | ✅ |
+| Defamation gate | ✅ |
+| PII redaction (Aadhaar/PAN/UPI/IFSC/phone/email) | ✅ |
+| Email send via Resend | ✅ |
+| Reply-token threading | ✅ |
+| Reply classification | ✅ |
+| Follow-up scheduling | ✅ |
+| Resolution capture | ✅ |
+| Public case page (private link) | ✅ |
+| DPDP data export + erase | ✅ |
+| Capability registry | ✅ |
+
+### Phase 2 — Modules 1-8 ✅
+
+| Module | What's Built |
+|--------|-------------|
+| **1. Complaint Swarm** | Entity resolution, swarm clustering, dossier generation, opt-in with consent |
+| **2. ASR / Voice Notes** | Whisper API transcription, WhatsApp audio download, Telegram voice, PII redaction |
+| **3. Indic Translation** | LLM-based language detection + translation, preferredLanguage on users, localized_texts table |
+| **4. Hybrid RAG** | Qdrant Cloud adapter (dense vectors + Postgres FTS + RRF), seed script |
+| **5. Portal Automation** | Playwright adapter, portal_assist_sessions, consent gate, receipt capture, degradation |
+| **6. Company Resolver** | Organizations, resolver threads, domain verification, masked case view |
+| **7. Social Publishing** | Social accounts/posts tables, copy-ready fallback, social card image generator |
+| **8. eCourts / CNR** | Surepass API adapter, case_lookups table, hearing reminder scheduling |
+
+### Phase 2 — Modules 9-11 ❌ (Not Built)
+
+| Module | What's Needed |
+|--------|--------------|
+| **9. Intermediary Compliance** | Takedown requests table, Grievance Officer contact, IT Rules 2021 timelines |
+| **10. AI Public Account** | @CourtMitraWatch — requires modules 7+9, legal review |
+| **11. Advanced Ledger** | Materialized views, route-success rates, repeat-offender clusters |
+
+---
+
+## 6. Known Issues / Limitations
+
+### Architecture
+- **No auth system** — browser missions use a hardcoded "Demo User". No login, no session management.
+- **Single-user demo** — all browser missions attach to the same demo user.
+- **Local file storage** — evidence files stored on local filesystem (`.storage/`). Production needs R2/Supabase Storage.
+
+### Data
+- **42 curated legal chunks** across 3 domains. No automated PDF ingestion (Docling skipped).
+- **3 route rules** (consumer, payment, cyber_fraud). No employment/RTI/landlord domains.
+- **All social posts are copy-ready** — no Meta App Review completed, no X API payment.
+
+### Code Quality
+- **26 lint warnings** — mostly `any` types in existing code, unused imports.
+- **No integration tests** — only unit tests with mocked DB.
+- **Some `as never` casts** in Inngest event sends (type safety gap).
+
+### Docker Images
+- `infra-api`: ~404MB
+- `infra-worker`: ~384MB
+- `infra-postgres`: ~640MB (pgvector base)
+
+---
+
+## 7. Key Commands Reference
+
+```bash
+# Development
+pnpm dev                    # Start all services (web + api + worker)
+pnpm build                  # Build all packages
+pnpm typecheck              # TypeScript type checking
+pnpm lint                   # ESLint
+pnpm test                   # Run all tests
+pnpm test:contracts         # Run contract tests only
+
+# Database
+pnpm db:migrate             # Run Drizzle migrations
+pnpm db:seed                # Seed legal data + route rules
+npx tsx scripts/seed-qdrant.ts  # Seed Qdrant with embeddings
+
+# Docker
+pnpm infra:up               # Start Postgres + Inngest
+pnpm infra:down             # Stop all containers
+docker compose -f infra/docker-compose.yml up -d    # Same as above
+docker compose -f infra/docker-compose.yml down     # Same as above
+
+# Verification (run before committing)
+pnpm typecheck && pnpm lint && pnpm test && pnpm test:contracts
+```
+
+---
+
+## 8. File Structure (Key Files)
+
+```
+CLAUDE.md                          # Project invariants — READ THIS FIRST
+PHASE_1_automation_loop.md         # Phase 1 spec + DoD
+PHASE_2_scale_and_intelligence.md  # Phase 2 spec (11 modules)
+
+packages/
+  db/src/schema.ts                 # All 30+ table definitions
+  db/src/migrations/               # 0000 through 0009
+  db/src/seeds/                    # legal-data.ts, route-rules-data.ts, run.ts
+  contracts/src/                   # All Zod schemas (enums, mission, action, swarm, asr, etc.)
+  ports/src/index.ts               # All port interfaces
+  adapters/src/                    # All adapter implementations
+    legal/pgvector-fts.ts          # Original Postgres RAG adapter
+    legal/qdrant-hybrid.ts         # Qdrant Cloud hybrid adapter
+    audio/whisper-api.ts           # Whisper ASR adapter
+    translation/llm-adapter.ts     # LLM translation adapter
+    portal/playwright-adapter.ts   # Playwright portal adapter
+    social/copy-ready-fallback.ts  # Social publishing fallback
+    social/card-generator.ts       # Social card image generator
+    case-lookup/surepass-adapter.ts # eCourts CNR adapter
+    channel/whatsapp.ts            # WhatsApp Cloud API
+    channel/telegram.ts            # Telegram Bot API
+    submission/resend-email.ts     # Resend email adapter
+    redaction/regex.ts             # PII regex redaction
+    redaction/regex-llm.ts         # LLM-assisted redaction
+    storage/local-fs.ts            # Local filesystem storage
+    index.ts                       # Adapter exports + capability registry
+
+apps/
+  api/src/
+    app.module.ts                  # Root NestJS module (all modules wired)
+    workflow/inngest.module.ts     # Inngest client + WorkflowService
+    actions/                       # Action preview, consent, execute
+    missions/                      # Mission CRUD + reply ingestion
+    evidence/                      # Evidence files + fact confirmation
+    webhooks/                      # WhatsApp, Telegram, email webhook handlers
+    conversations/                 # Message ingestion service
+    route-plans/                   # Route plan generation trigger
+    resolution/                    # Resolution capture
+    ledger/                        # Anonymized ledger
+    dpdp/                          # Data export + erase
+    swarm/                         # Complaint swarm
+    portal-assist/                 # Portal automation
+    resolver/                      # Company resolver
+    case-lookup/                   # eCourts CNR lookup
+    capabilities/                  # GET /api/capabilities
+  worker/src/
+    inngest.ts                     # All registered Inngest functions
+    deps.ts                        # Singleton dependencies
+    functions/
+      intake.ts                    # Run intake (extract facts)
+      evidence.ts                  # Parse evidence (vision-LLM or ASR)
+      route-plan.ts                # Generate route plan with citations
+      action-execute.ts            # Execute actions via ActionExecutor
+      followup-schedule.ts         # Schedule follow-up reminders
+      followup-handler.ts          # Handle due follow-ups
+      reply-classify.ts            # Classify inbound replies
+      resolution-verify.ts         # Verify resolution
+      swarm.ts                     # Entity resolution + clustering
+      portal-assist.ts             # Portal automation flow
+      case-lookup.ts               # eCourts CNR lookup
+    lib/
+      action-executor.ts           # The ONLY place side-effects happen
+  web/src/
+    app/page.tsx                   # Mission board (list + create)
+    app/missions/[id]/page.tsx     # Mission room (full detail)
+    app/p/[slug]/page.tsx          # Public case page (private link)
+    app/ledger/page.tsx            # Anonymized ledger dashboard
+    lib/api.ts                     # Typed API client
+```
+
+---
+
+## 9. Troubleshooting
+
+### "DATABASE_URL is not set"
+Make sure `infra/.env` exists and has `DATABASE_URL` set. The API and worker read from this file.
+
+### "Qdrant connection failed"
+Check `QDRANT_URL` and `QDRANT_API_KEY`. For local Qdrant: `docker run -p 6333:6333 qdrant/qdrant`.
+
+### "OpenAI API key not set" (Whisper)
+Set `OPENAI_API_KEY` for voice note transcription. Without it, audio evidence will be skipped.
+
+### "Resend API key not set" (Email)
+Set `RESEND_API_KEY` and `RESEND_FROM` for email sending. Without it, email actions will fail with `proof_state: failed`.
+
+### "Inngest function not found"
+Make sure the Inngest dev server is running (`pnpm infra:up`) and the worker is running (`pnpm dev`). Check http://localhost:8288 for the Inngest dashboard.
+
+### Tests failing
+```bash
+pnpm typecheck  # Check for type errors first
+pnpm test       # Run tests
+```
+
+### Docker build failing
+```bash
+# Rebuild from scratch
+docker compose -f infra/docker-compose.yml down -v
+docker compose -f infra/docker-compose.yml build --no-cache
+docker compose -f infra/docker-compose.yml up -d
+```
+
+---
+
+## 10. Architecture Invariants (from CLAUDE.md)
+
+These are non-negotiable. Violating them breaks the system's trustworthiness:
+
+1. **Hexagonal** — `packages/domain` has ZERO framework imports
+2. **Everything external is an Action** — side-effects only happen in `action-executor.ts`
+3. **No fake submissions** — `proof_state` must reflect reality
+4. **No legal claim without cited source** — literal substring check
+5. **No public output without redaction + consent** — Presidio redaction before share
+6. **Contracts-first** — all Zod schemas in `packages/contracts`
+7. **Idempotent writes** — every write endpoint accepts `Idempotency-Key`
+8. **Every schema change is a Drizzle migration** — no ad-hoc DB edits

infra/.env.example

@@ -100,3 +100,8 @@ BROWSERBASE_PROJECT_ID=
 # --- Surepass (eCourts CNR lookup) ---
 # Paid third-party API for Indian court case data
 SUREPASS_API_KEY=
+
+# --- Portal automation (optional) ---
+# Set to "1" to enable Playwright-based portal automation
+# Without this, portal assist degrades to manual packet submission
+ENABLE_PORTAL_AUTOMATION=