NITDAA_ARCHITECTURE_DESIGN.md

Nitdaa Sibling Project - Architecture Design

Version 1.0 Public Release

1. System Overview

Nitdaa is the mobile-friendly MVP sibling to the full HealthExpert enterprise platform. It removes administrative overhead, heavy RBAC UI components, and complex multi-pane layouts in favor of a lean, mobile-first edge interface.

2. Core Differences from Base HealthExpert

• UI/UX: Transitioned from a 3-pane desktop layout to a single-pane vertical mobile layout.
• Admin Constraints: Admin controls (Kill Switch, DB Purge) are entirely hidden in the HTML to prevent public tampering.
• Session Audit: Uses nitdaa_sessions.json and nitdaa_summary.json for granular system telemetry rather than database-backed logs. Features an integrated UI feedback loop that captures user sentiment (thumbs up/down) directly into the telemetry file.
• Resumable Event Streaming: The architecture uses an asynchronous Job ID system (/api/query/start and /api/query/stream) to survive mobile background disconnections.
• Concurrent RAG Isolation: Vector and Graph retrieval pipelines operate inside a Thread Pool with thread-local database connections, allowing multi-user concurrency without global UI locking.

3. Data Flow

1. Startup & Sync: Flask initializes. sync_he_data() runs asynchronously to pull the latest dataset from Sam-max1/he-data. If differences are found, it gracefully purges vector/graph databases and rebuilds. Concurrently, it performs a 2-way remote synchronization of logs with Sam-max1/mat_data.
2. Security & Connection: Client connections face an immediate Math CAPTCHA. All API calls pass through Flask-Limiter and Strict HTTP Content Security Policies.
3. Ingestion: If new files are detected, process_document_pipeline splits them into 512-token chunks, vectorizes them into ChromaDB, and extracts graph entities locally using spaCy.
4. Retrieval (Concurrent): When a query hits /api/query/start, an asynchronous thread pool handles Tri-modal Hybrid Retrieval (Vector + BM25 + Graph) allowing multiple users to fetch context simultaneously.
5. Synthesis & Feedback: A Cross-Encoder reranks the chunks, and the context is fed into the local LLM generation queue. The frontend streams events using Server-Sent Events with offset recovery. Upon completion, users optionally submit ratings securely stored in nitdaa_summary.json.

4. Resource Allocation

Nitdaa is tuned for the HuggingFace Spaces free tier:

• vCPU constraints: Threads are explicitly clamped to prevent CPU thrashing.
• Vector Limit: A hard limit of 10,000 chunks is enforced globally to prevent OutOfMemory crashes.
• Session Quotas: Max 5 file uploads per session to prevent abuse.

5. Project Intelligence & Deployment

• HuggingFace Space: Sam-max1/nitdaa
• HuggingFace ID: sam-max1
• Authentication: HF_TOKEN is provided via the system keyring for secure access to HuggingFace resources and pushing updates.
• Deployment Protocol: Code base updates are pushed directly to the HuggingFace Space repository. Note: Core functionality should not be altered, deleted, or impaired unless explicitly requested.
• Minimalist Deployment: Nitdaa must always contain only the bare minimum files required to run the application. Extraneous resources, local databases, or unneeded assets must be excluded from tracking to ensure only the bare minimum is pushed to HuggingFace.