Upload folder using huggingface_hub

NITDAA_ARCHITECTURE_DESIGN.md

@@ -0,0 +1,21 @@
+# Nitdaa Sibling Project - Architecture Design
+
+## 1. System Overview
+Nitdaa is the mobile-friendly MVP sibling to the full HealthExpert enterprise platform. It removes administrative overhead, heavy RBAC UI components, and complex multi-pane layouts in favor of a lean, mobile-first edge interface.
+
+## 2. Core Differences from Base HealthExpert
+- **UI/UX**: Transitioned from a 3-pane desktop layout to a single-pane vertical mobile layout.
+- **Admin Constraints**: Admin controls (Kill Switch, DB Purge) are entirely hidden in the HTML to prevent public tampering.
+- **Session Audit**: Uses `nitdaa_sessions.json` and `nitdaa_summary.json` for granular system telemetry rather than database-backed logs.
+
+## 3. Data Flow
+1. **Startup**: Flask initializes. `sync_he_data()` runs asynchronously to pull the latest dataset from `Sam-max1/he_data` using `huggingface_hub`.
+2. **Ingestion**: If new files are detected, `process_document_pipeline` splits them into 512-token chunks, vectorizes them into ChromaDB, and extracts graph entities locally using `spaCy`.
+3. **Retrieval**: When a query hits `/api/query`, it triggers Tri-modal Hybrid Retrieval (Vector + BM25 + Graph). 
+4. **Synthesis**: A Cross-Encoder reranks the chunks, and the context is fed into a reasoning-distilled tiny model (e.g., `Jackrong/Qwen3.5-2B-Claude-4.6-Opus-Reasoning-Distilled-GGUF`) running on `llama-cpp-python`.
+
+## 4. Resource Allocation
+Nitdaa is tuned for the HuggingFace Spaces free tier:
+- **vCPU constraints**: Threads are explicitly clamped to prevent CPU thrashing.
+- **Vector Limit**: A hard limit of 10,000 chunks is enforced globally to prevent `OutOfMemory` crashes.
+- **Session Quotas**: Max 5 file uploads per session to prevent abuse.

NITDAA_HEALTHEXPERT_USER_GUIDE.md

@@ -0,0 +1,31 @@
+# Nitdaa HealthExpert Mobile MVP - User Guide
+
+## Overview
+Nitdaa HealthExpert is a mobile-optimized, single-pane web interface designed to provide users with direct access to the HealthExpert AI knowledge base. Deployed on HuggingFace Spaces, Nitdaa brings the power of Agentic AI, Hybrid RAG, and Edge Compute into a sandboxed, responsive environment.
+
+## 1. Accessing the Application
+- Navigate to the HuggingFace Spaces URL provided by your administrator (e.g., `https://huggingface.co/spaces/Sam-max1/nitdaa`).
+- The application will load automatically. Note that the first boot may take 3-5 minutes as the system pulls the latest knowledge datasets securely from the cloud.
+- Once the **Status Pill** in the top-right corner turns **Green** (Ready), the system is online.
+
+## 2. Using the Dashboard
+The dashboard is designed for mobile efficiency.
+- **Top Header**: Displays the application branding ("Healthcare Policy Expert") and current status.
+- **Scrollable Output Window**: The large central area displays the conversation history. It automatically scrolls down as new answers are generated.
+- **Input Area**: Located at the bottom of the screen.
+
+## 3. Asking Questions
+You can query the system in two ways:
+1. **Quick Questions Dropdown**: Tap the dropdown menu above the text input to select a preset question. Selecting a question will automatically submit it to the AI.
+2. **Manual Input**: Type your query directly into the text box and press the **Send** button.
+
+## 4. Understanding Output
+When a question is submitted:
+- Your question will instantly appear in the output window.
+- The system will display a **"Generating answer.. (1.2s)"** timer.
+- Once context is retrieved from the Vector and Graph databases, the LLM will stream the Markdown-formatted answer back to the screen.
+- Responses remain preserved in the scrollable window for the duration of your session.
+
+## 5. Security & Session Handling
+- **Multi-tenant Isolation**: Your conversation is strictly isolated from other users using an invisible `X-Session-Token`.
+- **Session Expiration**: If the page is reloaded, or if you are inactive for an extended period, your session token resets, and previous context is cleared to ensure privacy.

NITDAA_HEALTHEXPERT_USER_GUIDE.pdf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:24283966d70a60ceef384e6a772e1e52106a7ce15beda33aba30e7480af3088b
+size 186919

NITDAA_HF_DEPLOYMENT_GUIDE.md

@@ -0,0 +1,32 @@
+# Nitdaa Sibling Project - HF Deployment Guide
+
+## Overview
+This document outlines the specific steps and constraints required to deploy the Nitdaa mobile sibling to HuggingFace Spaces free tier.
+
+## 1. Prerequisites
+- A HuggingFace account.
+- A HuggingFace Space configured with the **Docker** SDK.
+- Hardware allocation: Free Tier (2 vCPU, 16GB Disk, 12GB RAM).
+
+## 2. HuggingFace Secrets Configuration
+To enable dynamic Dataset Synchronization, you must configure the following variable in your Space's Settings under **Variables and Secrets**:
+- `HF_TOKEN`: A HuggingFace access token with "Read" permissions to the `he_data` dataset.
+- `ADMIN_MODE`: Set to `0` to completely lock down the UI for public usage.
+
+## 3. Deployment Steps
+1. Clone the `nitdaa` folder locally.
+2. Initialize a git repository pointing to your HF Space remote.
+3. Use the HuggingFace CLI to upload large directories without timing out:
+   ```bash
+   huggingface-cli upload <your_username>/nitdaa . . --repo-type space
+   ```
+4. HuggingFace will automatically detect the `Dockerfile` in the root and begin building.
+
+## 4. Cold Boot Times
+Due to the CPU-bound nature of the free tier:
+- **Docker Build**: ~5-8 minutes (first boot only).
+- **Dataset Sync**: Variable depending on the size of `he_data`.
+- **Model Load**: The `Qwen3.5-2B` GGUF model and PyTorch embeddings take ~1-2 minutes to spin up into memory on boot.
+
+## 5. Security Safeguards
+Nitdaa implements headless API blocking. Any direct programmatic access to the REST endpoints (without proper CORS headers or session tokens) will automatically return `403 Forbidden` to prevent bot-net abuse on the public URL.

app.py

@@ -80,6 +80,57 @@ app.secret_key = config.SECRET_KEY
 app.config["MAX_CONTENT_LENGTH"] = config.MAX_CONTENT_LENGTH
 os.makedirs(config.UPLOAD_FOLDER, exist_ok=True)
 
+from flask_limiter import Limiter
+from flask_limiter.util import get_remote_address
+from werkzeug.utils import escape
+from huggingface_hub import HfApi, hf_hub_download
+
+# Configure Rate Limiter (Memory Storage for MVP)
+limiter = Limiter(
+    get_remote_address,
+    app=app,
+    default_limits=["200 per day", "50 per hour"],
+    storage_uri="memory://"
+)
+
+@app.after_request
+def add_security_headers(response):
+    response.headers['X-Content-Type-Options'] = 'nosniff'
+    response.headers['X-Frame-Options'] = 'SAMEORIGIN'
+    response.headers['Strict-Transport-Security'] = 'max-age=31536000; includeSubDomains'
+    response.headers['Content-Security-Policy'] = "default-src 'self' 'unsafe-inline' 'unsafe-eval' https://cdn.jsdelivr.net;"
+    return response
+
+# Global background API for dataset upload
+_hf_api = None
+def get_hf_api():
+    global _hf_api
+    if _hf_api is None:
+        import keyring
+        token = os.environ.get("HF_TOKEN")
+        if not token:
+            try:
+                token = keyring.get_password('huggingface', 'hf_token') or keyring.get_password('huggingface', 'my_user')
+            except: pass
+        if token:
+            _hf_api = HfApi(token=token)
+    return _hf_api
+
+def async_sync_log(local_path: str, repo_path: str):
+    def _upload():
+        api = get_hf_api()
+        if api:
+            try:
+                api.upload_file(
+                    path_or_fileobj=local_path,
+                    path_in_repo=repo_path,
+                    repo_id="Sam-max1/mat_data",
+                    repo_type="dataset"
+                )
+            except Exception as e:
+                log.warning(f"Failed to push {repo_path} to mat_data: {e}")
+    threading.Thread(target=_upload, daemon=True).start()
+
 # In-memory job tracker for async ingestion
 _jobs: dict[str, dict] = {}
 _active_graph_tasks = 0
@@ -154,6 +205,8 @@ def log_session(event_type: str, token: str, ip: str):
         entry = {"timestamp": ts, "event": event_type, "session_token": token, "ip_address": ip}
         with open(session_file, "a") as f:
             f.write(json.dumps(entry) + "\n")
+        
+        async_sync_log(str(session_file), "nitdaa_sessions.json")
     except Exception as e:
         log.error(f"Failed to log session: {e}")
 
@@ -181,6 +234,8 @@ def log_query_summary(token: str, ip: str, query: str, chunks_retrieved: int, ge
         }
         with open(summary_file, "a") as f:
             f.write(json.dumps(entry) + "\n")
+            
+        async_sync_log(str(summary_file), "nitdaa_summary.json")
     except Exception as e:
         log.error(f"Failed to log query summary: {e}")
 
@@ -642,58 +697,108 @@ def process_document_pipeline(path: str, orig_name: str, tier: str, token: str,
 
 def sync_he_data():
     try:
-        import os
+        import os, shutil
         from huggingface_hub import snapshot_download
         import keyring
         
         token = os.environ.get("HF_TOKEN")
         if not token:
             try:
-                token = keyring.get_password('huggingface', 'my_user') or keyring.get_password('huggingface', 'hf_token')
+                token = keyring.get_password('huggingface', 'hf_token') or keyring.get_password('huggingface', 'my_user')
             except Exception:
                 pass
+                
+        # --- 2-Way Log Sync on Startup ---
+        log_dir = Path(__file__).parent / "app" / "logs"
+        log_dir.mkdir(parents=True, exist_ok=True)
+        try:
+            if token:
+                for log_file in ["nitdaa_sessions.json", "nitdaa_summary.json"]:
+                    local_p = log_dir / log_file
+                    try:
+                        dl_path = hf_hub_download(repo_id="Sam-max1/mat_data", filename=log_file, repo_type="dataset", token=token)
+                        if os.path.exists(dl_path):
+                            # Append any existing local logs not in the remote file to the remote file, then replace local
+                            remote_lines = set(open(dl_path).readlines())
+                            if local_p.exists():
+                                for line in open(local_p).readlines():
+                                    if line not in remote_lines:
+                                        remote_lines.add(line)
+                            with open(local_p, "w") as f:
+                                for line in sorted(list(remote_lines)):
+                                    f.write(line)
+                            log.info(f"Successfully merged {log_file} from mat_data.")
+                    except Exception as e:
+                        log.warning(f"Could not download {log_file} from mat_data (it may not exist yet): {e}")
+        except Exception as e:
+            log.warning(f"Log sync failed: {e}")
+        # ---------------------------------
         
         kbdocs_dir = Path(__file__).parent / "kbdocs"
         kbdocs_dir.mkdir(exist_ok=True)
         
-        log.info("Syncing fresh files from Sam-max1/he_data to local kbdocs...")
+        tmp_sync_dir = Path("/tmp/he_data_sync")
+        if tmp_sync_dir.exists():
+            shutil.rmtree(tmp_sync_dir)
+        tmp_sync_dir.mkdir(exist_ok=True)
+        
+        log.info("Syncing fresh files from Sam-max1/he-data to local /tmp...")
         snapshot_download(
-            repo_id="Sam-max1/he_data",
+            repo_id="Sam-max1/he-data",
             repo_type="dataset",
-            local_dir=str(kbdocs_dir),
+            local_dir=str(tmp_sync_dir),
             token=token,
             ignore_patterns=[".git*"]
         )
-        log.info("Successfully synced he_data repository.")
         
-        from pipeline import vector_store
+        from pipeline import vector_store, graph_store
         
-        existing_docs = vector_store.list_documents("admin")
-        existing_names = {d["source"] for d in existing_docs}
+        # Determine if there are differences between kbdocs and the downloaded he-data
+        local_files = {f.name: f.stat().st_size for f in kbdocs_dir.glob("*.*") if f.is_file()}
+        remote_files = {f.name: f.stat().st_size for f in tmp_sync_dir.glob("*.*") if f.is_file()}
         
-        ingested_count = 0
-        for file_path in kbdocs_dir.glob("*.*"):
-            if file_path.is_file() and file_path.name not in existing_names:
-                log.info(f"Auto-ingesting new file: {file_path.name}")
-                process_document_pipeline(str(file_path), file_path.name, "foundation", "admin", delete_after=False)
-                ingested_count += 1
-                
-        # show number of chunks and nodes
-        log.info("=== Full Data Ingestion Complete ===")
-        log.info(f"Vector DB Chunks: {vector_store.count()}")
+        is_different = False
+        if set(local_files.keys()) != set(remote_files.keys()):
+            is_different = True
+        else:
+            for k in local_files:
+                if local_files[k] != remote_files[k]:
+                    is_different = True
+                    break
         
-        from pipeline import graph_store
+        if is_different:
+            log.info("Detected changes in Sam-max1/he-data! Purging databases and re-syncing kbdocs.")
+            # Clear DBs
+            vector_store.purge()
+            if graph_store.is_available():
+                graph_store.purge()
+            
+            # Wipe local kbdocs and replace
+            shutil.rmtree(kbdocs_dir)
+            shutil.copytree(tmp_sync_dir, kbdocs_dir)
+            
+            ingested_count = 0
+            for file_path in kbdocs_dir.glob("*.*"):
+                if file_path.is_file():
+                    log.info(f"Auto-ingesting file: {file_path.name}")
+                    process_document_pipeline(str(file_path), file_path.name, "foundation", "admin", delete_after=False)
+                    ingested_count += 1
+            log.info("=== Full Data Re-Ingestion Complete ===")
+        else:
+            log.info("kbdocs is completely up to date with he-data. No ingestion needed.")
+
+        log.info(f"Vector DB Chunks: {vector_store.count()}")
         if graph_store.is_available():
             stats = graph_store.get_stats()
             log.info(f"Kuzu DB Nodes: {stats.get('nodes', 0)}, Edges: {stats.get('edges', 0)}")
             
     except Exception as e:
-        log.error("Failed to sync he_data: %s", e)
+        log.error("Failed to sync he-data: %s", e)
 
 threading.Thread(target=sync_he_data, daemon=True).start()
 
-
 @app.route("/api/ingest", methods=["POST"])
+@limiter.limit("50 per hour")
 def ingest():
     """Upload and asynchronously ingest one or more documents."""
     log.info("Ingest request received. Files in request: %s",
@@ -818,10 +923,11 @@ def delete_document(source_name: str):
 # ── Query ─────────────────────────────────────────────────────────────────────
 
 @app.route("/api/query", methods=["POST"])
+@limiter.limit("120 per minute")
 def query():
     """RAG query — returns a streaming SSE response."""
     data = request.get_json()
-    q    = (data or {}).get("query", "").strip()
+    q    = escape((data or {}).get("query", "").strip())
     top_k = (data or {}).get("top_k")
     max_tokens = (data or {}).get("max_tokens")
     use_vector = (data or {}).get("use_vector", True)

requirements.txt

@@ -59,3 +59,4 @@ psutil>=5.9.0
 rank_bm25
 spacy>=3.7.0
 en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
+Flask-Limiter>=3.0.0

requirements_hf.txt

@@ -58,3 +58,4 @@ psutil>=5.9.0
 rank_bm25
 spacy>=3.7.0
 en-core-web-sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.8.0/en_core_web_sm-3.8.0-py3-none-any.whl
+Flask-Limiter>=3.0.0

templates/index.html

@@ -160,10 +160,47 @@
       white-space: nowrap;
       border-width: 0;
     }
+    
+    /* CAPTCHA Modal Styling */
+    #captcha-modal {
+      position: fixed;
+      top: 0; left: 0; right: 0; bottom: 0;
+      background: var(--bg-color);
+      z-index: 9999;
+      display: flex;
+      flex-direction: column;
+      align-items: center;
+      justify-content: center;
+      padding: 1rem;
+      text-align: center;
+    }
+    #captcha-modal .card {
+      background: var(--white);
+      padding: 2rem;
+      border-radius: 8px;
+      box-shadow: 0 4px 12px rgba(0,0,0,0.15);
+      max-width: 320px;
+      width: 100%;
+    }
+    #captcha-modal h2 { margin-top: 0; color: var(--primary-color); }
+    #captcha-modal .challenge { font-size: 1.5rem; font-weight: bold; margin: 1rem 0; }
+    #captcha-error { color: #d32f2f; margin-top: 0.5rem; display: none; font-weight: bold; }
   </style>
 </head>
 <body>
 
+  <!-- Security CAPTCHA Overlay -->
+  <div id="captcha-modal">
+    <div class="card">
+      <h2>Security Check</h2>
+      <p>Please solve this math problem to continue:</p>
+      <div class="challenge" id="captcha-challenge"></div>
+      <input type="number" id="captcha-input" placeholder="Your answer" aria-label="Captcha answer">
+      <div id="captcha-error">Incorrect, try again!</div>
+      <button id="captcha-btn" style="margin-top: 1rem;">Verify</button>
+    </div>
+  </div>
+
   <header class="banner" role="banner">
     <div>
       <h1>Healthcare Policy Expert</h1>
@@ -224,9 +261,55 @@
     };
     let notReadyStartTime = null;
     const NOT_READY_TIMEOUT_MS = 300 * 1000;
-    const SESSION_TOKEN = crypto.randomUUID();
+    let sessionToken = localStorage.getItem('he_session_token');
+    if (!sessionToken) {
+      sessionToken = 'sess_' + Math.random().toString(36).substring(2, 11);
+      localStorage.setItem('he_session_token', sessionToken);
+    }
+
+    const SESSION_TOKEN = sessionToken;
 
     document.addEventListener("DOMContentLoaded", () => {
+      // --- CAPTCHA Logic ---
+      const captchaModal = document.getElementById('captcha-modal');
+      const captchaChallenge = document.getElementById('captcha-challenge');
+      const captchaInput = document.getElementById('captcha-input');
+      const captchaBtn = document.getElementById('captcha-btn');
+      const captchaError = document.getElementById('captcha-error');
+      let captchaExpected = 0;
+
+      function generateCaptcha() {
+        const num1 = Math.floor(Math.random() * 10) + 1;
+        const num2 = Math.floor(Math.random() * 10) + 1;
+        captchaExpected = num1 + num2;
+        captchaChallenge.textContent = `${num1} + ${num2} = ?`;
+        captchaInput.value = '';
+      }
+
+      function verifyCaptcha() {
+        const val = parseInt(captchaInput.value, 10);
+        if (val === captchaExpected) {
+          captchaModal.style.display = 'none';
+          sessionStorage.setItem('captcha_solved', 'true');
+          pollStatus();
+        } else {
+          captchaError.style.display = 'block';
+          generateCaptcha();
+        }
+      }
+
+      captchaBtn.addEventListener('click', verifyCaptcha);
+      captchaInput.addEventListener('keypress', (e) => {
+        if (e.key === 'Enter') verifyCaptcha();
+      });
+
+      if (sessionStorage.getItem('captcha_solved') === 'true') {
+        captchaModal.style.display = 'none';
+        pollStatus();
+      } else {
+        generateCaptcha();
+      }
+      // ---------------------
       const quickSelect = document.getElementById('quick-questions');
       const questionInput = document.getElementById('question-input');
       const sendBtn = document.getElementById('send-btn');
@@ -287,7 +370,6 @@
         }
         setTimeout(pollStatus, 5000);
       }
-      pollStatus();
 
       quickSelect.addEventListener('change', () => {
         if (quickSelect.value) {