---
title: Lumea Health Platform
emoji: π₯
colorFrom: blue
colorTo: green
sdk: docker
app_port: 7860
pinned: false
license: mit
short_description: AI preventive health companion with OCR and digital twin
---
# Lumea Health Platform
A full-stack health companion platform for preventive health management. Upload medical reports, extract health metrics via OCR, track trends, and receive AI-powered health recommendations.
> **Disclaimer**: This platform is a support tool for personal health tracking. It does not provide medical advice, diagnosis, or treatment. Always consult a licensed healthcare professional for medical decisions.
---
## Table of Contents
- [Overview](#overview)
- [Tech Stack](#tech-stack)
- [Architecture](#architecture)
- [Features](#features)
- [Memory & Knowledge Graph: Intelligent Health Context](#memory--knowledge-graph-intelligent-health-context)
- [Medicines: Find Cheap Alternatives](#medicines-find-cheap-alternatives)
- [Voice Agent: AI Health Assistant](#voice-agent-ai-health-assistant)
- [Physics Twin: Health Telemetry & Organ Scoring](#physics-twin-health-telemetry--organ-scoring)
- [Getting Started](#getting-started)
- [Environment Variables](#environment-variables)
- [Database](#database)
- [API Reference](#api-reference)
- [WebSocket Events](#websocket-events)
- [Testing](#testing)
- [Deployment](#deployment)
- [Contributing](#contributing)
- [License](#license)
---
## Overview
Lumea is a unified medical companion platform that enables users to:
- **Upload medical reports** (PDF, images) and automatically extract health metrics via OCR
- **Track health profiles** with comprehensive intake forms (conditions, medications, allergies, family history)
- **Monitor health trends** with a computed Health Index and interactive charts
- **Receive AI recommendations** based on extracted lab values and health patterns powered by OpenRouter (Claude Sonnet 4.5)
- **Compare reports over time** with AI-powered summaries and trend analysis
- **Chat with an AI assistant** grounded in your personal health data
- **Memory & Knowledge Graph**: Persistent health memory with Mem0 and interconnected insights via Neo4j/Graphiti for temporal reasoning and contradiction detection
- **Voice Agent**: Speak naturally to an AI health assistant for hands-free access to personalized health insights
- **Physics Twin**: Interactive 2D organ simulation with real-time telemetry, condition detection, and evidence-based health recommendations powered by your actual medical data
---
## Tech Stack
| Layer | Technology |
|-------|------------|
| **Frontend** | React 18, TypeScript, Vite, Framer Motion, Recharts, React Router, i18next |
| **Backend** | FastAPI (Python 3.13+), SQLAlchemy 2.0 (async), Pydantic v2 |
| **Database** | PostgreSQL (Neon or local), Alembic migrations |
| **Graph Database** | Neo4j (knowledge graph for health relationships) |
| **Vector Stores** | ChromaDB (RAG), Mem0 (persistent health memory) |
| **OCR/Extraction** | PaddleOCR, pdfplumber, PyMuPDF |
| **AI/LLM** | OpenRouter (Claude Sonnet 4.5 primary), Google Gemini API (fallback), Groq (fast inference), Ollama (local LLM/embeddings) |
| **Memory Layer** | Mem0 (user-scoped persistent memory with automatic fact extraction) |
| **Knowledge Graph** | Graphiti-core (temporal reasoning, entity relationships, contradiction detection) |
| **Embeddings** | Ollama (nomic-embed-text local), OpenAI-compatible APIs |
| **Voice Agent** | Web Speech API (STT), ElevenLabs TTS, Google Gemini |
| **Realtime** | WebSocket (FastAPI native), Server-Sent Events (telemetry streaming) |
| **Auth** | JWT (python-jose), bcrypt, rate limiting middleware |
---
## Architecture
```mermaid
flowchart TB
subgraph Client["Frontend (React)"]
UI[React UI]
WS[WebSocket Client]
end
subgraph API["Backend (FastAPI)"]
Auth[Auth Routes]
Dashboard[Dashboard Routes]
Reports[Reports Routes]
Profile[Profile Routes]
Recommendations[Recommendations Routes]
AISummary[AI Summary Routes]
Assistant[Assistant Routes]
WSServer[WebSocket Server]
end
subgraph Services["Backend Services"]
OCR[PDF/OCR Extractor]
Classifier[Document Classifier]
MetricExtractor[Metric Extractor]
MetricsService[Metrics Service]
RecommendationEngine[Recommendation Engine]
AISummaryService[AI Summary Service]
RAG[RAG Service]
LLM[LLM Service]
end
subgraph External["External Services"]
GrokAPI[Grok API]
Ollama[Ollama / MedGemma]
Gemini[Gemini API]
end
subgraph Storage["Data Layer"]
DB[(PostgreSQL / Neon)]
ChromaDB[(ChromaDB Vector Store)]
FileStorage[File Storage]
end
UI --> Auth
UI --> Dashboard
UI --> Reports
UI --> Profile
UI --> Recommendations
UI --> AISummary
UI --> Assistant
WS <--> WSServer
Reports --> OCR
OCR --> Classifier
Classifier --> MetricExtractor
MetricExtractor --> MetricsService
MetricsService --> RecommendationEngine
AISummary --> AISummaryService
AISummaryService --> GrokAPI
Assistant --> RAG
RAG --> ChromaDB
RAG --> LLM
LLM --> Ollama
LLM --> Gemini
Auth --> DB
Dashboard --> DB
Reports --> DB
Reports --> FileStorage
Profile --> DB
Recommendations --> DB
AISummary --> DB
```
### Data Flow: Report Upload to Health Index
```mermaid
sequenceDiagram
participant User
participant Frontend
participant Backend
participant OCR
participant Classifier
participant Extractor
participant DB
participant WebSocket
User->>Frontend: Upload PDF/Image
Frontend->>Backend: POST /api/reports/upload
Backend->>DB: Create report (status: uploaded)
Backend-->>Frontend: {id, status: uploaded}
Backend->>OCR: Extract text (background)
OCR-->>Backend: Raw text
Backend->>Classifier: Classify document
Classifier-->>Backend: Category + doc_type
Backend->>Extractor: Extract metrics
Extractor-->>Backend: Observations[]
Backend->>DB: Save observations + update report
Backend->>DB: Recompute health index
Backend->>WebSocket: emit report_parsed
WebSocket-->>Frontend: report_parsed event
Frontend->>Frontend: Refresh UI
```
---
## Features
### Physics Twin - Digital Health Telemetry & Organ Scoring
- **6-Organ System**: Deterministic health scoring for kidney, heart, liver, lungs, brain, blood
- **Real Data Integration**: Automatic computation from your actual extracted medical reports (not simulated)
- **Interactive 3D Body**: Click to explore organ scores with animated visualizations
- **Real-Time Telemetry**: Live vital signs stream (heart rate, blood pressure, SpOβ, glucose, stress, sleep)
- **Condition Detection**: 10 rule-based health conditions auto-detected with severity mapping
- **Body Impact Overlay**: 2D SVG visualization showing organs affected by detected conditions
- **Smart Recommendations**: Evidence-based health actions with YouTube educational links per condition
- **AI Video Recommendations**: OpenRouter-powered personalized health video suggestions for each organ based on abnormal metrics, with per-organ caching
- **Explainability Cards**: Side-by-side "Improve Your Health" and "Why This Score?" panels showing video recommendations and scoring breakdowns
- **Time-Series History**: View per-report organ scores and historical trends
- **Lifestyle Integration**: Incorporates self-reported profile data (sleep hours, stress level)
### Document Upload & OCR
- Supported formats: PDF, PNG, JPG, JPEG, TIFF
- Max file size: 50MB
- Automatic text extraction (text-first, OCR fallback)
- Document classification: Lab, Dental, MRI, X-ray, Prescription, Sleep
### Health Profile & Reminders
- Multi-step wizard with 6 steps (basics, measurements, conditions, medications, lifestyle, etc.)
- Tracks conditions, symptoms, medications, supplements, allergies
- Family medical history and genetic test results
- **Once completed, users are never re-asked** β profile status persists in DB
- "Profile Complete" indicator with quick-edit access via Settings page
- **Real-time SMS reminders** via Twilio (or mock mode for testing)
- Background scheduler processes due reminders every 60 seconds
- Default reminders auto-generated: medication, appointment, checkup, hydration
### Health Index & Trends
- Computed health index (0-100) based on lab values
- Factor contributions breakdown (glucose, lipids, vitamins, etc.)
- Time-series trends (1D, 1W, 1M views)
- Abnormal value flagging with reference ranges
### AI Recommendations
- Rule-based engine analyzing lab values vs reference ranges
- Severity levels: INFO, WARNING, URGENT
- Categories: lifestyle, screening, follow-up, urgent
- Evidence-based with citations
### AI Report Summary
- Single report AI summary with key findings
- Multi-report comparison (2-6 reports, same type)
- Highlights: positive, needs attention, next steps
- Cached results with hash-based invalidation
### Health Assistant
- RAG-powered chat grounded in user's health data
- Citations from reports and observations
- WebSocket streaming for real-time responses
### Voice Agent (AI Health Assistant)
- **Natural voice conversations** with your health data
- **Speech-to-Text**: Browser Web Speech API for hands-free input
- **AI Processing**: Google Gemini with personalized health context
- **Text-to-Speech**: ElevenLabs studio-quality voice (with browser fallback)
- **Safety features**: Emergency detection, dosage inquiry protection
- **Real-time feedback**: Visual orb animations for listening/thinking/speaking
- **Comprehensive context**: Uses profile, conditions, medications, allergies, reports, RAG data
- **Accessibility-first**: Ideal for hands-free use, visual impairments, or quick queries
---
## Memory & Knowledge Graph: Intelligent Health Context
### Overview
The **Memory & Knowledge Graph** system provides persistent, intelligent health context that evolves with your medical journey. Built on **Mem0** (memory layer), **Graphiti** (knowledge graph engine), and **Neo4j** (graph database), this architecture enables the platform to:
- **Remember** health facts across sessions (conditions, metrics, lifestyle patterns, medication changes)
- **Connect** disparate health data points into meaningful relationships
- **Reason temporally** about health trends and contradictions
- **Generate insights** from interconnected health knowledge
### Architecture
```mermaid
flowchart TB
subgraph Input["Health Data Sources"]
Profile[Health Profile]
Reports[Medical Reports]
Questionnaire[User Questionnaires]
Assistant[AI Assistant Chats]
end
subgraph Memory["Mem0 Memory Layer"]
Extraction[Fact Extraction
via Groq LLM]
Storage[ChromaDB
Vector Store]
Retrieval[Semantic Search]
end
subgraph Graph["Graphiti Knowledge Graph"]
Entities[Entity Nodes
conditions, metrics, meds]
Relationships[Temporal Relationships
affects, caused_by, treats]
Episodes[Health Episodes
time-stamped events]
Insights[AI-Generated Insights
contradictions, patterns]
end
subgraph DB["Neo4j Graph Database"]
Nodes[Entity Nodes]
Edges[Relationship Edges]
Indexes[Vector Indexes
Ollama embeddings]
end
Input --> Extraction
Extraction --> Storage
Storage --> Retrieval
Input --> Entities
Entities --> Relationships
Relationships --> Episodes
Episodes --> Insights
Entities --> Nodes
Relationships --> Edges
Edges --> Indexes
Retrieval -.->|Enriches| Recommendations[AI Recommendations]
Insights -.->|Powers| Dashboard[Features Dashboard]
```
### Key Features
#### 1. **Persistent Memory (Mem0)**
- **Automatic Fact Extraction**: Converts natural language health data into structured facts
- "Sleep is about 5 hours per night" β Extracted memory fact
- "Has condition High Blood Pressure" β Indexed and searchable
- "BMI is 25.2 which falls in overweight range" β Computed and stored
- **User-Scoped Storage**: Each user's memories are isolated and secure
- **Vector-Based Retrieval**: Semantic search powered by local Ollama embeddings (nomic-embed-text)
- **Deduplication**: Prevents redundant facts with intelligent merging
#### 2. **Knowledge Graph (Graphiti + Neo4j)**
- **Entity Relationship Mapping**:
- Conditions linked with symptoms, medications, and risk factors
- Lab values connected to reference ranges and health implications
- Lifestyle factors (sleep, diet, exercise) linked to health outcomes
- **Temporal Reasoning**:
- Track health changes over time (BP trend: 140/90 in Jan β 130/85 in Feb)
- Episode creation for significant health events
- Valid/expired relationships based on temporal context
- **Contradiction Detection**:
- Identifies conflicting health data (e.g., "no allergies" vs. "allergic to dust")
- Highlights diagnostic uncertainties
- Flags medication interactions
- **Graph-Powered Insights**:
- **Temporal Patterns**: "BP decreased after starting medication X"
- **Key Relationships**: "High BMI correlates with elevated cholesterol"
- **Contradictions**: "Sleep reported as both 5hrs and 8hrs - needs clarification"
#### 3. **Interactive Visualization**
**Features Dashboard** (`/features`) provides:
- **Memory Dashboard**:
- View all extracted health facts
- Search memories semantically
- Add/delete facts manually
- Real-time sync status
- **Knowledge Graph Viewer**:
- Interactive node-edge visualization
- Filter by conditions, medications, metrics
- Zoom/pan for detailed exploration
- Click nodes/edges for metadata
- **AI Insights Panel**:
- Temporal insights (trend analysis)
- Relationship insights (correlation patterns)
- Contradiction insights (data conflicts)
- Each insight includes evidence and confidence score
### Technical Implementation
#### Mem0 Configuration
```python
# Memory layer uses:
# - Groq (llama-3.3-70b-versatile) for fact extraction
# - ChromaDB for vector storage
# - HuggingFace embeddings (local)
# - User-scoped collections
```
#### Graphiti Configuration
```python
# Knowledge graph uses:
# - OpenRouter (Claude Sonnet 4.5) for entity extraction
# - Ollama (nomic-embed-text) for embeddings
# - Neo4j for graph persistence
# - User-scoped graph isolation via group_ids
```
#### Data Flow
1. **Profile Sync**: Health profile questionnaire auto-syncs facts to Mem0 and Graphiti
2. **Report Upload**: PDF extraction triggers memory and graph updates
3. **Assistant Chat**: Conversational context stored as episodic memories
4. **Background Processing**: Fact extraction and graph building happen asynchronously
### API Endpoints
#### Memory API (`/api/memory`)
- `GET /api/memory/facts` - List all user memories
- `POST /api/memory/search` - Semantic search across memories
- `POST /api/memory/add` - Manually add a memory fact
- `DELETE /api/memory/{memory_id}` - Delete specific memory
#### Graph API (`/api/graph`)
- `GET /api/graph/relationships?limit=30` - Get graph nodes and edges
- `POST /api/graph/insights` - Request AI-generated insights (temporal/relationships/contradictions)
- `GET /api/graph/stats` - Graph statistics (node count, relationship count)
### Performance & Scalability
- **Rate Limiting**: Groq free tier handles ~30 requests/minute (automatic retry with exponential backoff)
- **Local Embeddings**: Ollama eliminates API costs for vector operations
- **Caching**: Frequently accessed graph patterns cached in-memory
- **Async Processing**: Profile/report sync uses background tasks to avoid blocking HTTP requests
- **Neo4j Indexes**: Full-text and vector indexes optimize graph queries
### Privacy & Security
- **User Isolation**: Mem0 and Graphiti enforce user-scoped data via `user_id` and `group_ids`
- **No Cross-User Leakage**: Client-side filtering ensures graph results never leak between users
- **JWT Authentication**: All memory/graph endpoints require valid JWT tokens
- **Local Processing**: Embeddings done locally (Ollama) - no external API calls for vectors
### Use Cases
1. **Medication Adherence**: Graph tracks "Started medication X on date Y" β Generate reminders
2. **Lifestyle Impact Analysis**: Correlate sleep patterns with BP readings over time
3. **Family History Mapping**: Link genetic conditions to risk factors
4. **Report Comparison**: Memory of previous lab values enables delta analysis
5. **Contraindication Detection**: Graph identifies medication-condition conflicts
---
## Medicines: Find Cheap Alternatives
Lumea includes a comprehensive medicine management system that helps users find affordable generic alternatives to prescribed medicines and locate nearby pharmacies, including government-sponsored Jan Aushadhi Kendras offering subsidized medications.
> **Medical Disclaimer**: This feature is a support tool for informational purposes only. It does NOT provide medical advice. Always consult your doctor or pharmacist before switching medicines or starting new treatments.
### Overview
The Medicines feature enables users to:
- **Search for medicines** by brand name or free-text queries
- **Upload prescriptions** and automatically extract medicines via OCR/AI
- **Find affordable alternatives** with ranked matching by clinical equivalence
- **Compare prices** including Jan Aushadhi (government-fixed price) options
- **Locate nearby pharmacies** including generic and Jan Aushadhi pharmacies
- **Track medications** with personal notes and dosage schedules
### User Flow
1. **Input Medicine Information**
- User enters brand name (e.g., "Aspirin 500mg") or uploads prescription image
- System extracts text via OCR or Grok AI parsing
2. **Normalize Medicine Data**
- `MedicineNormalizer` service extracts: salt, strength, form (tablet/capsule/liquid), release type (immediate/sustained)
- Queries `GenericCatalog` table to validate and standardize extraction
3. **Find Substitutes**
- `SubstituteFinder` queries database for alternatives with 4-tier ranked matching:
- **Tier 1** (1.0 score): Exact match on salt + strength + form + release type
- **Tier 2** (0.8 score): Same salt + strength + form
- **Tier 3** (0.6 score): Same salt + strength
- **Tier 4** (0.4 score): Same salt only
- Results sorted by price (Jan Aushadhi first for lowest cost)
4. **Locate Pharmacies** (Optional)
- User enters location (latitude/longitude) and search radius
- `PharmacyLocator` queries Google Places API for pharmacies
- Results include Jan Aushadhi Kendras (government pharmacies) and private pharmacies
- Results cached for 1 hour to reduce API calls
5. **Save & Track**
- User can save medicines to personal list with notes
- Each save creates `UserSavedMedicine` entry for quick reference
### Architecture Diagram
```mermaid
flowchart TB
subgraph Frontend["Frontend (React)"]
MedicinesPage["Medicines.tsx Page"]
SubstitutePanel["Substitute Finder Panel"]
PharmacyPanel["Pharmacy Locator Panel"]
end
subgraph API["Backend API (FastAPI)"]
NormalizeRoute["POST /normalize
POST /normalize/batch"]
SubstituteRoute["POST /substitutes
POST /substitutes/from-text"]
PharmacyRoute["GET /pharmacies/nearby
GET /pharmacies/{id}
POST /pharmacies/{id}/click"]
SavedRoute["POST /saved
GET /saved
DELETE /saved/{id}"]
end
subgraph Services["Backend Services"]
MedicineNormalizer["MedicineNormalizer
- normalize()
- normalize_batch()"]
SubstituteFinder["SubstituteFinder
- find_substitutes()
- save_medicine()"]
PharmacyLocator["PharmacyLocator
- search_nearby()
- get_place_details()
- 1hr cache"]
GrokService["GrokMedicineService
- get_alternatives_for_text()"]
end
subgraph External["External Services"]
GooglePlaces["Google Places API
(Pharmacy Search)"]
GrokAPI["Grok xAI API
(Medicine Parsing)"]
end
subgraph Storage["Data Layer"]
DB[(PostgreSQL)]
GenericCatalog["GenericCatalog Table
(600K+ medicines)"]
UserSavedMedicine["UserSavedMedicine Table"]
SubstituteQuery["SubstituteQuery Table
(Analytics)"]
PharmacyClick["PharmacyClick Table
(Analytics)"]
end
MedicinesPage --> SubstitutePanel
MedicinesPage --> PharmacyPanel
SubstitutePanel --> SubstituteRoute
PharmacyPanel --> PharmacyRoute
PharmacyPanel --> SavedRoute
SubstituteRoute --> GrokService
SubstituteRoute --> MedicineNormalizer
SubstituteRoute --> SubstituteFinder
PharmacyRoute --> PharmacyLocator
MedicineNormalizer --> DB
MedicineNormalizer --> GenericCatalog
SubstituteFinder --> DB
SubstituteFinder --> GenericCatalog
SubstituteFinder --> SubstituteQuery
PharmacyLocator --> GooglePlaces
PharmacyLocator --> PharmacyClick
GrokService --> GrokAPI
GrokAPI -->|Parsed results| SubstituteFinder
```
### API Endpoints
| Method | Endpoint | Description | Auth | Key Parameters |
|--------|----------|-------------|------|-----------------|
| POST | `/api/medicines/normalize` | Normalize single medicine text | Yes | `text: str` β Returns `NormalizedMedicine` |
| POST | `/api/medicines/normalize/batch` | Normalize multiple medicines (prescription lines) | Yes | `texts: List[str]` β Returns `List[NormalizedMedicine]` |
| POST | `/api/medicines/substitutes` | Find substitutes from structured data | Yes | `salt, strength, form, release_type` β Returns ranked substitutes |
| POST | `/api/medicines/substitutes/from-text` | Find substitutes from free-form text (AI-powered) | Yes | `text: str` β Grok AI parsing β Returns alternatives with prices |
| GET | `/api/medicines/pharmacies/nearby` | Search nearby pharmacies by location | Yes | `lat, lng, radius_m=1000, type=all, page_token` β Returns paginated pharmacies |
| GET | `/api/medicines/pharmacies/{place_id}` | Get pharmacy details | Yes | `place_id: str` β Returns address, phone, hours, rating |
| POST | `/api/medicines/pharmacies/{place_id}/click` | Log pharmacy interaction (analytics) | Yes | `action: directions\|call\|website` |
| POST | `/api/medicines/saved` | Save medicine to user list | Yes | `brand_name, salt, strength, form, notes` |
| GET | `/api/medicines/saved` | Get user's saved medicines | Yes | Returns list of `UserSavedMedicine` |
| DELETE | `/api/medicines/saved/{medicine_id}` | Delete saved medicine | Yes | `medicine_id: UUID` |
### Request/Response Examples
**POST /api/medicines/substitutes/from-text**
Request:
```json
{
"text": "Aspirin 500mg tablets"
}
```
Response:
```json
{
"original_text": "Aspirin 500mg tablets",
"normalized": {
"brand_name": "Aspirin",
"salt": "Acetylsalicylic Acid",
"strength": "500",
"form": "tablet",
"release_type": "immediate",
"confidence": 0.95
},
"substitutes": [
{
"rank": 1,
"product_name": "Jan Aushadhi Aspirin 500mg",
"salt": "Acetylsalicylic Acid",
"strength": "500",
"form": "tablet",
"mrp": "5.00",
"is_jan_aushadhi": true,
"match_score": 1.0,
"match_reason": "Exact match: Same salt, strength, form, and type"
},
{
"rank": 2,
"product_name": "Ecosprin 500mg",
"salt": "Acetylsalicylic Acid",
"strength": "500",
"form": "tablet",
"mrp": "8.50",
"is_jan_aushadhi": false,
"match_score": 1.0,
"match_reason": "Exact match: Same salt, strength, form, and type"
}
],
"disclaimer": "Always confirm with your doctor or pharmacist before switching medicines"
}
```
**GET /api/medicines/pharmacies/nearby**
Request:
```
GET /api/medicines/pharmacies/nearby?lat=40.7128&lng=-74.0060&radius_m=1000&type=all
```
Response:
```json
{
"pharmacies": [
{
"place_id": "ChIJN1blbgBQwokRzKgy6E_B_1Q",
"name": "Jan Aushadhi Kendra - Downtown",
"address": "123 Main St, New York, NY 10001",
"latitude": 40.7128,
"longitude": -74.0060,
"rating": 4.7,
"is_open": true,
"is_jan_aushadhi": true,
"phone": "+1-212-555-0123"
},
{
"place_id": "ChIJrc_p_1BQwokRzKgy6E_B_2Q",
"name": "Metro Pharmacy",
"address": "456 Broadway, New York, NY 10002",
"latitude": 40.7150,
"longitude": -74.0030,
"rating": 4.2,
"is_open": true,
"is_jan_aushadhi": false,
"phone": "+1-212-555-0456"
}
],
"next_page_token": null,
"total_results": 2
}
```
---
## Voice Agent: AI Health Assistant
Lumea includes a fully integrated **Voice Agent** that provides an ElevenLabs-style conversational health assistant experience. Users can speak naturally to the AI agent, which responds with personalized health guidance based on their profile, conditions, medications, allergies, and recent lab reports.
> **Medical Disclaimer**: The Voice Agent is an informational support tool. It does NOT provide medical diagnosis, dosage recommendations, or emergency care. Always consult licensed healthcare professionals for medical decisions.
### Overview
The Voice Agent enables users to interact with their health data through natural voice conversations powered by AI.
**Key Capabilities**:
- **Voice Interaction**: Speak naturally using browser-based speech recognition (Web Speech API)
- **Personalized AI Responses**: Powered by Google Gemini with access to your complete health profile
- **Natural Speech Output**: Voice responses via ElevenLabs TTS (with browser fallback)
- **Medical Safety Filters**: Detects emergency keywords, dosage inquiries, and provides appropriate safety responses
- **Real-time Feedback**: Visual orb animations reflect listening/thinking/speaking states
- **Transcript View**: Optional chat-mode panel displays conversation history
- **Context-Aware**: Accesses your health conditions, medications, allergies, recent reports, and RAG-indexed data
### How It Works
The Voice Agent uses a three-stage pipeline to deliver personalized health conversations:
```mermaid
graph LR
A[User Speaks] --> B[Browser STT
Web Speech API]
B --> C[LLM Processing
Google Gemini]
C --> D[Health Context
Profile + Reports + RAG]
D --> C
C --> E[Text-to-Speech
ElevenLabs API]
E --> F[Browser Fallback
Web Speech API]
E --> G[User Hears Response]
F --> G
```
#### Stage 1: Speech-to-Text (STT)
- **Technology**: Web Speech API (browser native)
- **Process**: Real-time audio capture and transcription
- **Benefits**: No external API costs, works offline for transcription
- **Language Support**: Multi-language support via browser capabilities
#### Stage 2: AI Processing & Context Retrieval
- **LLM Provider**: Google Gemini API (`gemini-flash-latest`)
- **Health Context Integration**:
- User demographics (age, gender, BMI)
- Active medical conditions
- Current medications (name, dose, frequency)
- Known allergies and their severity
- Recent lab reports and observations
- Lifestyle factors (sleep, exercise, smoking, alcohol)
- **RAG Enhancement**: Queries ChromaDB vector store for relevant historical health data
- **Safety Layer**: HIPAA-compliant system prompt with strict medical guidelines:
- Never diagnose conditions
- Never recommend medication dosages
- Always suggest professional consultation
- Detect emergency situations and direct to 911
#### Stage 3: Text-to-Speech (TTS)
- **Primary**: ElevenLabs TTS API
- Model: `eleven_turbo_v2_5` (optimized for speed, free tier compatible)
- Voice: Rachel (`21m00Tcm4TlvDq8ikWAM`)
- Quality: Natural voice synthesis
- **Fallback**: Browser Text-to-Speech API
- Activates automatically if ElevenLabs unavailable (503 errors)
- Uses system voices
- Zero latency, no API costs
### Real-World Benefits
**For Everyday Users**:
- **Hands-Free Access**: Check health info while cooking, exercising, or commuting
- **Natural Conversations**: Ask questions in plain language, no medical jargon required
- **Accessibility**: Ideal for users with visual impairments or reading difficulties
- **Quick Insights**: Faster than navigating multiple screens and charts
**For Health-Conscious Individuals**:
- **Report Interpretation**: "What do my cholesterol levels mean?"
- **Medication Context**: "Why am I taking this medication?"
- **Trend Analysis**: "How has my blood pressure changed over time?"
- **Personalized Guidance**: Responses tailored to your specific health profile
**Safety & Reliability**:
- **Emergency Detection**: Recognizes crisis keywords (chest pain, can't breathe, etc.)
- **Dosage Protection**: Refuses to recommend medication changes
- **HIPAA Guidelines**: All responses include disclaimers and professional consultation advice
- **Fallback Mechanisms**: Graceful degradation if AI/TTS services unavailable
### API Endpoints
| Method | Endpoint | Description | Auth | Response |
|--------|----------|-------------|------|----------|
| GET | `/api/voice/context` | Get user health context summary with profile completeness status | Yes | `{profile_complete: bool, summary: {...}, has_personalization: bool}` |
| POST | `/api/voice/answer` | Generate personalized AI answer using Gemini + health context | Yes | `{answer_text: string, flags: [string], used_context: {...}}` |
| POST | `/api/voice/tts` | Convert text to speech using ElevenLabs API | Yes | Binary audio stream (MP3) |
| GET | `/api/voice/tts/status` | Check ElevenLabs TTS configuration status | Yes | `{configured: bool, voice_id: string, runtime_check: {...}}` |
**Context Response Details**:
```json
{
"profile_complete": true,
"has_personalization": true,
"summary": {
"name": "Darshan Ved",
"age": 49,
"gender": "Male",
"bmi": 21.5,
"conditions": ["Hypertension"],
"medications": ["Lisinopril 10mg"],
"allergies": ["Penicillin"],
"reports_count": 5,
"sleep_hours": 7,
"exercise_frequency": "3-4 times/week"
}
}
```
**Safety Flags**:
- `emergency`: Detected emergency keywords β directs to 911
- `dosage_inquiry`: Detected medication dosage question β refuses specific advice
- `error`: Processing error β fallback message
### Configuration & Setup
#### Required Environment Variables
Add to **root `.env`** (for Docker) and **`backend/.env`** (for local development):
```env
# ===== AI/LLM Configuration =====
# Google Gemini API (Primary LLM for Voice Agent)
USE_GEMINI_FALLBACK=true
GEMINI_API_KEY=AIzaSy... # Get from https://aistudio.google.com/apikey
# ===== Text-to-Speech Configuration =====
# ElevenLabs TTS for Voice Agent (Optional - has browser fallback)
ELEVENLABS_API_KEY=sk_... # Get from https://elevenlabs.io/
ELEVENLABS_VOICE_ID=21m00Tcm4TlvDq8ikWAM # Rachel voice (recommended)
```
#### Docker Configuration
Ensure `docker-compose.yml` includes these environment variables in the backend service:
```yaml
services:
backend:
environment:
# ... other env vars ...
- GEMINI_API_KEY=${GEMINI_API_KEY:-}
- USE_GEMINI_FALLBACK=${USE_GEMINI_FALLBACK:-true}
- ELEVENLABS_API_KEY=${ELEVENLABS_API_KEY:-}
- ELEVENLABS_VOICE_ID=${ELEVENLABS_VOICE_ID:-21m00Tcm4TlvDq8ikWAM}
```
#### Getting API Keys
**Google Gemini API** (Free Tier Available):
1. Visit [Google AI Studio](https://aistudio.google.com/apikey)
2. Sign in with your Google account
3. Click "Get API Key" β "Create API key"
4. Copy the key (starts with `AIzaSy...`)
5. Free tier includes: 15 requests/minute, 1 million tokens/day
6. Supported model: `gemini-flash-latest` (auto-maps to best available)
**ElevenLabs API** (Optional - Free Tier Available):
1. Sign up at [ElevenLabs](https://elevenlabs.io/)
2. Navigate to Settings β API Keys
3. Create new API key (starts with `sk_...`)
4. Free tier includes: 10,000 characters/month
5. Recommended voice IDs:
- `21m00Tcm4TlvDq8ikWAM` - Rachel (conversational, female)
- `ErXwobaYiN019PkySvjV` - Antoni (clear, male)
- `EXAVITQu4vr4xnSDxMaL` - Bella (warm, female)
**Free Tier Model Compatibility**:
- Works: `gemini-flash-latest`, `gemini-1.5-flash-8b`
- Quota exceeded: `gemini-2.0-flash`, `gemini-2.5-flash` (requires paid tier)
### Technology Stack
| Component | Technology | Purpose |
|-----------|------------|---------|
| **Frontend STT** | Web Speech API (`webkitSpeechRecognition`) | Real-time voice transcription |
| **LLM Provider** | Google Gemini API (`gemini-flash-latest`) | Natural language understanding & generation |
| **Vector Database** | ChromaDB | RAG-based retrieval of historical health data |
| **Primary TTS** | ElevenLabs API (`eleven_turbo_v2_5`) | Studio-quality voice synthesis |
| **Fallback TTS** | Web Speech API (`SpeechSynthesis`) | Browser-native text-to-speech |
| **Backend Framework** | FastAPI | REST API endpoints |
| **Authentication** | JWT | Secure user-specific health context |
### Frontend Implementation
**Location**: `frontend/src/pages/VoiceAgent.tsx`
**Key Features**:
- **Orb Animation**: Uses Framer Motion for fluid listening/thinking/speaking states
- **State Management**: React hooks for recording, processing, speaking states
- **Error Handling**: Automatic fallback to browser TTS on ElevenLabs 503 errors
- **Accessibility**: Keyboard shortcuts, ARIA labels, screen reader support
- **Responsive Design**: Matches Lumea's light theme with purple accent colors
**User Flow**:
1. Click microphone button or press spacebar
2. Speak question (e.g., "What do my cholesterol levels mean?")
3. Watch orb animate while processing
4. Hear personalized response with health context
5. View transcript in chat panel (optional)
### Testing Voice Agent
#### 1. Check TTS Configuration Status
```bash
curl -X GET http://localhost:8000/api/voice/tts/status \
-H "Authorization: Bearer "
```
**Expected Response**:
```json
{
"configured": true,
"voice_id": "21m00Tcm4TlvDq8ikWAM",
"runtime_check": {
"api_key_loaded": true,
"api_key_prefix": "sk_d78...",
"voice_id": "21m00Tcm4TlvDq8ikWAM"
}
}
```
#### 2. Test Health Context Retrieval
```bash
curl -X GET http://localhost:8000/api/voice/context \
-H "Authorization: Bearer "
```
**Expected Response**:
```json
{
"profile_complete": true,
"has_personalization": true,
"summary": {
"name": "Darshan Ved",
"age": 49,
"gender": "Male",
"bmi": 21.5,
"conditions": ["Hypertension"],
"medications": ["Lisinopril"],
"allergies": ["Penicillin"],
"reports_count": 5
}
}
```
#### 3. Test AI Answer Generation
```bash
curl -X POST http://localhost:8000/api/voice/answer \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{
"text": "What do my cholesterol levels mean?"
}'
```
**Expected Response**:
```json
{
"answer_text": "Based on your recent lab reports, your total cholesterol is 195 mg/dL, which falls within the desirable range (below 200 mg/dL). Your LDL cholesterol is 110 mg/dL, also in the optimal range. However, I recommend discussing these results with your healthcare provider for personalized guidance based on your hypertension condition.",
"flags": [],
"used_context": {
"has_profile": true,
"conditions_count": 1,
"medications_count": 1,
"has_rag_context": true
}
}
```
#### 4. Test Text-to-Speech Conversion
```bash
curl -X POST http://localhost:8000/api/voice/tts \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"text": "Hello! Your health data shows positive trends."}' \
--output test_voice.mp3
# Play the audio file
# Windows: start test_voice.mp3
# Mac: open test_voice.mp3
# Linux: mpg123 test_voice.mp3
```
#### 5. Test Safety Features
**Emergency Detection Test**:
```bash
curl -X POST http://localhost:8000/api/voice/answer \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"text": "I am having severe chest pain"}'
```
**Expected Response**:
```json
{
"answer_text": "I'm detecting words that suggest this might be an emergency situation. Please call emergency services immediately (911) or go to the nearest emergency room...",
"flags": ["emergency"],
"used_context": {}
}
```
**Dosage Inquiry Test**:
```bash
curl -X POST http://localhost:8000/api/voice/answer \
-H "Authorization: Bearer " \
-H "Content-Type: application/json" \
-d '{"text": "Can I double my medication dosage?"}'
```
**Expected Response**:
```json
{
"answer_text": "I cannot provide specific dosage recommendations. Please consult your doctor or pharmacist before making any changes to your medications...",
"flags": ["dosage_inquiry"],
"used_context": {...}
}
```
### Troubleshooting
#### Issue: "TTS service not configured" (503 Error)
**Symptoms**: Voice Agent uses browser TTS instead of ElevenLabs
**Solutions**:
1. **Check API Key Loading**:
```bash
docker exec ggw-backend python -c "from app.settings import settings; print(f'API Key: {settings.ELEVENLABS_API_KEY[:20] if settings.ELEVENLABS_API_KEY else None}...')"
```
2. **Verify Environment Variables**:
- Ensure `ELEVENLABS_API_KEY` is in root `.env` file
- Ensure `docker-compose.yml` includes the env var mapping
- Restart Docker containers: `docker compose restart backend`
3. **Check API Key Validity**:
```bash
curl https://api.elevenlabs.io/v1/voices \
-H "xi-api-key: sk_your_key_here"
```
#### Issue: Gemini API 404 or 429 Errors
**Symptoms**: "Model not found" or "Quota exceeded"
**Solutions**:
1. **Verify Model Compatibility**:
```bash
docker exec ggw-backend python -c "import google.generativeai as genai; import os; genai.configure(api_key=os.environ['GEMINI_API_KEY']); models = [m.name for m in genai.list_models() if 'generateContent' in m.supported_generation_methods]; print('\n'.join(models))"
```
2. **Check Free Tier Limits**:
- Free tier model: `gemini-flash-latest`
- Paid tier models: `gemini-2.0-flash`, `gemini-2.5-flash`
- Rate limits: 15 requests/minute, 1M tokens/day
3. **Update Model Name** (if needed):
- Edit `backend/app/services/llm_service.py`
- Change model to `gemini-flash-latest`
- Restart backend
#### Issue: "ProfileMedication object has no attribute 'medication_name'"
**Symptoms**: Context loading errors in logs
**Solution**: Already fixed in codebase - medication attribute is `name`, not `medication_name`
#### Issue: Browser STT Not Working
**Symptoms**: Microphone icon doesn't activate
**Solutions**:
1. **Check Browser Compatibility**: Use Chrome, Edge, or Safari (Firefox has limited support)
2. **Enable Microphone Permissions**: Allow microphone access in browser settings
3. **Use HTTPS or localhost**: Web Speech API requires secure context
4. **Check Console Errors**: Open DevTools β Console for detailed error messages
### Example Conversations
**General Health Query**:
- User: "What does my health profile show?"
- Assistant: "You're a 49-year-old male with a healthy BMI of 21.5. Your profile shows hypertension, and you're currently taking Lisinopril. You have 5 health reports on file. Is there specific information you'd like me to explain?"
**Lab Result Interpretation**:
- User: "Explain my recent cholesterol test"
- Assistant: "Your recent cholesterol panel shows total cholesterol at 195 mg/dL, which is in the desirable range. Your LDL is 110 mg/dL and HDL is 55 mg/dL. Given your hypertension, maintaining these levels is important. Discuss with your doctor about your diet and medication effectiveness."
**Medication Context**:
- User: "Why am I taking Lisinopril?"
- Assistant: "Lisinopril is an ACE inhibitor prescribed for managing hypertension, which is noted in your health profile. It helps lower blood pressure by relaxing blood vessels. Always take it as prescribed and discuss any side effects with your healthcare provider."
---
## Physics Twin: Health Telemetry & Organ Scoring
Lumea's **Physics Twin** is a comprehensive digital health analysis engine that transforms raw health metrics from uploaded medical reports into deterministic organ-level health scores and real-time condition detection. It provides an interactive 3D body visualization, time-series telemetry monitoring, and AI-powered health recommendations grounded in your actual medical data.
> **Medical Disclaimer**: The Physics Twin is a visualization and analysis support tool. It does NOT provide medical diagnosis or treatment. Values are computed from extracted lab data and should be verified by healthcare professionals. Use for personal health tracking onlyβalways consult a licensed healthcare professional for medical decisions.
### Overview
The Physics Twin delivers:
- **Deterministic Organ Scoring**: Evidence-based scoring for 6 organs (kidney, heart, liver, lungs, brain, blood) based on reference ranges and clinical normalisation
- **Real Data Integration**: Pulls metrics directly from your uploaded medical reports (not simulated) via the `Observation` table
- **Interactive 3D Body**: Clickable organ hotspots with animated scoring visualizations and trend indicators
- **Real-Time Telemetry**: Simulated vital signs stream via SSE (Server-Sent Events) with SpO2, heart rate, blood pressure, stress, and glucose tracking
- **Condition Detection**: 10 rule-based conditions (Hypertension, Tachycardia, Kidney Stress, Liver Stress, Hyperglycemia, Sleep Deprivation, etc.) automatically detected from current metrics
- **Body Impact Overlay**: 2D SVG body silhouette highlighting organs affected by detected conditions with severity-based color coding
- **Smart Recommendations**: Context-aware health recommendations with YouTube educational links for each detected condition
- **AI Video Recommendations**: OpenRouter-powered personalized health video suggestions for each organ based on abnormal metrics, displayed in a side-by-side "Improve Your Health" card with per-organ caching
- **Explainability Panel**: Side-by-side "Why This Score?" card showing metric contributions, weights, coverage, and confidence for each organ
- **Time-Series History**: View per-report organ scores, historical trends, and scored snapshots over time
- **Lifestyle Integration**: Incorporates UserProfile self-reported data (sleep hours, stress level, activity level) into organ scoring
### How It Works
The Physics Twin operates in a three-layer pipeline:
```
LAYER 1: Data Aggregation
ββ Queries Observation table for authentic lab metrics (last 90 days)
ββ Extracts canonical metric names (systolic_bp, glucose, alt, etc.)
ββ Enriches with UserProfile lifestyle data (sleep_hours, stress_level)
ββ Deduplicates per metric_name, keeping most recent value
LAYER 2: Deterministic Scoring
ββ Applies physics_config scoring rules
ββ Normalises each metric to 0-1 range based on reference bounds
ββ Computes per-organ weighted scores (0-100)
ββ Calculates overall composite score across all organs
ββ Generates contribution breakdown for explainability
LAYER 3: Condition Detection & Recommendations
ββ Evaluates current metrics against 10+ condition rules
ββ Detects severity (mild/moderate/severe) per condition
ββ Maps conditions to affected organs
ββ Generates evidence-based recommendations
ββ Enriches with YouTube educational links per condition
```
### Architecture & Data Flow
```mermaid
graph TB
subgraph Database["PostgreSQL"]
Observation["Observation Table
(extracted metrics)"]
UserProfile["UserProfile Table
(lifestyle data)"]
Report["Report Table
(source documents)"]
end
subgraph Backend["Backend Services"]
PhysicsConfig["physics_config.py
- Organ specs
- Metric weights
- Reference ranges"]
PhysicsRoute["physics.py Routes
- GET /latest (auto-compute)
- GET /history (per-report)
- POST /metrics (manual)
- GET /config"]
Scorer["Scoring Engine
- compute_organ_score()
- compute_all_organs()"]
ConditionEngine["conditions.py
- detect_conditions()
- 10 rule-based conditions
- Severity mapping
- Organ mapping"]
TelemetryRoute["telemetry.py Routes
- GET /stream (SSE)
- GET /latest
- GET /history"]
end
subgraph Frontend["Frontend Components"]
PhysicsTwin["PhysicsTwin.tsx
- Main page orchestrator
- Tab navigation (Twin/Metrics/History)"]
TwinViewer["TwinViewer.tsx
- 3D body with Three.js
- Organ hotspots
- Auto-rotate"]
BodyOverlay["BodyImpactOverlay.tsx
- SVG body silhouette
- Animated organ zones
- Gender toggle"]
OrganTelemetry["OrganTelemetryCard.tsx
- Overall score display
- Selected organ detail
- Metric values"]
Explainability["ExplainabilityCard.tsx
- Scoring breakdown
- Weight contributions
- Coverage/confidence"]
Recommendations["RecommendationsCard.tsx
- Detected conditions
- Evidence-based actions
- YouTube links"]
ConditionsEngine["conditionsEngine.ts
- Client-side condition detection
- Rapid re-evaluation
- Trend computation"]
TelemetryHook["useTelemetryStream.ts
- SSE connection manager
- Fallback simulation
- History buffer (120 readings)"]
end
Observation -->|Query last 90 days| PhysicsRoute
UserProfile -->|Enrich with lifestyle| PhysicsRoute
Report -->|Group snapshots by| PhysicsRoute
PhysicsRoute -->|Load config| PhysicsConfig
PhysicsRoute -->|Compute scores| Scorer
Scorer -->|Detect conditions| ConditionEngine
TelemetryRoute -->|Generate readings| TelemetryHook
PhysicsRoute -->|Return snapshot| PhysicsTwin
PhysicsTwin -->|Render organs| TwinViewer
PhysicsTwin -->|Show overlay| BodyOverlay
PhysicsTwin -->|Display score| OrganTelemetry
PhysicsTwin -->|Show breakdown| Explainability
PhysicsTwin -->|Detect conditions| ConditionsEngine
ConditionsEngine -->|Map to organs| BodyOverlay
ConditionsEngine -->|Show recommendations| Recommendations
TelemetryHook -->|Stream vitals| PhysicsTwin
```
### Core Concepts
#### Organ Systems (6 Total)
| Organ | Metrics | Purpose |
|-------|---------|---------|
| **Kidney** | creatinine, urea, egfr, sodium, potassium, systolic_bp | Filtration & electrolyte balance |
| **Heart** | heart_rate, systolic_bp, diastolic_bp, spo2, cholesterol, triglycerides | Cardiovascular function |
| **Liver** | alt, ast, bilirubin_total, albumin, alp, total_protein | Detoxification & synthesis |
| **Lungs** | spo2, respiratory_rate, hemoglobin | Oxygen exchange & gas transfer |
| **Brain** | sleep_hours, stress_level, glucose, systolic_bp, tsh | Neurological health |
| **Blood** | hemoglobin, wbc_total, platelet_count, rbc_count, glucose, hba1c, ferritin | Cell counts & glucose |
#### Scoring Methodology
Each metric is **normalised to 0-1** based on clinical reference ranges:
```
If metric_value is in reference_range [ref_min, ref_max]:
normalised_score = 1.0 (perfect)
Else if metric_value < ref_min:
normalised_score = (metric_value - abs_min) / (ref_min - abs_min)
(linear decay to 0 at absolute minimum)
Else if metric_value > ref_max:
normalised_score = 1.0 - (metric_value - ref_max) / (abs_max - ref_max)
(linear decay to 0 at absolute maximum)
```
**Organ Score** = weighted average of present metrics:
```
score = Ξ£(weight_i Γ normalised_i) / Ξ£(weight_present) Γ 100
```
**Overall Score** = mean of all organ scores with data
**Status Classification**:
- **Healthy**: 75-100
- **Watch**: 50-74
- **Risk**: 0-49
#### Metric Name Alignment
All metric names match PostgreSQL `Observation.metric_name` canonical keys:
```python
# Correct DB names (used everywhere)
"systolic_bp", "diastolic_bp", "heart_rate", "spo2", "respiratory_rate",
"creatinine", "urea", "egfr", "sodium", "glucose", "ast", "alt",
"bilirubin_total", "hemoglobin", "stress_level", "sleep_hours"
```
This ensures consistent data flow from OCR extraction β database storage β physics scoring.
#### Conditions Detection
10 rule-based conditions evaluated against current metrics:
| Condition | Severity | Trigger Metrics | Affected Organs |
|-----------|----------|-----------------|-----------------|
| Hypertension | mild/moderate/severe | systolic_bp β₯ 130/140/160 mmHg | heart, kidney, brain |
| Tachycardia | mild/moderate/severe | heart_rate β₯ 100/120/150 bpm | heart |
| Bradycardia | mild/moderate/severe | heart_rate β€ 55/50/40 bpm | heart, brain |
| Hypoxemia | mild/moderate/severe | spo2 β€ 94/90/85 % | lungs, heart, brain |
| Kidney Stress | mild/moderate/severe | creatinine β₯ 1.3/1.8/3.0 mg/dL | kidney |
| Liver Stress | mild/moderate/severe | alt β₯ 60/100/200 U/L | liver |
| Hyperglycemia | mild/moderate/severe | glucose β₯ 110/140/200 mg/dL | kidney, heart, brain |
| High Stress | mild/moderate/severe | stress_level β₯ 4/6/8 | brain, heart |
| Sleep Deprivation | mild/moderate/severe | sleep_hours β€ 6.5/5/4 hrs | brain, heart |
| Tachypnea | mild/moderate/severe | respiratory_rate β₯ 22/26/30 bpm | lungs |
### Backend Implementation
#### Files Modified/Created
1. **`app/services/physics_config.py`**: Organ & metric specifications
- 6 organ definitions with metric weights
- Reference ranges & absolute bounds per metric
- `compute_organ_score()`: Deterministic scoring function
- `compute_all_organs()`: Multi-organ aggregation
2. **`app/routes/physics.py`**: Smart API endpoints
- `GET /api/physics/latest`: Auto-compute from real DB data (90-day window)
- `GET /api/physics/history`: Per-report snapshots
- `POST /api/physics/metrics`: Manual submission (fallback)
- `GET /api/physics/config`: Frontend config download
3. **`app/services/conditions.py`**: Rule engine
- 10 `ConditionRule` definitions
- `detect_conditions()`: Main evaluation function
- Threshold checking with severity mapping
- Organ affection mapping
4. **`app/routes/telemetry.py`**: Simulated SSE stream
- Baseline vital signs (heart_rate, bp, spo2, glucose, etc.)
- Per-user simulation state tracking
- Realistic jitter + periodic spikes
- `GET /stream` SSE endpoint
#### Data QueryQuery Pattern (physics.py)
```python
# Gather last 90 days of metrics for user
result = await db.execute(
select(Observation)
.where(
and_(
Observation.user_id == user_id,
Observation.observed_at >= cutoff # 90 days ago
)
)
.order_by(desc(Observation.observed_at))
)
# Deduplicate: keep most recent value per metric_name
metrics = {}
for obs in sorted_observations:
if obs.metric_name not in seen:
metrics[obs.metric_name] = float(obs.value)
```
### Frontend Implementation
#### Main Components
1. **`PhysicsTwin.tsx`** (616 lines): Page orchestrator
- Three tabs: Twin View (3D/overlay), Metrics (vitals grid), History (trends + snapshots)
- Auto-loads latest snapshot on mount
- Streams real-time telemetry via SSE hook
- Auto-detects conditions from current metrics
- Handles organ selection state
2. **`TwinViewer.tsx`** (297 lines): 3D body renderer
- Uses `@react-three/fiber` + `@react-three/drei` + Three.js
- Stylized torso + head mesh
- 6 animated organ orbs with onClick handlers
- Auto-rotating when no organ selected
- Smooth camera tween to selected organ
- Responsive scaling & glow effects
3. **`BodyImpactOverlay.tsx`** (263 lines): 2D condition visualization
- SVG body silhouette (male/female toggle)
- 6 organ zones with animated pulsing
- Severity-based color coding (mild/moderate/severe)
- Condition pill badges at top
- Gender-specific body paths
4. **`OrganTelemetryCard.tsx`** (159 lines): Score summary
- Large animated overall score display
- Selected organ detail (coverage %, status badge)
- Per-metric contribution breakdown with visual bars
- Last updated timestamp
5. **`ExplainabilityCard.tsx`**: Scoring transparency
- Breakdown table: metric name, raw value, normalised score, weight
- Contribution % calculation
- Coverage metric (% of available metrics present)
- Confidence score (weighted average)
6. **`YouTubeRecommendationsCard.tsx`**: AI-powered video recommendations
- Generates organ-specific health video suggestions via OpenRouter (GPT-4o-mini)
- Per-organ caching with `useRef