Spaces:

thisismrismail
/

scenarist

Sleeping

App Files Files Community

scenarist / SETUP.md

github-actions[bot]

Sync backend to Hugging Face Space (commit: 39b5c807918249fa80049d49f4b6a74d6a0ed1fc)

6d86412 22 days ago

preview code

Raw

History Blame Contribute Delete

27.3 kB

Orsync Scenarist v7.0 — Complete Setup & Running Guide

Goal: Walk you through every download, account, and command needed to run the Orsync Scenarist backend on a fresh machine — from zero to curl http://localhost:7860/healthz.

Accounts You Need
Software to Download & Install
Clone the Repository
Start Infrastructure Services (Redis, Neo4j, ChromaDB, RabbitMQ)
Create a Python Virtual Environment
Install Python Dependencies
Configure Environment Variables (.env)
Generate Seed Data
Run the Backend Server
Verify Everything Works
Run the Test Suite
Deploy to Hugging Face Spaces
Docker Deployment (Local)
Production Checklist
API Route Map
Troubleshooting

1. Accounts You Need

Before you start, create accounts on the following services. Some are required, others are optional depending on which features you want.

Account	Required?	Why	Sign-up Link
Ollama Cloud	Yes (for LLM features)	Cloud API for running gemma4:31b-cloud and other large models. Without it, LLM features fall back to rule-based templates.	https://ollama.com/
Neo4j AuraDB	Production only	Managed graph database. For local dev you use the free Docker Community edition.	https://neo4j.com/cloud/aura-free/
Hume AI	Optional	Prosody & sentiment analysis during WebRTC simulations.	https://www.hume.ai/
D-ID	Optional	AI video avatar synthesis for simulation personas.	https://www.d-id.com/
Docker Hub	Optional	Only needed if you want to push custom images. Docker Desktop itself is free.	https://hub.docker.com/signup
Hugging Face	For HF Spaces deploy	Free account to deploy the app to Hugging Face Spaces.	https://huggingface.co/join

API Keys to Collect

After creating accounts, gather these keys (you'll paste them into your .env file in Step 7):

Ollama Cloud API Key — Go to https://ollama.com/settings/keys → Create Key
Hume API Key (optional) — Hume dashboard → API Keys
D-ID API Key (optional) — D-ID dashboard → API Keys

2. Software to Download & Install

Install the following tools on your machine. Follow each link and use the installer for your operating system.

2.1 — Python 3.12+

OS	How to Install
Windows	Download from https://www.python.org/downloads/ → Run installer → Check "Add python.exe to PATH"
macOS	`brew install python@3.12` (requires Homebrew)
Linux (Ubuntu/Debian)	`sudo apt update && sudo apt install python3.12 python3.12-venv python3-pip`

Verify:

python --version
# → Python 3.12.x

2.2 — Git

OS	How to Install
Windows	Download from https://git-scm.com/download/win → Run installer (defaults are fine)
macOS	`xcode-select --install` or `brew install git`
Linux	`sudo apt install git`

Verify:

git --version
# → git version 2.x.x

2.3 — Docker Desktop

Docker is used to run infrastructure services (Redis, Neo4j, ChromaDB, RabbitMQ) locally. It is also used for containerised deployment.

OS	How to Install
Windows	Download from https://www.docker.com/products/docker-desktop/ → Run installer → Restart PC if prompted. Requires WSL 2 (the installer will prompt you to enable it).
macOS	Download from https://www.docker.com/products/docker-desktop/ → Drag to Applications
Linux	Follow https://docs.docker.com/engine/install/ for your distro

Verify (after restarting terminal):

docker --version
# → Docker version 24.x.x or newer

docker compose version
# → Docker Compose version v2.x.x

2.4 — (Optional) Redis CLI

The Redis CLI (redis-cli) is useful for debugging but not required — you can skip it and just rely on the Docker container.

OS	How to Install
Windows	Included with Docker. Or install via `winget install Redis.Redis`
macOS	`brew install redis`
Linux	`sudo apt install redis-tools`

2.5 — curl (or Invoke-WebRequest)

Used in this guide to test endpoints. Most systems have it pre-installed.

Windows PowerShell: curl is an alias for Invoke-WebRequest. For the exact same syntax as this guide, use curl.exe or install Git Bash.
macOS / Linux: Pre-installed.

3. Clone the Repository

Open a terminal (PowerShell on Windows, Terminal on macOS/Linux):

# Step 3.1 — Navigate to where you keep your projects
cd ~/code               # or wherever you prefer

# Step 3.2 — Clone
git clone <your-repo-url> orsync-scenarist

# Step 3.3 — Enter the project
cd orsync-scenarist

You should see this structure:

orsync-scenarist/
├── backend/            ← FastAPI backend (this guide focuses here)
├── data-sandbox/       ← Data pipeline (bronze → silver → gold)
├── README.md
└── SETUP.md            ← This file

4. Start Infrastructure Services

The backend uses Redis (cache/sessions/outbox), Neo4j (knowledge graph), ChromaDB (vector store), and RabbitMQ (alternative message transport). Redis, Neo4j, and ChromaDB are started automatically by run.py and the Docker image. RabbitMQ is optional — the app falls back to a stub if it's missing.

Important: Make sure Docker Desktop is running if you want RabbitMQ.

Step 4.1 — Redis (automatic — no setup needed)

Redis is started automatically in two ways:

Local development: run.py starts an embedded redis-server process on localhost:6379
Docker / HF Spaces: The Dockerfile installs and starts redis-server inside the container

If you prefer to manage Redis yourself, skip the auto-start with --no-redis:

python run.py --no-redis

To install redis-server locally (optional — only needed for local dev):

OS	How to Install
Windows	`winget install Redis.Redis` or `scoop install redis`
macOS	`brew install redis`
Linux	`sudo apt install redis-server`

Step 4.2 — Neo4j (automatic — no setup needed)

Neo4j is started automatically in two ways:

Local development: run.py starts a neo4j console process on bolt://localhost:7687
Docker / HF Spaces: The Dockerfile installs Neo4j Community Edition and starts it inside the container

If you prefer to manage Neo4j yourself, skip the auto-start with --no-neo4j:

python run.py --no-neo4j

To install Neo4j locally (optional — only needed for local dev):

OS	How to Install
Windows	Download from https://neo4j.com/download/ or `winget install Neo4j.Neo4j`
macOS	`brew install neo4j`
Linux	See https://neo4j.com/docs/operations-manual/current/installation/linux/

Note: Neo4j requires Java 17+. Most Neo4j installers include it automatically.

Step 4.3 — ChromaDB (automatic — no setup needed)

ChromaDB is started automatically in two ways:

Local development: run.py starts a chroma run process on localhost:8000
Docker / HF Spaces: The Dockerfile runs the chroma server inside the container

If you prefer to manage ChromaDB yourself, skip the auto-start with --no-chroma:

python run.py --no-chroma

ChromaDB is installed as part of the Python dependencies (pip install chromadb), which includes both the client library and the chroma CLI server.

Step 4.4 — Start RabbitMQ (optional)

docker run -d --name scenarist-rabbitmq \
  -p 5672:5672 -p 15672:15672 \
  rabbitmq:3-management

Verify: Open http://localhost:15672 in your browser. Log in with:

Username: guest
Password: guest

Quick Reference — Stop / Restart Optional Services

# Stop all
docker stop scenarist-rabbitmq

# Start again (data is preserved)
docker start scenarist-rabbitmq

# Remove containers entirely (data is lost)
docker rm -f scenarist-rabbitmq

5. Create a Python Virtual Environment

# Step 5.1 — Make sure you're in the project root (orsync-scenarist/)

# Step 5.2 — Create a virtual environment
python -m venv .venv

Step 5.3 — Activate the virtual environment

You must activate it every time you open a new terminal.

Windows (PowerShell):

.venv\Scripts\Activate.ps1

If you get an execution policy error, run this first (one-time):
Set-ExecutionPolicy -Scope CurrentUser -ExecutionPolicy RemoteSigned

Windows (Command Prompt):

.venv\Scripts\activate.bat

macOS / Linux:

source .venv/bin/activate

You'll see (.venv) at the start of your prompt when it's active.

6. Install Python Dependencies

With the virtual environment activated (see Step 5.3):

Step 6.1 — Install all dependencies

pip install -r requirements.txt

This installs everything: FastAPI, Redis, Neo4j, ChromaDB, Ollama SDK, scikit-learn, pytest, fakeredis, and all other packages.

Step 6.2 — (Optional) Install PyTorch for medical embeddings

Only needed if you want to use the PubMedBERT embedding model instead of the default ONNX MiniLM model. This is a ~2 GB download.

CPU-only (recommended — smaller and sufficient for inference):

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install sentence-transformers

With GPU support (if you have an NVIDIA GPU with CUDA):

pip install torch sentence-transformers

Verify Installation

python -c "import fastapi; print('FastAPI', fastapi.__version__)"
# → FastAPI 0.115.x

python -c "import chromadb; print('ChromaDB', chromadb.__version__)"
# → ChromaDB 1.0.x

7. Configure Environment Variables (.env)

The backend reads configuration from environment variables. The easiest way is to create a .env file inside the backend/ folder.

Step 7.1 — Create the file

Windows (PowerShell):

New-Item -Path .env -ItemType File

macOS / Linux:

touch .env

Step 7.2 — Paste your configuration

Open .env in any text editor and paste the following. Replace placeholder values with your actual keys where noted.

# ──────────────────────────────────────────────────────────────────────
# Orsync Scenarist v7.0 — Environment Configuration
# ──────────────────────────────────────────────────────────────────────

# ─── General ──────────────────────────────────────────────────────────
ENVIRONMENT=development          # development | staging | production
APP_NAME=Orsync Scenarist Backend

# ─── Infrastructure ───────────────────────────────────────────────────
REDIS_URL=redis://localhost:6379/0
RABBITMQ_URL=amqp://guest:guest@localhost:5672/
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=scenarist123
CHROMA_HOST=localhost
CHROMA_PORT=8000
OUTBOX_TRANSPORT=redis_streams    # redis_streams | amqp

# ─── LLM / AI (Ollama Cloud) ─────────────────────────────────────────
# 👉 REPLACE with your real Ollama Cloud API key from https://ollama.com/settings/keys
OLLAMA_HOST=https://ollama.com
OLLAMA_API_KEY=your-ollama-api-key-here
OLLAMA_MODEL=gemma4:31b-cloud

# ─── Embedding Model ─────────────────────────────────────────────────
# "onnx-minilm"              → built-in 384-dim (local, no API key needed)
# "ollama:nomic-embed-text"  → Ollama embed API (768-dim)
EMBEDDING_MODEL=onnx-minilm

# ─── External Simulation APIs (optional) ─────────────────────────────
HUME_API_KEY=                    # Hume AI prosody/sentiment
DID_API_KEY=                     # D-ID video synthesis

# ─── JWT Authentication ──────────────────────────────────────────────
JWT_SECRET_KEY=my-strong-random-secret-at-least-32-chars
JWT_ALGORITHM=HS256
JWT_ACCESS_TOKEN_EXPIRE_MINUTES=60

# ─── CORS ─────────────────────────────────────────────────────────────
CORS_ALLOWED_ORIGINS=http://localhost:3000,http://127.0.0.1:3000

# ─── Neural Projection Bridge ────────────────────────────────────────
PROJECTION_WEIGHTS_PATH=projection_weights.pt.npz

# ─── Strategy Engine ─────────────────────────────────────────────────
STRATEGY_REJECTION_DISTANCE_THRESHOLD=3.0

Minimum Viable `.env` (no LLM calls)

If you just want to start the server without an Ollama key, use this minimal config. LLM-dependent features will fall back to rule-based template responses:

ENVIRONMENT=development
REDIS_URL=redis://localhost:6379/0
NEO4J_URI=bolt://localhost:7687
NEO4J_USERNAME=neo4j
NEO4J_PASSWORD=scenarist123

Environment Variable Reference

Variable	Default	Description
`ENVIRONMENT`	`development`	`development`, `staging`, or `production`. Production mode enforces real secrets.
`REDIS_URL`	`redis://localhost:6379/0`	Redis connection string
`NEO4J_URI`	`bolt://localhost:7687`	Neo4j Bolt protocol URI
`NEO4J_USERNAME`	`neo4j`	Neo4j username
`NEO4J_PASSWORD`	`scenarist123`	Neo4j password
`CHROMA_HOST`	`localhost`	ChromaDB hostname
`CHROMA_PORT`	`8000`	ChromaDB port
`RABBITMQ_URL`	`amqp://guest:guest@localhost:5672/`	RabbitMQ AMQP URI
`OLLAMA_HOST`	`https://ollama.com`	Ollama Cloud API endpoint
`OLLAMA_API_KEY`	(none)	Ollama Cloud API key
`OLLAMA_MODEL`	`gemma4:31b-cloud`	LLM model name
`EMBEDDING_MODEL`	`onnx-minilm`	Embedding model identifier
`JWT_SECRET_KEY`	`changeme...`	JWT signing secret (change in prod!)
`JWT_ACCESS_TOKEN_EXPIRE_MINUTES`	`60`	Token TTL in minutes
`CORS_ALLOWED_ORIGINS`	`*`	Comma-separated list of allowed origins
`HUME_API_KEY`	(none)	Hume AI key (optional)
`DID_API_KEY`	(none)	D-ID key (optional)
`PROJECTION_WEIGHTS_PATH`	`projection_weights.pt.npz`	Path to neural projection weights

8. Generate Seed Data

The backend ships with a script that generates 200 synthetic HCP (Healthcare Professional) records across four GMM archetypes.

Step 8.1 — Generate the JSON file

# Make sure you're in the project root with .venv activated

python -m scripts.generate_seed_data
# → Generated 200 doctor records -> data/gold/doctors_unified.json

The output file is at data/gold/doctors_unified.json.

Step 8.2 — Ingest into Neo4j (after the server is running)

Once the backend is running (see Step 9), seed Neo4j:

curl -X POST http://localhost:7860/api/pipeline/seed

On Windows PowerShell, use:

Invoke-WebRequest -Method POST -Uri http://localhost:7860/api/pipeline/seed

9. Run the Backend Server

Step 9.1 — Make sure prerequisites are ready

Before starting, confirm:

Docker containers are running (Step 4)
Virtual environment is activated (Step 5.3)
Dependencies are installed (Step 6)
.env file is created (Step 7)

Step 9.2 — Start the development server

uvicorn app.main:app --host 0.0.0.0 --port 7860 --reload

The --reload flag enables auto-reload when you edit code. The server starts on http://localhost:7860.

On startup the server will:

Validate production secrets (skipped in development mode)
Pre-load the embedding model (all-MiniLM-L6-v2 ONNX by default)
Pre-load projection bridge weights (Xavier random defaults if no .npz file)
Start the outbox dispatcher (background thread)
Start the event consumer (background thread, reads Redis Streams)

What happens if services are missing?

Scenario	What happens
No `redis-server` installed	`run.py` warns, app uses in-memory fallback (sessions/cache not persisted)
No Neo4j installed	`run.py` warns, graph queries return empty results (no-op stub)
No ChromaDB installed	`run.py` warns, vector search returns empty results (no-op stub)
No `OLLAMA_API_KEY`	LLM features fall back to rule-based template responses
No `projection_weights.pt.npz`	Projection bridge uses Xavier-initialised random weights (safe for dev)
`ENVIRONMENT=production` + placeholder secrets	Server refuses to boot (fail-fast safety)

10. Verify Everything Works

Step 10.1 — Health check

curl http://localhost:7860/healthz
# → {"status":"ok"}

Windows PowerShell:

(Invoke-WebRequest http://localhost:7860/healthz).Content

Step 10.2 — Open the API docs

Open your browser and navigate to:

Swagger UI: http://localhost:7860/docs
ReDoc: http://localhost:7860/redoc

Step 10.3 — Register a test user

curl -X POST http://localhost:7860/api/auth/register \
  -H "Content-Type: application/json" \
  -d '{"username": "testuser", "password": "testpass123"}'

Step 10.4 — Log in

curl -X POST http://localhost:7860/api/auth/login \
  -H "Content-Type: application/json" \
  -d '{"username": "testuser", "password": "testpass123"}'
# → {"access_token": "eyJ...", "token_type": "bearer"}

Step 10.5 — Seed Neo4j (if not done already)

curl -X POST http://localhost:7860/api/pipeline/seed

11. Run the Test Suite

Tests use fakeredis to mock Redis, so no running infrastructure is required.

# Step 11.1 — Run all tests
pytest

# Step 11.2 — Run with verbose output
pytest -v

# Step 11.3 — Run a specific test file
pytest tests/test_health.py
pytest tests/test_strategy_heatmap.py
pytest tests/test_eventual_consistency.py
pytest tests/test_math_engine_validation.py
pytest tests/test_warroom_latency.py
pytest tests/test_medallion_pipeline.py

# Step 11.4 — Run with coverage report
pytest --cov=app --cov-report=term-missing

12. Deploy to Hugging Face Spaces

The project is pre-configured for one-click deployment to Hugging Face Spaces using the Docker SDK.

Step 12.1 — Create a Hugging Face account

Go to https://huggingface.co/join and create a free account (if you don't have one).

Step 12.2 — Create a new Space

Go to https://huggingface.co/new-space
Fill in:
- Owner: your username
- Space name: scenarist (or any name you prefer)
- SDK: select Docker
- Visibility: Public or Private
Click Create Space.

Step 12.3 — Push the repo to the Space

# Add the HF Space as a remote (replace YOUR_USERNAME)
git remote add hf https://huggingface.co/spaces/YOUR_USERNAME/scenarist

# Push
git push hf main

Hugging Face will build the Docker image and start the app automatically.

Step 12.4 — (Optional) Add Secrets for full functionality

In your Space settings → Repository secrets, add any of these:

Secret	Purpose
`OPENROUTER_API_KEY`	Enables LLM features via OpenRouter (campaign optimisation, persona dialogue)
`JWT_SECRET_KEY`	Secure JWT signing key

Without any secrets, the app still starts — Redis, Neo4j, and ChromaDB run embedded inside the container, while LLM features use rule-based templates.

Step 12.5 — Verify

Once the build finishes, your app will be live at:

https://YOUR_USERNAME-scenarist.hf.space/docs

13. Docker Deployment (Local)

13.1 — Build & Run Standalone

# Build the image
docker build -t scenarist-backend:7.0 .

# Run (pass env vars via .env file)
docker run -d \
  --name scenarist-api \
  -p 7860:7860 \
  --env-file .env \
  scenarist-backend:7.0

13.2 — Docker Compose (Full Stack)

Create a docker-compose.yml at the repo root:

version: "3.9"

services:
  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]

  neo4j:
    image: neo4j:5-community
    ports: ["7474:7474", "7687:7687"]
    environment:
      NEO4J_AUTH: neo4j/scenarist123

  chroma:
    image: chromadb/chroma:latest
    ports: ["8000:8000"]

  rabbitmq:
    image: rabbitmq:3-management
    ports: ["5672:5672", "15672:15672"]

  backend:
    build: .
    ports: ["7860:7860"]
    env_file: .env
    environment:
      REDIS_URL: redis://redis:6379/0
      NEO4J_URI: bolt://neo4j:7687
      CHROMA_HOST: chroma
      RABBITMQ_URL: amqp://guest:guest@rabbitmq:5672/
    depends_on:
      - redis
      - neo4j
      - chroma
      - rabbitmq

# Start everything
docker compose up -d

# Check logs
docker compose logs -f backend

# Seed Neo4j
curl -X POST http://localhost:7860/api/pipeline/seed

# Stop everything
docker compose down

14. Production Checklist

Before deploying with ENVIRONMENT=production, complete every item:

JWT_SECRET_KEY — Set to a cryptographically random string (>= 32 chars). Generate one: python -c "import secrets; print(secrets.token_urlsafe(48))"
NEO4J_PASSWORD — Not a placeholder (scenarist123, changeme, etc.)
OPENROUTER_API_KEY — Set to a valid API key
CORS_ALLOWED_ORIGINS — Set to your actual frontend domains (no *)
REDIS_URL — Points to your managed Redis (e.g. AWS ElastiCache)
NEO4J_URI — Points to your managed Neo4j (e.g. AuraDB)
CHROMA_HOST / CHROMA_PORT — Points to your managed ChromaDB
RABBITMQ_URL — Points to your managed RabbitMQ (e.g. Amazon MQ)
projection_weights.pt.npz — Trained projection weights file is deployed

The server will refuse to start in production mode if critical secrets are left at placeholder defaults. This is a deliberate fail-fast safety mechanism.

15. API Route Map

Once running, interactive OpenAPI docs are at http://localhost:7860/docs.

Prefix	Tag	Key Endpoints
`/api/auth`	auth	`POST /register`, `POST /login`, `GET /me`
`/api/strategy`	strategy	`POST /full-evaluate`, `POST /heatmap`, `POST /optimize`, `POST /evaluate`
`/api/simulation`	simulation	`POST /start`, `POST /handshake`, `POST /ice-candidate`, `POST /turn`
`/api/persona`	persona	`GET /from-cluster/{id}`, `GET /{code_name}`
`/api/graph`	graph	`POST /ingest`, `GET /doctor/{name}`, `GET /cluster/{id}/doctors`
`/api/mohp`	mohp	`POST /evaluate`
`/api/math`	math	`POST /vectorize`, `POST /cluster`
`/api/pipeline`	pipeline	`POST /ingest`, `POST /dispatch`, `POST /seed`
`/api/analytics`	analytics	`GET /sessions`, `GET /session/{id}`
`/api/stats`	stats	`GET /embedding`, `GET /projection`, `GET /cache`, `GET /dlq`, `GET /outbox`
`/admin/embeddings`	admin	`GET /status`, `POST /swap`, `POST /reindex`
`/healthz`	—	`GET` → `{"status": "ok"}`

16. Troubleshooting

"FATAL: Production environment detected with insecure / placeholder secrets"

You're running with ENVIRONMENT=production but haven't changed default passwords. Set real values for JWT_SECRET_KEY, NEO4J_PASSWORD, and OPENROUTER_API_KEY.

ChromaDB connection refused

httpx.ConnectError: [Errno 111] Connection refused

ChromaDB is not running on the expected host/port. It should start automatically:

Local: python run.py starts it (ensure chromadb is installed: pip install chromadb)
Docker: the Dockerfile starts it inside the container

If managing it yourself, check CHROMA_HOST / CHROMA_PORT in your .env.

Neo4j authentication failure

neo4j.exceptions.AuthError: {code: Neo.ClientError.Security.Unauthorized}

Your NEO4J_PASSWORD doesn't match what Neo4j was initialised with. If using Docker, destroy and recreate the container:

docker rm -f scenarist-neo4j
docker run -d --name scenarist-neo4j -p 7474:7474 -p 7687:7687 \
  -e NEO4J_AUTH=neo4j/scenarist123 neo4j:5-community

Redis connection error on startup

The server will start, but outbox/session/cache operations fail. Ensure Redis is running:

docker start scenarist-redis
# or create a new container:
docker run -d --name scenarist-redis -p 6379:6379 redis:7-alpine

"sentence-transformers is required for custom embedding models"

You selected EMBEDDING_MODEL=pritamdeka/S-PubMedBert-MS-MARCO but haven't installed the optional torch dependencies:

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install sentence-transformers

Tests fail with import errors

Make sure you're in the project root with the virtual environment activated, and have installed dependencies:

.venv\Scripts\Activate.ps1          # Windows
# source .venv/bin/activate          # macOS/Linux
pip install -r requirements.txt
pytest

Projection bridge warning: "using Xavier-initialised defaults"

This is expected in development. The projection bridge generates random weights as a fallback. For production, train and place projection_weights.pt.npz in the backend root or configure PROJECTION_WEIGHTS_PATH.

PowerShell `curl` returns HTML instead of JSON

PowerShell aliases curl to Invoke-WebRequest. Use curl.exe instead, or:

(Invoke-WebRequest http://localhost:7860/healthz).Content

Docker "port already in use"

Another process is using the port. Find and stop it:

# Windows
netstat -ano | findstr :7860
taskkill /PID <pid> /F

# macOS / Linux
lsof -i :7860
kill <pid>

Or change the port mapping in the docker run command (e.g. -p 9090:7860).