DEPLOYMENT.md

Deployment Guide

This document covers all deployment options for Irminsul, the cost tradeoffs between them, and the architectural decisions behind the live demo setup.

---

Deployment Options

Irminsul supports two LLM backends and multiple hosting targets. Choose based on your infrastructure and budget.

Backend	Where to Run	GPU Required	Cost
Groq (recommended)	Anywhere — no GPU	No	Free tier available
Local Llama (fine-tuned model)	Local machine / GPU VM	Yes (6GB+ VRAM)	Hardware cost / ~$0.50–1.50/hr on Azure

---

Live Demo: HuggingFace Spaces + Groq

Why this is the live demo environment:

The fine-tuned Llama 3.1 8B model is 16GB on disk and requires a GPU-enabled instance to serve at acceptable latency. On Azure, the minimum viable GPU instance for this model is the NC4as T4 v3 (~$0.50/hr, ~$360/month). Running this persistently for a portfolio project is not cost-effective.

The live demo instead uses:

• HuggingFace Spaces — free CPU hosting for the FastAPI container
• Groq API — runs llama-3.3-70b-versatile on Groq's Language Processing Units (LPUs) at ~300 tokens/second, for free under the public tier

This demonstrates the identical RAG architecture — the LLM backend is swapped via a single environment variable (LLM_BACKEND=groq). The retrieval pipeline, guardrails, response format, and API contract are unchanged.

Live demo:  https://huggingface.co/spaces/MukulRay/Irminsul

---

Option A: Local Development

The full stack including the fine-tuned model runs locally on an RTX 3060 6GB:

# 1. Clone and install
git clone https://github.com/MukulRay1603/Irminsul.git
cd Irminsul
python -m venv venv && source venv/bin/activate
pip install -r requirements.txt

# 2. Configure
cp .env.example .env
# Edit .env — set MODEL_PATH, PINECONE_API_KEY

# 3. Ingest corpus
python ingest.py --dir ./docs --chunk-size 300 --chunk-overlap 40

# 4. Serve
uvicorn main:app --host 0.0.0.0 --port 8000

Memory profile:

Component	VRAM
Llama 3.1 8B @ 4-bit NF4	~4.5 GB
all-MiniLM-L6-v2 embedder	~90 MB
Inference headroom	~1.2 GB
Total	~5.8 GB

The model loads with max_memory={0: "5.5GiB", "cpu": "24GiB"} — layers that don't fit on GPU overflow to RAM automatically via accelerate.

---

Option B: Docker (Local or Any Cloud)

The Dockerfile is intentionally slim — the model is not baked in. It's injected at runtime via MODEL_PATH.

# Build
docker build -t irminsul:latest .

# Run with Groq backend (no GPU needed)
docker run -p 8000:8000 \
  -e PINECONE_API_KEY=your_key \
  -e GROQ_API_KEY=your_key \
  -e PINECONE_INDEX=llmops-rag \
  -e LLM_BACKEND=groq \
  irminsul:latest

# Run with local model (GPU required)
docker run -p 8000:8000 \
  --gpus all \
  -v /path/to/models:/app/models \
  -e PINECONE_API_KEY=your_key \
  -e MODEL_PATH=/app/models/merged/exp2_lr2e-4_r16 \
  -e LLM_BACKEND=local \
  irminsul:latest

---

Option C: Azure Container Apps

Azure Container Apps (ACA) is the production deployment target. The deploy_azure.sh script provisions the full stack in one command.

Prerequisites

# Install Azure CLI
# macOS:
brew install azure-cli

# Linux:
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash

# Windows: https://aka.ms/installazurecliwindows

# Log in
az login
az account show  # confirm your subscription

One-shot deploy

export PINECONE_API_KEY=your_pinecone_key
export GROQ_API_KEY=your_groq_key
chmod +x deploy_azure.sh
./deploy_azure.sh

The script:

1. Creates resource group irminsul-rg in East US
2. Creates Azure Container Registry irminsulacr
3. Builds the Docker image via ACR Tasks — the source code is uploaded and built in Azure's cloud; no local Docker daemon needed
4. Creates a Container Apps environment
5. Deploys the app with secrets injected as environment variables
6. Outputs the live HTTPS URL

Tearing down

# Delete everything — stops all billing immediately
az group delete --name irminsul-rg --yes --no-wait

Cost breakdown (Groq backend, no GPU)

Resource	SKU	Cost
Container Apps	Consumption plan	Free (180k vCPU-s/month)
ACR	Basic	~$5/month
Outbound bandwidth	First 100GB	Free
Total		~$5/month

On Azure for Students ($100 credit), this runs for ~20 months.

Why not GPU on Azure?

To serve the fine-tuned Llama model in production, a GPU instance is required:

Instance	GPU	VRAM	Cost
NC4as T4 v3	Tesla T4	16 GB	$0.50/hr = **$360/month**
NC6s v3	Tesla V100	16 GB	$0.90/hr = **$648/month**

At these prices, a portfolio project running 24/7 would exhaust the $100 student credit in under a week. The Groq backend delivers the same RAG functionality at zero marginal cost, making it the right engineering tradeoff.

Serving the fine-tuned model on Azure (production path)

If cost were not a constraint, the correct architecture is:

1. Upload model to Azure Blob Storage (~$0.02/GB/month for 16GB = ~$0.32/month)
2. Mount as a volume in Container Apps — the container sees it at /app/models/
3. Switch to GPU SKU — replace --cpu 1.0 --memory 2.0Gi in deploy_azure.sh with a GPU-enabled workload profile
4. Set LLM_BACKEND=local in env vars

The Docker image and application code require zero changes for this path. The abstraction was designed for it.

---

Environment Variables Reference

Variable	Required	Default	Description
PINECONE_API_KEY	Yes	—	Pinecone serverless API key
PINECONE_INDEX	No	llmops-rag	Pinecone index name
LLM_BACKEND	No	groq	groq or local
GROQ_API_KEY	If Groq	—	Groq API key
GROQ_MODEL	No	llama-3.3-70b-versatile	Groq model name
MODEL_PATH	If local	./models/merged/exp2_lr2e-4_r16	Path to merged model
EMBED_MODEL	No	sentence-transformers/all-MiniLM-L6-v2	Embedding model

---

CI/CD (Planned)

The intended CI/CD pipeline:

git push main
    │
    ▼
GitHub Actions
    ├── Run tests
    ├── Build Docker image
    ├── Push to ACR
    └── az containerapp update --image new-tag

This would give zero-downtime rolling deploys on every push to main. Currently, re-running deploy_azure.sh achieves the same result with a cold start.