Spaces:

MukulRay
/

Irminsul

Sleeping

App Files Files Community

Irminsul / README.md

MukulRay

docs: overhaul README, add DEPLOYMENT.md, deploy_azure.sh, .env.example

3fbe97b 4 months ago

16.7 kB

title: Irminsul
colorFrom: green
colorTo: green
sdk: docker
pinned: false

Irminsul

A production-shaped LLMOps stack — QLoRA fine-tuning on Colab, RAG pipeline, containerized serving, and cloud deployment.

Most LLM projects stop at inference. This one builds the full stack: a QLoRA fine-tuned Llama 3.1 8B served through a RAG pipeline, with guardrails, a domain-specific knowledge base, and a containerized FastAPI server designed for cloud deployment.

→ Try the live demo

What This Is

Irminsul is a domain-specific AI assistant for Genshin Impact — built not because Genshin needed an AI assistant, but because it provided a concrete, evaluable knowledge domain to build an LLMOps pipeline around. Every component was chosen deliberately:

A knowledge domain rich enough to evaluate retrieval quality (characters, mechanics, lore)
Ground truth data available (KQM Theorycrafting Library, game stat APIs) to measure hallucination
Community signal data (patch notes, meta shifts) to test corpus freshness

The domain is the test harness. The pipeline is the project.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│                         User Query                              │
└─────────────────────┬───────────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────────┐
│                    Guardrails Layer                             │
│  • Injection detection (pattern matching)                       │
│  • Domain validation (cosine similarity vs anchor embeddings)   │
│  • Output sanitization                                          │
└─────────────────────┬───────────────────────────────────────────┘
                      │
                      ▼
┌─────────────────────────────────────────────────────────────────┐
│                  FastAPI /generate                               │
└──────────┬──────────────────────────────────────────────────────┘
           │
           ├─── Embed query (sentence-transformers, local, CPU)
           │              │
           │              ▼
           │         Pinecone ── semantic search ──► top-k chunks
           │              │
           ▼              ▼
    LangChain RetrievalQA (stuff chain)
           │
           ▼
  ┌────────────────────────────────────┐
  │         LLM Backend                │
  │                                    │
  │  Groq (live demo)                  │
  │  llama-3.3-70b-versatile           │
  │  ~300 tok/s, free tier             │
  │             ──── OR ────           │
  │  Local (fine-tuned)                │
  │  Llama 3.1 8B QLoRA               │
  │  4-bit NF4, RTX 3060 6GB          │
  │  (inference only — trained on     │
  │   Colab A100)                     │
  └────────────────────────────────────┘
           │
           ▼
    Grounded answer + source attribution

Components

Fine-Tuned Model

Llama 3.1 8B Instruct fine-tuned with QLoRA on a custom instruction dataset, trained on Google Colab Pro (A100). Local inference runs in 4-bit NF4 quantization on an RTX 3060 6GB.

→ View training notebook on Colab

Parameter	Value
Base model	`meta-llama/Llama-3.1-8B-Instruct`
Method	QLoRA via PEFT
Rank / Alpha	r=16, α=32
Learning rate	2e-4
Quantization (inference)	4-bit NF4, bfloat16 compute
Training infra	Google Colab Pro (A100)
Experiment tracking	MLflow (3 runs)

Three experiments were tracked in MLflow. Winning checkpoint selected by faithfulness score (0.826) and ROUGE-L (0.466) on a held-out eval set.

RAG Pipeline

Documents are chunked, embedded locally with sentence-transformers/all-MiniLM-L6-v2 (384-dim, zero API cost), and stored in Pinecone serverless. Retrieval is semantic, top-k configurable per query.

Component	Choice	Reason
Embedder	all-MiniLM-L6-v2	Runs locally, strong semantic retrieval, 384-dim fits free Pinecone tier
Vector DB	Pinecone serverless	Zero ops, cosine similarity, free tier sufficient for corpus size
Chunking	Word-level, 300 words, 40-word overlap	Preserves semantic units across chunk boundaries
Chain	LangChain RetrievalQA (stuff)	Simple, inspectable, returns source documents

Knowledge Corpus

Corpus is maintained in a separate repository with an autonomous update pipeline. It ingests from three tiers of sources with different trust levels:

Tier	Source	Files	Trust
1 — Ground Truth	KQM Theorycrafting Library (peer-reviewed mechanics)	~305	Highest — cite in builds
1 — Ground Truth	genshin-db API (exact character/weapon/artifact stats)	~406	Highest — exact game data
2 — Expert Synthesis	Gemini-authored prose grounded in Tier 1	~83	High — no hallucinated stats
3 — Community Signal	Official patch notes, banner history, event calendar	~80	Medium — tagged explicitly

A GitHub Actions workflow runs every Sunday at 2am UTC, pulls fresh data, commits the docs, and re-ingests ~4,000 vectors to Pinecone automatically.

Guardrails

Two layers of input validation before any LLM call:

Injection detection — pattern matching against known jailbreak phrases (ignore previous instructions, act as, DAN mode, etc.)
Domain validation — cosine similarity between the query embedding and a set of Genshin-domain anchor sentences. Queries scoring below threshold (0.35) are rejected with a domain-scoped error message before touching the LLM.

Output is sanitized to strip generation artifacts (</s> tokens, trailing whitespace) and length-checked.

Serving Layer

FastAPI with:

Async lifespan model loading (model loads once at startup, not per request)
Typed Pydantic request/response models with blocked flag for guardrail rejections
CORS enabled for cross-origin UI
/health endpoint reporting model load status
Browser UI served from the same process (no separate frontend server)

Stack

Layer	Technology
Base model	Llama 3.1 8B Instruct
Fine-tuning	QLoRA via PEFT (r=16, α=32, lr=2e-4)
Experiment tracking	MLflow
Quantization	BitsAndBytes 4-bit NF4, bfloat16 compute
Embeddings	sentence-transformers/all-MiniLM-L6-v2
Vector DB	Pinecone serverless (cosine, 384-dim)
RAG chain	LangChain RetrievalQA
Serving	FastAPI + Uvicorn
Containerization	Docker (python:3.12-slim)
Live demo hosting	HuggingFace Spaces (CPU Basic)
Production deployment	Azure Container Apps + ACR
LLM backend (demo)	Groq API (llama-3.3-70b-versatile)
Corpus pipeline	GitHub Actions (weekly, autonomous)

Quickstart

Option 1 — Groq backend (no GPU required)

# 1. Clone
git clone https://github.com/MukulRay1603/Irminsul.git
cd Irminsul

# 2. Install
python -m venv venv && source venv/bin/activate
# Windows: venv\Scripts\activate
pip install -r requirements.txt

# 3. Configure
cp .env.example .env
# Set PINECONE_API_KEY and GROQ_API_KEY in .env
# LLM_BACKEND=groq is the default

# 4. Ingest corpus (or use pre-ingested Pinecone index)
python ingest.py --dir ./docs --chunk-size 300 --chunk-overlap 40

# 5. Run
uvicorn main:app --reload --port 8000
# UI: http://localhost:8000
# API docs: http://localhost:8000/docs

Option 2 — Local fine-tuned model (GPU required for inference, 6GB+ VRAM)

# Same steps 1–3, then:
# Set LLM_BACKEND=local and MODEL_PATH in .env

# 4. Download model
# Place the merged QLoRA model at: ./models/merged/exp2_lr2e-4_r16/
# (Or update MODEL_PATH in .env)

# 5. Run
uvicorn main:app --reload --port 8000

Docker

# Groq backend (no GPU)
docker build -t irminsul:latest .
docker run -p 8000:8000 \
  -e PINECONE_API_KEY=your_key \
  -e GROQ_API_KEY=your_key \
  -e LLM_BACKEND=groq \
  irminsul:latest

API

Method	Endpoint	Description
`GET`	`/`	Browser UI
`GET`	`/health`	Model load status + ready flag
`POST`	`/generate`	RAG query → grounded answer + sources
`POST`	`/ingest`	Ingest docs from a local directory path

Request:

{
  "query": "What weapons should Hu Tao use on a budget?",
  "top_k": 3
}

Response:

{
  "answer": "For Hu Tao on a budget, Dragon's Bane is the strongest F2P option — it scales with Elemental Mastery and deals significant bonus damage on vaporized hits. White Tassel is the best 3-star alternative for pure Normal Attack scaling.",
  "sources": ["docs/generated/characters/hu_tao.md", "docs/tcl/characters/pyro/hutao.md"],
  "latency_ms": 1240.5,
  "blocked": false
}

If a query is rejected by guardrails, blocked: true is returned with the rejection reason in answer. No LLM call is made.

Deployment

See DEPLOYMENT.md for the full guide covering:

Local development setup
Docker (local and cloud)
Azure Container Apps (one-shot deploy_azure.sh)
Cost breakdown and the reasoning behind the demo setup
GPU serving path for the fine-tuned model

Why the live demo runs on HuggingFace + Groq, not Azure GPU:

Serving the fine-tuned Llama 3.1 8B requires a GPU instance. The minimum viable option on Azure (NC4as T4 v3) costs ~$360/month — not justified for a portfolio project. The Dockerfile and deploy_azure.sh are written for the Azure path; the live demo swaps the LLM backend to Groq via a single environment variable. The RAG pipeline, guardrails, and serving layer are identical.

Project Structure

Irminsul/
├── main.py              # FastAPI app: endpoints, lifespan, CORS, response models
├── rag.py               # LangChain RAG chain, dual backend (Groq / local Llama)
├── embedder.py          # sentence-transformers singleton (loads once, reused)
├── ingest.py            # Doc loader → word chunker → Pinecone upsert
├── guardrails.py        # Input validation: injection detection + domain cosine check
├── index.html           # Browser UI: dark Dendro theme, query history, source display
│
├── Dockerfile           # python:3.12-slim, model NOT baked in
├── deploy_azure.sh      # One-shot ACR build + Container Apps deploy
├── .env.example         # Environment variable reference
│
├── DEPLOYMENT.md        # Full deployment guide + cost analysis
├── requirements.txt
├── images/              # Screenshots and assets used in this README
│   ├── banner.png
│   ├── ui_main.png
│   ├── ui_response.png
│   └── mlflow_runs.png
└── docs/
    ├── corpus/          # Legacy manual corpus docs
    └── demo.html        # GitHub Pages demo page

Evaluation

The fine-tuned model was evaluated during training with a held-out set:

Metric	Score
Faithfulness	0.826
ROUGE-L	0.466

Full RAG pipeline evaluation (context recall, answer relevance) is a planned addition — see What's Next.

Screenshots

Screenshots coming soon — try the live demo to see it in action.

What's Next

RAGAS evaluation — systematic RAG eval (faithfulness, context recall, answer relevance) on a held-out question set
MarkdownHeaderTextSplitter — replace naive word chunker for section-aware chunking that respects document structure
Metadata filtering — filter Pinecone queries by character, content tier, or topic category
Streaming responses — SSE for lower perceived latency on long answers
CI/CD pipeline — GitHub Actions → ACR build → az containerapp update on push to main
Corpus expansion — constellation effects, rotation guides, and ER/EM thresholds per character

Related: irminsul-corpus

The knowledge base is maintained in a companion repository:

MukulRay1603/irminsul-corpus

It runs a fully autonomous weekly pipeline: pulls fresh game data from the KQM Theorycrafting Library and genshin-db API, synthesizes prose with Gemini 2.5 Flash, commits ~800 documents to the repo, and re-ingests ~4,000 vectors to Pinecone — without any manual intervention.

License

MIT — see LICENSE for details.

Genshin Impact is owned by HoYoverse. This project is not affiliated with or endorsed by HoYoverse.

Built to learn the full MLOps lifecycle — fine-tuning, quantization, retrieval, serving, and cloud deployment — on consumer hardware. Every component chosen deliberately, not for hype.