README.md

---
title: Doc Ingestion RAG Demo
emoji: 📚
colorFrom: blue
colorTo: indigo
sdk: streamlit
sdk_version: "1.37.0"
app_file: spaces/app.py
pinned: false
license: mit
---

# Doc-Ingestion

Doc-Ingestion is a citation-aware RAG system that turns private document collections into grounded question-answering experiences. It demonstrates how to ingest documents, retrieve the right evidence, generate answers from that evidence, and return citations plus truthfulness signals through a Streamlit app, FastAPI service, and CLI.

> **[Try the live demo on Hugging Face Spaces](https://huggingface.co/spaces/vampokala/doc-ingestion)** - no install required.

## Why This Project Exists

Most teams have knowledge scattered across PDFs, Word docs, markdown notes, text files, and HTML exports. Traditional search can find matching words, but it does not synthesize answers. Generic LLMs can synthesize answers, but they may not know what is inside your documents and can hallucinate without evidence.

This project solves that gap: ingest your documents, ask natural-language questions, and receive answers grounded in retrieved source chunks with citations and quality signals.

## What It Showcases

For non-technical reviewers, this is a working document Q&A product: load documents, ask questions, inspect answers, and verify sources.

For technical reviewers, this is an end-to-end RAG reference implementation with:

- Multi-format ingestion for `.pdf`, `.docx`, `.txt`, `.md`, and `.html`
- Token-aware chunking and persistent document indexes
- Hybrid retrieval using BM25 keyword search plus vector search
- Weighted Reciprocal Rank Fusion (RRF) across sparse and dense results
- Optional cross-encoder reranking for stronger final context
- Multi-provider LLM routing across Ollama, OpenAI, Anthropic, and Gemini
- Citation tracking, citation verification, and inline truthfulness scoring
- FastAPI, Streamlit, CLI, Docker, Redis-backed rate limiting, and offline evals

## Product Capabilities

This is the user-facing flow: documents become a searchable knowledge base, and users ask questions against that knowledge base instead of relying on ungrounded model memory.

```mermaid
flowchart LR
  subgraph userLayer [User Experience]
    upload[Upload Or Select Documents]
    ask[Ask Natural Language Questions]
    review[Review Answer With Citations]
  end

  subgraph knowledgeLayer [Knowledge Base]
    ingest[Ingest Documents]
    stored[Documents Stored And Indexed]
  end

  subgraph outcomeLayer [Business Outcome]
    grounded[Grounded RAG Answer]
    citations[Source Citations]
    trust[Truthfulness Signal]
  end

  upload --> ingest --> stored
  stored --> ask
  ask --> grounded
  grounded --> citations
  grounded --> trust
  citations --> review
  trust --> review
```

## Under The Hood

The technical pipeline combines ingestion, sparse retrieval, semantic retrieval, rank fusion, reranking, model routing, citation verification, and answer scoring.

```mermaid
flowchart TB
  subgraph ingestionLayer [Ingestion Layer]
    direction LR
    documents[Documents]
    ingest[Ingest]
    chunk[Chunk]
    embed[Create Embeddings]
    vectorStore[Chroma Or Qdrant]
    keywordIndex[BM25 Keyword Index]
  end

  subgraph retrievalLayer [Retrieval Layer]
    direction LR
    query[User Query]
    keyword[Keyword Retrieval]
    semantic[Semantic Retrieval]
    rrf[Weighted RRF Fusion]
    rerank[Cross Encoder Rerank]
  end

  subgraph generationLayer [Generation And Trust Layer]
    direction LR
    context[Context Optimizer]
    aggregator[LLM Provider Aggregator]
    answer[Answer]
    citations[Citation Verification]
    truth[Truthfulness Score]
  end

  documents --> ingest
  ingest --> chunk
  chunk --> embed --> vectorStore
  chunk --> keywordIndex

  query --> keyword
  query --> semantic
  keywordIndex --> keyword
  vectorStore --> semantic
  keyword --> rrf
  semantic --> rrf

  rrf --> rerank --> context --> aggregator --> answer
  answer --> citations --> truth
```

## How Answer Quality Is Protected

Doc-Ingestion is designed around a grounding contract: retrieve evidence first, generate from that evidence, then report how well the answer is supported.

- **Hybrid retrieval:** BM25 catches exact terms, acronyms, and names; vector search catches semantic matches. The results are fused with weighted RRF in [`src/core/hybrid_retriever.py`](src/core/hybrid_retriever.py).
- **Reranking:** A cross-encoder reranker narrows the final context before generation in [`src/core/reranker.py`](src/core/reranker.py).
- **Context control:** Retrieved chunks are packed into the prompt within a configured token budget in [`src/core/context_optimizer.py`](src/core/context_optimizer.py).
- **Provider routing:** The same query path can route to Ollama, OpenAI, Anthropic, or Gemini through [`src/core/llm_provider.py`](src/core/llm_provider.py).
- **Citations:** Generated citation markers are mapped back to retrieved chunks by [`src/core/citation_tracker.py`](src/core/citation_tracker.py) and verified by [`src/core/citation_verifier.py`](src/core/citation_verifier.py).
- **Truthfulness:** Each response can include NLI faithfulness and citation groundedness from [`src/evaluation/truthfulness.py`](src/evaluation/truthfulness.py).

## What You Can Try

- Use the hosted [Hugging Face Spaces demo](https://huggingface.co/spaces/vampokala/doc-ingestion) with preloaded sample documents.
- Upload or ingest your own files locally.
- Ask questions through Streamlit, FastAPI, or the CLI.
- Inspect answers, citations, source evidence, and truthfulness scores.
- Switch LLM providers and models per request when credentials are configured.

In hosted demo mode (`DOC_PROFILE=demo`), Streamlit executes queries in-process through the shared orchestrator so the demo is not blocked by localhost API startup races. Local non-demo mode uses the standard split architecture where Streamlit calls FastAPI over HTTP.

## Tech Stack Snapshot

- **App and API:** Streamlit, FastAPI, Pydantic, Uvicorn
- **Document processing:** PyPDF2, python-docx, BeautifulSoup, markdown parsing, token-aware chunking
- **Retrieval:** BM25, Chroma, Qdrant, sentence-transformers, Ollama embeddings
- **Ranking:** weighted RRF fusion, `cross-encoder/ms-marco-MiniLM-L-6-v2`
- **Generation:** Ollama, OpenAI, Anthropic, Gemini
- **Evaluation:** NLI faithfulness, citation groundedness, golden datasets, RAGAS-style offline harness
- **Operations:** Docker Compose, Redis-backed rate limiting with in-memory fallback, Hugging Face Spaces deployment

## Quickstart

### Try Online

Open the [Hugging Face Spaces demo](https://huggingface.co/spaces/vampokala/doc-ingestion). Sample documents about RAG, vector databases, and BM25 are preloaded. Paste your OpenAI, Anthropic, or Gemini key in the sidebar if you want to use a cloud provider.

### Run Locally With Docker

```bash
git clone https://github.com/vampokala/Doc-Ingestion
cd Doc-Ingestion
cp docker/.env.example docker/.env
# Edit docker/.env to add your API keys if needed.
docker compose -f docker/docker-compose.yml up
```

Open `http://localhost:8501` for Streamlit or `http://localhost:8000` for the API.

### Run From Source

```bash
git clone https://github.com/vampokala/Doc-Ingestion
cd Doc-Ingestion
bash scripts/bootstrap_demo.sh
```

The bootstrap script creates a virtual environment, installs dependencies, ingests sample documents, and pulls Ollama models when Ollama is installed.

```bash
source .venv/bin/activate

# API server
PYTHONPATH=. uvicorn src.api.main:app --reload --port 8000

# Streamlit UI in a second terminal
PYTHONPATH=. streamlit run src/web/streamlit_app.py

# CLI query
PYTHONPATH=. python -m src.query "What is RAG?"
```

For a full local and Docker runbook, see [`Docs/RUNBOOK.md`](Docs/RUNBOOK.md).

## API Usage

```bash
uvicorn src.api.main:app --reload --port 8000
```

```bash
curl -X POST http://127.0.0.1:8000/query \
  -H "X-API-Key: dev-key-1" \
  -H "Content-Type: application/json" \
  -d '{"query": "What is hybrid retrieval?", "provider": "ollama", "model": "qwen2.5:7b"}'
```

Response includes answer text, citations, retrieved evidence, and a `truthfulness` block:

```json
{
  "answer": "Hybrid retrieval combines BM25 sparse search with dense vector search...",
  "truthfulness": {
    "nli_faithfulness": 0.87,
    "citation_groundedness": 0.91,
    "uncited_claims": 1,
    "score": 0.89
  },
  "citations": []
}
```

Endpoints: `GET /health`, `GET /metrics`, `POST /query`, `POST /query/stream` (SSE).

## Evaluation

Every `/query` response can include a `truthfulness` object:

| Field | What it measures |
|-------|------------------|
| `nli_faithfulness` | Fraction of response sentences entailed by retrieved chunks |
| `citation_groundedness` | Mean citation verification score |
| `uncited_claims` | Count of answer sentences without citation markers |
| `score` | Weighted aggregate of faithfulness and groundedness |

Run the offline harness against the included datasets:

```bash
pip install -r requirements/eval.txt

PYTHONPATH=. python -m evals.run_evals \
  --dataset evals/datasets/golden.jsonl \
  --judge-provider anthropic \
  --judge-model claude-haiku-4-5 \
  --output evals/reports/

PYTHONPATH=. python -m evals.run_evals \
  --dataset evals/datasets/smoke.jsonl \
  --mock \
  --no-nli \
  --output evals/reports/
```

Reports are written to `evals/reports/` as JSON and Markdown.

## Project Map

- [`src/core/`](src/core/) - retrieval, reranking, generation, citations, orchestration
- [`src/api/`](src/api/) - FastAPI models and routes
- [`src/web/`](src/web/) - Streamlit UI and ingestion service
- [`src/evaluation/`](src/evaluation/) - truthfulness scorer, generation metrics, retrieval metrics
- [`src/utils/`](src/utils/) - config, logging, and vector database integrations
- [`evals/`](evals/) - offline eval harness, golden datasets, RAGAS adapter
- [`data/sample/`](data/sample/) - preloaded sample documents for demos
- [`spaces/`](spaces/) - Hugging Face Spaces deployment files
- [`docker/`](docker/) - Docker Compose stack for API, Streamlit, Redis, and Qdrant
- [`Docs/`](Docs/) - architecture notes, runbook, roadmap, phase documentation

## Where To Go Deeper

- [`Docs/PROJECT_OVERVIEW.md`](Docs/PROJECT_OVERVIEW.md) - system architecture and reader-friendly project overview
- [`Docs/RUNBOOK.md`](Docs/RUNBOOK.md) - local setup, Docker setup, API keys, rate limiting, troubleshooting
- [`Docs/phase2_hybrid_retrieval.md`](Docs/phase2_hybrid_retrieval.md) - hybrid retrieval and RRF design
- [`Docs/phase3_reranking_generation.md`](Docs/phase3_reranking_generation.md) - reranking, generation, and context optimization
- [`Docs/phase4_citation_api.md`](Docs/phase4_citation_api.md) - citation and API design
- [`Docs/performance_baseline.md`](Docs/performance_baseline.md) - FastAPI overhead baseline
- [`Docs/ROADMAP.md`](Docs/ROADMAP.md) - delivery status and planned improvements

## Development

```bash
.venv/bin/python -m pytest tests/unit -q
.venv/bin/python -m pytest tests/integration -q
```

Multi-provider API key environment variables:

```bash
export OPENAI_API_KEY=...
export ANTHROPIC_API_KEY=...
export GEMINI_API_KEY=...
export DOC_API_KEYS=dev-key-1
```

## Troubleshooting

- **Empty results after ingest:** Run `python -m src.ingest --docs data/documents` and verify `data/embeddings/` exists.
- **Embedding model error:** Ensure Ollama is running and `nomic-embed-text` is pulled, or switch to a different embedding provider in `config.yaml`.
- **Dimension mismatch after model change:** Re-ingest all documents to rebuild the vector index.
- **Cloud provider fails:** Check the relevant `*_API_KEY` env var is set.
- **Truthfulness score always 0:** The NLI model (`cross-encoder/nli-deberta-v3-small`) downloads on first use. Check internet access or set `evaluation.inline_enabled: false` in `config.yaml` to disable.