--- title: Doc Ingestion RAG Demo emoji: 📚 colorFrom: blue colorTo: indigo sdk: streamlit sdk_version: "1.37.0" app_file: spaces/app.py pinned: false license: mit --- # Doc-Ingestion Doc-Ingestion is a citation-aware RAG system that turns private document collections into grounded question-answering experiences. It demonstrates how to ingest documents, retrieve the right evidence, generate answers from that evidence, and return citations plus truthfulness signals through a Streamlit app, FastAPI service, and CLI. > **[Try the live demo on Hugging Face Spaces](https://huggingface.co/spaces/vampokala/doc-ingestion)** - no install required. ## Why This Project Exists Most teams have knowledge scattered across PDFs, Word docs, markdown notes, text files, and HTML exports. Traditional search can find matching words, but it does not synthesize answers. Generic LLMs can synthesize answers, but they may not know what is inside your documents and can hallucinate without evidence. This project solves that gap: ingest your documents, ask natural-language questions, and receive answers grounded in retrieved source chunks with citations and quality signals. ## What It Showcases For non-technical reviewers, this is a working document Q&A product: load documents, ask questions, inspect answers, and verify sources. For technical reviewers, this is an end-to-end RAG reference implementation with: - Multi-format ingestion for `.pdf`, `.docx`, `.txt`, `.md`, and `.html` - Token-aware chunking and persistent document indexes - Hybrid retrieval using BM25 keyword search plus vector search - Weighted Reciprocal Rank Fusion (RRF) across sparse and dense results - Optional cross-encoder reranking for stronger final context - Multi-provider LLM routing across Ollama, OpenAI, Anthropic, and Gemini - Citation tracking, citation verification, and inline truthfulness scoring - FastAPI, Streamlit, CLI, Docker, Redis-backed rate limiting, and offline evals ## Product Capabilities This is the user-facing flow: documents become a searchable knowledge base, and users ask questions against that knowledge base instead of relying on ungrounded model memory. ```mermaid flowchart LR subgraph userLayer [User Experience] upload[Upload Or Select Documents] ask[Ask Natural Language Questions] review[Review Answer With Citations] end subgraph knowledgeLayer [Knowledge Base] ingest[Ingest Documents] stored[Documents Stored And Indexed] end subgraph outcomeLayer [Business Outcome] grounded[Grounded RAG Answer] citations[Source Citations] trust[Truthfulness Signal] end upload --> ingest --> stored stored --> ask ask --> grounded grounded --> citations grounded --> trust citations --> review trust --> review ``` ## Under The Hood The technical pipeline combines ingestion, sparse retrieval, semantic retrieval, rank fusion, reranking, model routing, citation verification, and answer scoring. ```mermaid flowchart TB subgraph ingestionLayer [Ingestion Layer] direction LR documents[Documents] ingest[Ingest] chunk[Chunk] embed[Create Embeddings] vectorStore[Chroma Or Qdrant] keywordIndex[BM25 Keyword Index] end subgraph retrievalLayer [Retrieval Layer] direction LR query[User Query] keyword[Keyword Retrieval] semantic[Semantic Retrieval] rrf[Weighted RRF Fusion] rerank[Cross Encoder Rerank] end subgraph generationLayer [Generation And Trust Layer] direction LR context[Context Optimizer] aggregator[LLM Provider Aggregator] answer[Answer] citations[Citation Verification] truth[Truthfulness Score] end documents --> ingest ingest --> chunk chunk --> embed --> vectorStore chunk --> keywordIndex query --> keyword query --> semantic keywordIndex --> keyword vectorStore --> semantic keyword --> rrf semantic --> rrf rrf --> rerank --> context --> aggregator --> answer answer --> citations --> truth ``` ## How Answer Quality Is Protected Doc-Ingestion is designed around a grounding contract: retrieve evidence first, generate from that evidence, then report how well the answer is supported. - **Hybrid retrieval:** BM25 catches exact terms, acronyms, and names; vector search catches semantic matches. The results are fused with weighted RRF in [`src/core/hybrid_retriever.py`](src/core/hybrid_retriever.py). - **Reranking:** A cross-encoder reranker narrows the final context before generation in [`src/core/reranker.py`](src/core/reranker.py). - **Context control:** Retrieved chunks are packed into the prompt within a configured token budget in [`src/core/context_optimizer.py`](src/core/context_optimizer.py). - **Provider routing:** The same query path can route to Ollama, OpenAI, Anthropic, or Gemini through [`src/core/llm_provider.py`](src/core/llm_provider.py). - **Citations:** Generated citation markers are mapped back to retrieved chunks by [`src/core/citation_tracker.py`](src/core/citation_tracker.py) and verified by [`src/core/citation_verifier.py`](src/core/citation_verifier.py). - **Truthfulness:** Each response can include NLI faithfulness and citation groundedness from [`src/evaluation/truthfulness.py`](src/evaluation/truthfulness.py). ## What You Can Try - Use the hosted [Hugging Face Spaces demo](https://huggingface.co/spaces/vampokala/doc-ingestion) with preloaded sample documents. - Upload or ingest your own files locally. - Ask questions through Streamlit, FastAPI, or the CLI. - Inspect answers, citations, source evidence, and truthfulness scores. - Switch LLM providers and models per request when credentials are configured. In hosted demo mode (`DOC_PROFILE=demo`), Streamlit executes queries in-process through the shared orchestrator so the demo is not blocked by localhost API startup races. Local non-demo mode uses the standard split architecture where Streamlit calls FastAPI over HTTP. ## Tech Stack Snapshot - **App and API:** Streamlit, FastAPI, Pydantic, Uvicorn - **Document processing:** PyPDF2, python-docx, BeautifulSoup, markdown parsing, token-aware chunking - **Retrieval:** BM25, Chroma, Qdrant, sentence-transformers, Ollama embeddings - **Ranking:** weighted RRF fusion, `cross-encoder/ms-marco-MiniLM-L-6-v2` - **Generation:** Ollama, OpenAI, Anthropic, Gemini - **Evaluation:** NLI faithfulness, citation groundedness, golden datasets, RAGAS-style offline harness - **Operations:** Docker Compose, Redis-backed rate limiting with in-memory fallback, Hugging Face Spaces deployment ## Quickstart ### Try Online Open the [Hugging Face Spaces demo](https://huggingface.co/spaces/vampokala/doc-ingestion). Sample documents about RAG, vector databases, and BM25 are preloaded. Paste your OpenAI, Anthropic, or Gemini key in the sidebar if you want to use a cloud provider. ### Run Locally With Docker ```bash git clone https://github.com/vampokala/Doc-Ingestion cd Doc-Ingestion cp docker/.env.example docker/.env # Edit docker/.env to add your API keys if needed. docker compose -f docker/docker-compose.yml up ``` Open `http://localhost:8501` for Streamlit or `http://localhost:8000` for the API. ### Run From Source ```bash git clone https://github.com/vampokala/Doc-Ingestion cd Doc-Ingestion bash scripts/bootstrap_demo.sh ``` The bootstrap script creates a virtual environment, installs dependencies, ingests sample documents, and pulls Ollama models when Ollama is installed. ```bash source .venv/bin/activate # API server PYTHONPATH=. uvicorn src.api.main:app --reload --port 8000 # Streamlit UI in a second terminal PYTHONPATH=. streamlit run src/web/streamlit_app.py # CLI query PYTHONPATH=. python -m src.query "What is RAG?" ``` For a full local and Docker runbook, see [`Docs/RUNBOOK.md`](Docs/RUNBOOK.md). ## API Usage ```bash uvicorn src.api.main:app --reload --port 8000 ``` ```bash curl -X POST http://127.0.0.1:8000/query \ -H "X-API-Key: dev-key-1" \ -H "Content-Type: application/json" \ -d '{"query": "What is hybrid retrieval?", "provider": "ollama", "model": "qwen2.5:7b"}' ``` Response includes answer text, citations, retrieved evidence, and a `truthfulness` block: ```json { "answer": "Hybrid retrieval combines BM25 sparse search with dense vector search...", "truthfulness": { "nli_faithfulness": 0.87, "citation_groundedness": 0.91, "uncited_claims": 1, "score": 0.89 }, "citations": [] } ``` Endpoints: `GET /health`, `GET /metrics`, `POST /query`, `POST /query/stream` (SSE). ## Evaluation Every `/query` response can include a `truthfulness` object: | Field | What it measures | |-------|------------------| | `nli_faithfulness` | Fraction of response sentences entailed by retrieved chunks | | `citation_groundedness` | Mean citation verification score | | `uncited_claims` | Count of answer sentences without citation markers | | `score` | Weighted aggregate of faithfulness and groundedness | Run the offline harness against the included datasets: ```bash pip install -r requirements/eval.txt PYTHONPATH=. python -m evals.run_evals \ --dataset evals/datasets/golden.jsonl \ --judge-provider anthropic \ --judge-model claude-haiku-4-5 \ --output evals/reports/ PYTHONPATH=. python -m evals.run_evals \ --dataset evals/datasets/smoke.jsonl \ --mock \ --no-nli \ --output evals/reports/ ``` Reports are written to `evals/reports/` as JSON and Markdown. ## Project Map - [`src/core/`](src/core/) - retrieval, reranking, generation, citations, orchestration - [`src/api/`](src/api/) - FastAPI models and routes - [`src/web/`](src/web/) - Streamlit UI and ingestion service - [`src/evaluation/`](src/evaluation/) - truthfulness scorer, generation metrics, retrieval metrics - [`src/utils/`](src/utils/) - config, logging, and vector database integrations - [`evals/`](evals/) - offline eval harness, golden datasets, RAGAS adapter - [`data/sample/`](data/sample/) - preloaded sample documents for demos - [`spaces/`](spaces/) - Hugging Face Spaces deployment files - [`docker/`](docker/) - Docker Compose stack for API, Streamlit, Redis, and Qdrant - [`Docs/`](Docs/) - architecture notes, runbook, roadmap, phase documentation ## Where To Go Deeper - [`Docs/PROJECT_OVERVIEW.md`](Docs/PROJECT_OVERVIEW.md) - system architecture and reader-friendly project overview - [`Docs/RUNBOOK.md`](Docs/RUNBOOK.md) - local setup, Docker setup, API keys, rate limiting, troubleshooting - [`Docs/phase2_hybrid_retrieval.md`](Docs/phase2_hybrid_retrieval.md) - hybrid retrieval and RRF design - [`Docs/phase3_reranking_generation.md`](Docs/phase3_reranking_generation.md) - reranking, generation, and context optimization - [`Docs/phase4_citation_api.md`](Docs/phase4_citation_api.md) - citation and API design - [`Docs/performance_baseline.md`](Docs/performance_baseline.md) - FastAPI overhead baseline - [`Docs/ROADMAP.md`](Docs/ROADMAP.md) - delivery status and planned improvements ## Development ```bash .venv/bin/python -m pytest tests/unit -q .venv/bin/python -m pytest tests/integration -q ``` Multi-provider API key environment variables: ```bash export OPENAI_API_KEY=... export ANTHROPIC_API_KEY=... export GEMINI_API_KEY=... export DOC_API_KEYS=dev-key-1 ``` ## Troubleshooting - **Empty results after ingest:** Run `python -m src.ingest --docs data/documents` and verify `data/embeddings/` exists. - **Embedding model error:** Ensure Ollama is running and `nomic-embed-text` is pulled, or switch to a different embedding provider in `config.yaml`. - **Dimension mismatch after model change:** Re-ingest all documents to rebuild the vector index. - **Cloud provider fails:** Check the relevant `*_API_KEY` env var is set. - **Truthfulness score always 0:** The NLI model (`cross-encoder/nli-deberta-v3-small`) downloads on first use. Check internet access or set `evaluation.inline_enabled: false` in `config.yaml` to disable.