Redesigned the README and added Citiation and truthfullness in responses

.github/workflows/ci.yml

@@ -108,3 +108,37 @@ jobs:
         with:
           name: eval-report
           path: evals/reports/
+
+  evals-golden:
+    name: Golden evals (Anthropic Haiku)
+    if: ${{ secrets.ANTHROPIC_API_KEY != '' }}
+    runs-on: ubuntu-latest
+    env:
+      ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.13"
+          cache: pip
+
+      - name: Install dependencies
+        run: pip install -r requirements/base.txt
+
+      - name: Run golden evals (live pipeline, Anthropic Haiku)
+        run: |
+          PYTHONPATH=. python -m evals.run_evals \
+            --dataset evals/datasets/golden_ci.jsonl \
+            --judge-provider anthropic \
+            --judge-model claude-haiku-4-5 \
+            --output evals/reports/ \
+            --faithfulness-threshold 0.7 \
+            --correctness-threshold 0.2
+
+      - name: Upload golden eval report
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: eval-report-golden
+          path: evals/reports/

README.md

@@ -12,60 +12,171 @@ license: mit
 
 # Doc-Ingestion
 
-Citation-aware RAG system for ingesting documents and generating grounded answers with truthfulness scores — CLI, API, and Streamlit UI.
+Doc-Ingestion is a citation-aware RAG system that turns private document collections into grounded question-answering experiences. It demonstrates how to ingest documents, retrieve the right evidence, generate answers from that evidence, and return citations plus truthfulness signals through a Streamlit app, FastAPI service, and CLI.
 
-> **[Try the live demo on Hugging Face Spaces](https://huggingface.co/spaces/vampokala/doc-ingestion)** — no install required.
+> **[Try the live demo on Hugging Face Spaces](https://huggingface.co/spaces/vampokala/doc-ingestion)** - no install required.
 
-## What it does
+## Why This Project Exists
 
-- Ingests `.pdf`, `.docx`, `.txt`, `.md`, `.html` files into a hybrid BM25 + vector index
-- Retrieves using Reciprocal Rank Fusion across sparse and dense search, with optional cross-encoder reranking
-- Generates answers via any LLM provider (Ollama, OpenAI, Anthropic, Gemini) with citation tracking
-- Scores every response for **truthfulness** (NLI faithfulness + citation groundedness)
-- Exposes a FastAPI backend and a Streamlit UI
+Most teams have knowledge scattered across PDFs, Word docs, markdown notes, text files, and HTML exports. Traditional search can find matching words, but it does not synthesize answers. Generic LLMs can synthesize answers, but they may not know what is inside your documents and can hallucinate without evidence.
 
----
+This project solves that gap: ingest your documents, ask natural-language questions, and receive answers grounded in retrieved source chunks with citations and quality signals.
 
-## Quickstart
+## What It Showcases
 
-### Option 1 — Try online (no install)
+For non-technical reviewers, this is a working document Q&A product: load documents, ask questions, inspect answers, and verify sources.
 
-Open the [Hugging Face Spaces demo](https://huggingface.co/spaces/vampokala/doc-ingestion). Sample documents about RAG, vector databases, and BM25 are pre-loaded. Paste your OpenAI, Anthropic, or Gemini key in the sidebar.
-In hosted demo mode (`DOC_PROFILE=demo`), Streamlit executes queries in-process through the shared orchestrator so demo usage is not blocked by localhost API startup races.
+For technical reviewers, this is an end-to-end RAG reference implementation with:
 
----
+- Multi-format ingestion for `.pdf`, `.docx`, `.txt`, `.md`, and `.html`
+- Token-aware chunking and persistent document indexes
+- Hybrid retrieval using BM25 keyword search plus vector search
+- Weighted Reciprocal Rank Fusion (RRF) across sparse and dense results
+- Optional cross-encoder reranking for stronger final context
+- Multi-provider LLM routing across Ollama, OpenAI, Anthropic, and Gemini
+- Citation tracking, citation verification, and inline truthfulness scoring
+- FastAPI, Streamlit, CLI, Docker, Redis-backed rate limiting, and offline evals
 
-### Option 2 — Run locally with Docker (one command)
+## Product Capabilities
 
-```bash
-git clone https://github.com/vampokala/Doc-Ingestion
-cd Doc-Ingestion
-cp docker/.env.example docker/.env
-# Edit docker/.env to add your API keys (OPENAI_API_KEY etc.)
-docker compose -f docker/docker-compose.yml up
+This is the user-facing flow: documents become a searchable knowledge base, and users ask questions against that knowledge base instead of relying on ungrounded model memory.
+
+```mermaid
+flowchart LR
+  subgraph userLayer [User Experience]
+    upload[Upload Or Select Documents]
+    ask[Ask Natural Language Questions]
+    review[Review Answer With Citations]
+  end
+
+  subgraph knowledgeLayer [Knowledge Base]
+    ingest[Ingest Documents]
+    stored[Documents Stored And Indexed]
+  end
+
+  subgraph outcomeLayer [Business Outcome]
+    grounded[Grounded RAG Answer]
+    citations[Source Citations]
+    trust[Truthfulness Signal]
+  end
+
+  upload --> ingest --> stored
+  stored --> ask
+  ask --> grounded
+  grounded --> citations
+  grounded --> trust
+  citations --> review
+  trust --> review
 ```
 
-Open http://localhost:8501 (Streamlit UI) or http://localhost:8000 (API).
+## Under The Hood
 
----
+The technical pipeline combines ingestion, sparse retrieval, semantic retrieval, rank fusion, reranking, model routing, citation verification, and answer scoring.
+
+```mermaid
+flowchart TB
+  subgraph ingestionLayer [Ingestion Layer]
+    direction LR
+    documents[Documents]
+    ingest[Ingest]
+    chunk[Chunk]
+    embed[Create Embeddings]
+    vectorStore[Chroma Or Qdrant]
+    keywordIndex[BM25 Keyword Index]
+  end
+
+  subgraph retrievalLayer [Retrieval Layer]
+    direction LR
+    query[User Query]
+    keyword[Keyword Retrieval]
+    semantic[Semantic Retrieval]
+    rrf[Weighted RRF Fusion]
+    rerank[Cross Encoder Rerank]
+  end
+
+  subgraph generationLayer [Generation And Trust Layer]
+    direction LR
+    context[Context Optimizer]
+    aggregator[LLM Provider Aggregator]
+    answer[Answer]
+    citations[Citation Verification]
+    truth[Truthfulness Score]
+  end
+
+  documents --> ingest
+  ingest --> chunk
+  chunk --> embed --> vectorStore
+  chunk --> keywordIndex
+
+  query --> keyword
+  query --> semantic
+  keywordIndex --> keyword
+  vectorStore --> semantic
+  keyword --> rrf
+  semantic --> rrf
+
+  rrf --> rerank --> context --> aggregator --> answer
+  answer --> citations --> truth
+```
+
+## How Answer Quality Is Protected
+
+Doc-Ingestion is designed around a grounding contract: retrieve evidence first, generate from that evidence, then report how well the answer is supported.
 
-### Option 3 — Run from source (Python venv)
+- **Hybrid retrieval:** BM25 catches exact terms, acronyms, and names; vector search catches semantic matches. The results are fused with weighted RRF in [`src/core/hybrid_retriever.py`](src/core/hybrid_retriever.py).
+- **Reranking:** A cross-encoder reranker narrows the final context before generation in [`src/core/reranker.py`](src/core/reranker.py).
+- **Context control:** Retrieved chunks are packed into the prompt within a configured token budget in [`src/core/context_optimizer.py`](src/core/context_optimizer.py).
+- **Provider routing:** The same query path can route to Ollama, OpenAI, Anthropic, or Gemini through [`src/core/llm_provider.py`](src/core/llm_provider.py).
+- **Citations:** Generated citation markers are mapped back to retrieved chunks by [`src/core/citation_tracker.py`](src/core/citation_tracker.py) and verified by [`src/core/citation_verifier.py`](src/core/citation_verifier.py).
+- **Truthfulness:** Each response can include NLI faithfulness and citation groundedness from [`src/evaluation/truthfulness.py`](src/evaluation/truthfulness.py).
+
+## What You Can Try
+
+- Use the hosted [Hugging Face Spaces demo](https://huggingface.co/spaces/vampokala/doc-ingestion) with preloaded sample documents.
+- Upload or ingest your own files locally.
+- Ask questions through Streamlit, FastAPI, or the CLI.
+- Inspect answers, citations, source evidence, and truthfulness scores.
+- Switch LLM providers and models per request when credentials are configured.
+
+In hosted demo mode (`DOC_PROFILE=demo`), Streamlit executes queries in-process through the shared orchestrator so the demo is not blocked by localhost API startup races. Local non-demo mode uses the standard split architecture where Streamlit calls FastAPI over HTTP.
+
+## Tech Stack Snapshot
+
+- **App and API:** Streamlit, FastAPI, Pydantic, Uvicorn
+- **Document processing:** PyPDF2, python-docx, BeautifulSoup, markdown parsing, token-aware chunking
+- **Retrieval:** BM25, Chroma, Qdrant, sentence-transformers, Ollama embeddings
+- **Ranking:** weighted RRF fusion, `cross-encoder/ms-marco-MiniLM-L-6-v2`
+- **Generation:** Ollama, OpenAI, Anthropic, Gemini
+- **Evaluation:** NLI faithfulness, citation groundedness, golden datasets, RAGAS-style offline harness
+- **Operations:** Docker Compose, Redis-backed rate limiting with in-memory fallback, Hugging Face Spaces deployment
+
+## Quickstart
+
+### Try Online
+
+Open the [Hugging Face Spaces demo](https://huggingface.co/spaces/vampokala/doc-ingestion). Sample documents about RAG, vector databases, and BM25 are preloaded. Paste your OpenAI, Anthropic, or Gemini key in the sidebar if you want to use a cloud provider.
+
+### Run Locally With Docker
 
 ```bash
 git clone https://github.com/vampokala/Doc-Ingestion
 cd Doc-Ingestion
-bash scripts/bootstrap_demo.sh   # creates venv, installs deps, ingests sample docs
+cp docker/.env.example docker/.env
+# Edit docker/.env to add your API keys if needed.
+docker compose -f docker/docker-compose.yml up
 ```
 
-The script pulls Ollama models automatically if Ollama is installed.
-To use a cloud provider instead, skip Ollama and set:
+Open `http://localhost:8501` for Streamlit or `http://localhost:8000` for the API.
+
+### Run From Source
 
 ```bash
-export OPENAI_API_KEY=...
-# or ANTHROPIC_API_KEY / GEMINI_API_KEY
+git clone https://github.com/vampokala/Doc-Ingestion
+cd Doc-Ingestion
+bash scripts/bootstrap_demo.sh
 ```
 
-Then start:
+The bootstrap script creates a virtual environment, installs dependencies, ingests sample documents, and pulls Ollama models when Ollama is installed.
 
 ```bash
 source .venv/bin/activate
@@ -73,80 +184,16 @@ source .venv/bin/activate
 # API server
 PYTHONPATH=. uvicorn src.api.main:app --reload --port 8000
 
-# Streamlit UI (second terminal)
+# Streamlit UI in a second terminal
 PYTHONPATH=. streamlit run src/web/streamlit_app.py
 
-# Or query from CLI
+# CLI query
 PYTHONPATH=. python -m src.query "What is RAG?"
 ```
 
-Note: non-demo local mode keeps the standard split architecture (Streamlit calls FastAPI over HTTP), so running both API and UI processes is still required.
+For a full local and Docker runbook, see [`Docs/RUNBOOK.md`](Docs/RUNBOOK.md).
 
----
-
-## Features
-
-- Multi-format ingestion (PDF, DOCX, TXT, MD, HTML)
-- Hybrid retrieval — BM25 + vector with weighted RRF fusion
-- Optional cross-encoder reranking (`cross-encoder/ms-marco-MiniLM-L-6-v2`)
-- Multi-provider LLM routing: Ollama (local), OpenAI, Anthropic, Gemini — switchable per request
-- Citation tracking and per-citation verification
-- **Inline truthfulness scoring** on every response (NLI faithfulness + citation groundedness)
-- **Offline eval harness** — RAGAS-style metrics over a golden dataset
-- FastAPI with auth, rate limiting (Redis or in-memory), streaming SSE
-- Streamlit UI with per-request provider/model switching and truthfulness badge
-
----
-
-## Architecture
-
-<details>
-<summary>System diagram</summary>
-
-```mermaid
-flowchart LR
-  userClient[UserClient] --> cliLayer[CLI]
-  userClient --> apiLayer[FastAPI]
-  userClient --> streamlitUi[StreamlitUI]
-  cliLayer --> orchestrator[RAGOrchestrator]
-  apiLayer --> orchestrator
-  streamlitUi --> orchestrator
-  orchestrator --> hybridRetriever[HybridRetriever]
-  hybridRetriever --> reranker[CrossEncoderReranker]
-  reranker --> contextOptimizer[ContextOptimizer]
-  contextOptimizer --> generator[RAGGenerator]
-  generator --> citationTracker[CitationTracker]
-  citationTracker --> citationVerifier[CitationVerifier]
-  citationVerifier --> truthfulness[TruthfulnessScorer]
-  generator --> llmRouter[LLMProviderRouter]
-  llmRouter --> ollamaProvider[Ollama]
-  llmRouter --> openaiProvider[OpenAI]
-  llmRouter --> anthropicProvider[Claude]
-  llmRouter --> geminiProvider[Gemini]
-  orchestrator --> bm25Store[BM25Index]
-  orchestrator --> vectorStore[ChromaOrQdrant]
-```
-</details>
-
-<details>
-<summary>Query flow</summary>
-
-```mermaid
-flowchart TD
-  q[UserQuery] --> retrieve[HybridRetrieve]
-  retrieve --> fuse[RRFWeightedFusion]
-  fuse --> rerank[RerankOptional]
-  rerank --> prompt[BuildPrompt]
-  prompt --> llm[ProviderModelSelectedPerRequest]
-  llm --> cite[ExtractAndVerifyCitations]
-  cite --> truth[TruthfulnessScore]
-  truth --> response[APIorCLIorUIResponse]
-```
-</details>
-
----
-
-## API usage
+## API Usage
 
 ```bash
 uvicorn src.api.main:app --reload --port 8000
@@ -159,7 +206,7 @@ curl -X POST http://127.0.0.1:8000/query \
   -d '{"query": "What is hybrid retrieval?", "provider": "ollama", "model": "qwen2.5:7b"}'
 ```
 
-Response includes a `truthfulness` block:
+Response includes answer text, citations, retrieved evidence, and a `truthfulness` block:
 
 ```json
 {
@@ -170,45 +217,34 @@ Response includes a `truthfulness` block:
     "uncited_claims": 1,
     "score": 0.89
   },
-  "citations": [...]
+  "citations": []
 }
 ```
 
 Endpoints: `GET /health`, `GET /metrics`, `POST /query`, `POST /query/stream` (SSE).
 
----
-
 ## Evaluation
 
-### Inline (every response)
-
-Every `/query` response includes a `truthfulness` object with:
+Every `/query` response can include a `truthfulness` object:
 
 | Field | What it measures |
-|-------|-----------------|
-| `nli_faithfulness` | Fraction of response sentences entailed by the retrieved chunks (NLI model) |
+|-------|------------------|
+| `nli_faithfulness` | Fraction of response sentences entailed by retrieved chunks |
 | `citation_groundedness` | Mean citation verification score |
-| `uncited_claims` | Count of sentences without a citation marker |
-| `score` | Weighted aggregate (60% NLI + 40% groundedness) |
-
-The Streamlit UI renders a coloured badge: 🟢 ≥ 0.8 / 🟡 0.5–0.8 / 🔴 < 0.5.
+| `uncited_claims` | Count of answer sentences without citation markers |
+| `score` | Weighted aggregate of faithfulness and groundedness |
 
-### Offline batch evaluation
-
-Run the RAGAS-style harness against the included golden dataset:
+Run the offline harness against the included datasets:
 
 ```bash
-# Install eval extras
 pip install -r requirements/eval.txt
 
-# Run against full dataset (needs a running LLM)
 PYTHONPATH=. python -m evals.run_evals \
-  --dataset evals/datasets/sample.jsonl \
-  --judge-provider ollama \
-  --judge-model qwen2.5:7b \
+  --dataset evals/datasets/golden.jsonl \
+  --judge-provider anthropic \
+  --judge-model claude-haiku-4-5 \
   --output evals/reports/
 
-# Smoke test (no LLM required — for CI / quick check)
 PYTHONPATH=. python -m evals.run_evals \
   --dataset evals/datasets/smoke.jsonl \
   --mock \
@@ -216,26 +252,30 @@ PYTHONPATH=. python -m evals.run_evals \
   --output evals/reports/
 ```
 
-Reports are written to `evals/reports/` as JSON + Markdown.
+Reports are written to `evals/reports/` as JSON and Markdown.
 
----
+## Project Map
 
-## Project map
-
-| Path | Purpose |
-|------|---------|
-| `src/core/` | Retrieval, reranking, generation, citations, orchestration |
-| `src/api/` | FastAPI models and routes |
-| `src/web/` | Streamlit UI and ingestion service |
-| `src/evaluation/` | Truthfulness scorer, generation and retrieval metrics |
-| `src/utils/` | Config and vector database integrations |
-| `evals/` | Offline eval harness, golden datasets, RAGAS adapter |
-| `data/sample/` | Pre-ingested sample documents for demos |
-| `spaces/` | Hugging Face Spaces deployment files |
-| `docker/` | Docker Compose stack (API + Streamlit + Redis + Qdrant) |
-| `Docs/` | Roadmap, runbook, and phase documentation |
+- [`src/core/`](src/core/) - retrieval, reranking, generation, citations, orchestration
+- [`src/api/`](src/api/) - FastAPI models and routes
+- [`src/web/`](src/web/) - Streamlit UI and ingestion service
+- [`src/evaluation/`](src/evaluation/) - truthfulness scorer, generation metrics, retrieval metrics
+- [`src/utils/`](src/utils/) - config, logging, and vector database integrations
+- [`evals/`](evals/) - offline eval harness, golden datasets, RAGAS adapter
+- [`data/sample/`](data/sample/) - preloaded sample documents for demos
+- [`spaces/`](spaces/) - Hugging Face Spaces deployment files
+- [`docker/`](docker/) - Docker Compose stack for API, Streamlit, Redis, and Qdrant
+- [`Docs/`](Docs/) - architecture notes, runbook, roadmap, phase documentation
 
----
+## Where To Go Deeper
+
+- [`Docs/PROJECT_OVERVIEW.md`](Docs/PROJECT_OVERVIEW.md) - system architecture and reader-friendly project overview
+- [`Docs/RUNBOOK.md`](Docs/RUNBOOK.md) - local setup, Docker setup, API keys, rate limiting, troubleshooting
+- [`Docs/phase2_hybrid_retrieval.md`](Docs/phase2_hybrid_retrieval.md) - hybrid retrieval and RRF design
+- [`Docs/phase3_reranking_generation.md`](Docs/phase3_reranking_generation.md) - reranking, generation, and context optimization
+- [`Docs/phase4_citation_api.md`](Docs/phase4_citation_api.md) - citation and API design
+- [`Docs/performance_baseline.md`](Docs/performance_baseline.md) - FastAPI overhead baseline
+- [`Docs/ROADMAP.md`](Docs/ROADMAP.md) - delivery status and planned improvements
 
 ## Development
 
@@ -250,22 +290,13 @@ Multi-provider API key environment variables:
 export OPENAI_API_KEY=...
 export ANTHROPIC_API_KEY=...
 export GEMINI_API_KEY=...
+export DOC_API_KEYS=dev-key-1
 ```
 
----
-
 ## Troubleshooting
 
 - **Empty results after ingest:** Run `python -m src.ingest --docs data/documents` and verify `data/embeddings/` exists.
 - **Embedding model error:** Ensure Ollama is running and `nomic-embed-text` is pulled, or switch to a different embedding provider in `config.yaml`.
 - **Dimension mismatch after model change:** Re-ingest all documents to rebuild the vector index.
 - **Cloud provider fails:** Check the relevant `*_API_KEY` env var is set.
-- **Truthfulness score always 0:** The NLI model (`cross-encoder/nli-deberta-v3-small`) downloads on first use (~140 MB). Check internet access or set `evaluation.inline_enabled: false` in `config.yaml` to disable.
-
----
-
-## Documentation
-
-- [Production Runbook](Docs/RUNBOOK.md)
-- [Roadmap](Docs/ROADMAP.md)
-- [Project overview](Docs/PROJECT_OVERVIEW.md)
+- **Truthfulness score always 0:** The NLI model (`cross-encoder/nli-deberta-v3-small`) downloads on first use. Check internet access or set `evaluation.inline_enabled: false` in `config.yaml` to disable.

config.yaml

@@ -1,6 +1,7 @@
 # Document processing
-chunk_size: 1000
-overlap: 200
+chunk_size: 600
+overlap: 100
+chunk_tokenizer: gpt2
 
 # File paths
 data_dir: data/documents

docker/.env.example

@@ -10,7 +10,7 @@ OLLAMA_BASE_URL=http://host.docker.internal:11434
 
 # Optional cloud provider keys
 OPENAI_API_KEY=
-ANTHROPIC_API_KEY=
+ANTHROPIC_API_KEY=replace-with-anthropic-api-key
 GEMINI_API_KEY=
 
 # Hugging Face cache/offline toggles

evals/datasets/golden.jsonl

@@ -0,0 +1,52 @@
+{"user_input": "What is Retrieval-Augmented Generation?", "reference": "Retrieval-Augmented Generation (RAG) grounds model answers by retrieving relevant external context before generation.", "reference_contexts": ["Retrieval-Augmented Generation (RAG) enhances LLM answers by retrieving relevant context before generating a response."]}
+{"user_input": "What are the two main phases in RAG?", "reference": "RAG has ingestion and query phases: documents are indexed first, then relevant chunks are retrieved at query time.", "reference_contexts": ["Ingestion prepares chunks and embeddings.", "Query-time retrieval fetches relevant chunks for generation."]}
+{"user_input": "What is BM25 used for?", "reference": "BM25 is a sparse retrieval ranking function that scores lexical relevance between query terms and documents.", "reference_contexts": ["BM25 ranks documents using term frequency and document length normalization."]}
+{"user_input": "What does BM25 k1 control?", "reference": "The k1 parameter controls how quickly BM25 term-frequency gains saturate.", "reference_contexts": ["k1 controls term frequency saturation in BM25."]}
+{"user_input": "What is BM25 b parameter?", "reference": "The b parameter controls document length normalization strength in BM25.", "reference_contexts": ["b controls how strongly BM25 normalizes for document length."]}
+{"user_input": "What is a common BM25 weakness?", "reference": "BM25 struggles with synonym and paraphrase matching because it depends on lexical overlap.", "reference_contexts": ["BM25 cannot reliably match semantic synonyms without shared keywords."]}
+{"user_input": "What is dense retrieval?", "reference": "Dense retrieval compares embedding vectors to find semantically similar chunks.", "reference_contexts": ["Dense retrieval uses vector similarity between query and document embeddings."]}
+{"user_input": "What is hybrid retrieval?", "reference": "Hybrid retrieval combines sparse BM25 and dense vector retrieval to improve robustness across query types.", "reference_contexts": ["BM25 and dense retrieval are complementary and often fused together."]}
+{"user_input": "What is Reciprocal Rank Fusion?", "reference": "Reciprocal Rank Fusion combines rankings by summing 1/(k+rank) contributions from each retriever.", "reference_contexts": ["RRF score is computed as the sum of reciprocal rank terms across retrieval methods."]}
+{"user_input": "Why rerank after retrieval?", "reference": "Reranking improves final top-k quality by re-scoring candidate chunks with a stronger cross-encoder model.", "reference_contexts": ["Cross-encoder reranking refines candidate ordering after initial retrieval."]}
+{"user_input": "What is a cross-encoder reranker?", "reference": "A cross-encoder reranker jointly encodes query and chunk text to score relevance more precisely.", "reference_contexts": ["Cross-encoders evaluate query-document pairs directly for relevance."]}
+{"user_input": "Why keep citations in RAG answers?", "reference": "Citations make answers auditable and help detect unsupported claims.", "reference_contexts": ["Citation tracking is used to verify whether claims are grounded in retrieved evidence."]}
+{"user_input": "What does faithfulness measure?", "reference": "Faithfulness measures whether answer statements are supported by retrieved evidence.", "reference_contexts": ["Faithfulness checks if generated claims are entailed by retrieval context."]}
+{"user_input": "What is NLI-based faithfulness?", "reference": "NLI-based faithfulness uses entailment models to estimate if answer statements are supported by context.", "reference_contexts": ["Natural Language Inference can score support versus contradiction for generated statements."]}
+{"user_input": "What is citation groundedness?", "reference": "Citation groundedness reflects how well cited evidence supports the associated answer statements.", "reference_contexts": ["Groundedness scores estimate support strength from cited chunks."]}
+{"user_input": "Why chunk documents before indexing?", "reference": "Chunking creates retrieval-sized units that improve recall and context precision.", "reference_contexts": ["Large documents are split into smaller chunks for better retrieval granularity."]}
+{"user_input": "What is token-based chunking?", "reference": "Token-based chunking splits text by model token counts instead of characters.", "reference_contexts": ["Token-aware chunking aligns chunk boundaries with LLM/embedding tokenization."]}
+{"user_input": "Why use chunk overlap?", "reference": "Overlap preserves context continuity across adjacent chunks and reduces boundary loss.", "reference_contexts": ["Chunk overlap keeps neighboring context that might otherwise be truncated."]}
+{"user_input": "What is a vector database?", "reference": "A vector database stores embeddings and supports nearest-neighbor similarity search.", "reference_contexts": ["Vector databases index high-dimensional vectors for semantic retrieval."]}
+{"user_input": "What is HNSW in vector search?", "reference": "HNSW is a graph-based approximate nearest-neighbor index optimized for fast high-recall search.", "reference_contexts": ["HNSW provides efficient ANN search for vector retrieval systems."]}
+{"user_input": "How does ingestion use metadata?", "reference": "Ingestion metadata like title and file type can improve filtering, ranking, and diagnostics.", "reference_contexts": ["Metadata is attached to chunks for filtering, ranking features, and traceability."]}
+{"user_input": "What does duplicate detection prevent?", "reference": "Duplicate detection prevents re-indexing identical content and reduces noisy retrieval redundancy.", "reference_contexts": ["Content hashing can skip duplicate documents during ingestion."]}
+{"user_input": "Why combine BM25 and vectors instead of one?", "reference": "Combining both captures exact keywords and semantic intent, reducing each method's blind spots.", "reference_contexts": ["Sparse retrieval handles lexical precision, dense retrieval handles semantic similarity."]}
+{"user_input": "What is context precision?", "reference": "Context precision measures how many retrieved chunks are relevant among the returned set.", "reference_contexts": ["Precision evaluates the fraction of retrieved context that is relevant."]}
+{"user_input": "What is context recall?", "reference": "Context recall measures how much of the relevant reference context appears in retrieval results.", "reference_contexts": ["Recall evaluates whether needed supporting context was retrieved."]}
+{"user_input": "What does answer relevancy evaluate?", "reference": "Answer relevancy evaluates semantic alignment between the user question and generated answer.", "reference_contexts": ["Relevancy checks if the answer addresses the asked question."]}
+{"user_input": "What does ROUGE-L indicate in evals?", "reference": "ROUGE-L estimates overlap with a reference answer using longest common subsequence matching.", "reference_contexts": ["ROUGE-L is commonly used to compare generated and reference answer similarity."]}
+{"user_input": "Why have a smoke eval dataset?", "reference": "A smoke dataset validates evaluation harness plumbing quickly with low runtime.", "reference_contexts": ["Smoke tests check that the eval pipeline executes end to end."]}
+{"user_input": "Why maintain a golden dataset?", "reference": "A golden dataset provides stable regression coverage for retrieval and generation quality over time.", "reference_contexts": ["Golden eval sets are used to detect quality regressions across releases."]}
+{"user_input": "Why gate CI on quality thresholds?", "reference": "Quality gates prevent merges that degrade faithfulness or answer correctness below acceptable levels.", "reference_contexts": ["CI thresholds enforce minimum evaluation quality before merge."]}
+{"user_input": "What is an API key auth pattern in this app?", "reference": "Protected endpoints require an X-API-Key that matches configured server keys.", "reference_contexts": ["API key middleware validates incoming keys against configured allowed keys."]}
+{"user_input": "Why add rate limiting to RAG APIs?", "reference": "Rate limiting protects service stability and controls abuse under high request bursts.", "reference_contexts": ["Rate limits are applied per client to reduce overload and misuse."]}
+{"user_input": "What is SSE in query streaming?", "reference": "Server-Sent Events stream partial response chunks over a long-lived HTTP response.", "reference_contexts": ["SSE enables token-by-token or chunk-by-chunk response streaming."]}
+{"user_input": "What is provider routing in generation?", "reference": "Provider routing allows selecting Ollama, OpenAI, Anthropic, or Gemini per request.", "reference_contexts": ["A provider router dispatches generation calls to configured LLM backends."]}
+{"user_input": "Why use fallback caching?", "reference": "Fallback caching preserves responsiveness when external dependencies are slow or unavailable.", "reference_contexts": ["Response cache layers can reduce repeated generation latency."]}
+{"user_input": "What does deterministic retrieval behavior mean?", "reference": "Deterministic retrieval returns stable rankings for identical inputs and corpus state.", "reference_contexts": ["Stable ranking behavior is important for reproducible evaluation."]}
+{"user_input": "Why include health endpoints?", "reference": "Health endpoints expose service readiness and dependency status for operations monitoring.", "reference_contexts": ["/health endpoints are used by orchestrators and uptime checks."]}
+{"user_input": "What is metrics endpoint purpose?", "reference": "Metrics endpoints expose counters and latency series for observability systems.", "reference_contexts": ["/metrics provides telemetry for monitoring and alerting."]}
+{"user_input": "What is citation verification?", "reference": "Citation verification checks whether cited chunks actually support the linked answer claims.", "reference_contexts": ["Verification distinguishes supported citations from weak or unsupported ones."]}
+{"user_input": "Why track uncited claims?", "reference": "Uncited claims indicate potential hallucination risk and lower answer trustworthiness.", "reference_contexts": ["Claims lacking citations are higher-risk for unsupported output."]}
+{"user_input": "What is context window optimization?", "reference": "Context window optimization packs the most relevant chunks within token budget limits.", "reference_contexts": ["Retrieved chunks are selected and trimmed to fit model max context tokens."]}
+{"user_input": "Why use configurable models?", "reference": "Configurable models let teams balance quality, cost, and latency per environment.", "reference_contexts": ["Model selection by config supports portability and controlled rollouts."]}
+{"user_input": "Why keep cloud providers optional?", "reference": "Optional cloud providers preserve local-first operation while enabling production elasticity.", "reference_contexts": ["The system can run locally without mandatory external APIs."]}
+{"user_input": "What does local-first RAG imply?", "reference": "Local-first RAG prioritizes local indexing/retrieval and optional external services.", "reference_contexts": ["Core retrieval can run on local infrastructure without cloud lock-in."]}
+{"user_input": "Why test integration in addition to unit tests?", "reference": "Integration tests validate end-to-end behavior across components and contracts.", "reference_contexts": ["Unit tests check isolated logic, integration tests validate composed workflows."]}
+{"user_input": "Why use markdown and JSON eval reports?", "reference": "JSON enables automation while markdown improves human review in CI artifacts.", "reference_contexts": ["Dual report formats support machine processing and stakeholder readability."]}
+{"user_input": "How does CI skip missing optional secrets?", "reference": "Conditional job execution can skip provider-specific evaluations when required secrets are absent.", "reference_contexts": ["GitHub Actions job conditions can check secret presence before running."]}
+{"user_input": "What does a faithfulness threshold represent?", "reference": "A faithfulness threshold is the minimum acceptable average support score for generated answers.", "reference_contexts": ["Thresholds define pass/fail criteria for evaluation gating."]}
+{"user_input": "Why keep architecture simple here?", "reference": "Simple architecture reduces maintenance overhead and keeps behavior understandable and testable.", "reference_contexts": ["Avoiding unnecessary abstractions improves reliability and delivery speed."]}
+{"user_input": "How does chunk overlap affect retrieval?", "reference": "Appropriate overlap reduces context fragmentation and can improve answer completeness.", "reference_contexts": ["Overlap ensures boundary sentences remain retrievable in adjacent chunks."]}
+{"user_input": "What is the role of orchestrator in RAG?", "reference": "The orchestrator coordinates retrieval, reranking, context building, generation, and citations.", "reference_contexts": ["RAG orchestration composes retrieval and generation stages into one request pipeline."]}
+{"user_input": "Why monitor evaluation latency?", "reference": "Monitoring eval latency helps control CI runtime and detect regressions in model responsiveness.", "reference_contexts": ["Latency tracking complements quality metrics in evaluation operations."]}

evals/datasets/golden_ci.jsonl

@@ -0,0 +1,20 @@
+{"user_input": "What is Retrieval-Augmented Generation?", "reference": "Retrieval-Augmented Generation (RAG) grounds model answers by retrieving relevant external context before generation.", "reference_contexts": ["Retrieval-Augmented Generation (RAG) enhances LLM answers by retrieving relevant context before generating a response."]}
+{"user_input": "What are the two main phases in RAG?", "reference": "RAG has ingestion and query phases: documents are indexed first, then relevant chunks are retrieved at query time.", "reference_contexts": ["Ingestion prepares chunks and embeddings.", "Query-time retrieval fetches relevant chunks for generation."]}
+{"user_input": "What is BM25 used for?", "reference": "BM25 is a sparse retrieval ranking function that scores lexical relevance between query terms and documents.", "reference_contexts": ["BM25 ranks documents using term frequency and document length normalization."]}
+{"user_input": "What does BM25 k1 control?", "reference": "The k1 parameter controls how quickly BM25 term-frequency gains saturate.", "reference_contexts": ["k1 controls term frequency saturation in BM25."]}
+{"user_input": "What is BM25 b parameter?", "reference": "The b parameter controls document length normalization strength in BM25.", "reference_contexts": ["b controls how strongly BM25 normalizes for document length."]}
+{"user_input": "What is a common BM25 weakness?", "reference": "BM25 struggles with synonym and paraphrase matching because it depends on lexical overlap.", "reference_contexts": ["BM25 cannot reliably match semantic synonyms without shared keywords."]}
+{"user_input": "What is dense retrieval?", "reference": "Dense retrieval compares embedding vectors to find semantically similar chunks.", "reference_contexts": ["Dense retrieval uses vector similarity between query and document embeddings."]}
+{"user_input": "What is hybrid retrieval?", "reference": "Hybrid retrieval combines sparse BM25 and dense vector retrieval to improve robustness across query types.", "reference_contexts": ["BM25 and dense retrieval are complementary and often fused together."]}
+{"user_input": "What is Reciprocal Rank Fusion?", "reference": "Reciprocal Rank Fusion combines rankings by summing 1/(k+rank) contributions from each retriever.", "reference_contexts": ["RRF score is computed as the sum of reciprocal rank terms across retrieval methods."]}
+{"user_input": "Why rerank after retrieval?", "reference": "Reranking improves final top-k quality by re-scoring candidate chunks with a stronger cross-encoder model.", "reference_contexts": ["Cross-encoder reranking refines candidate ordering after initial retrieval."]}
+{"user_input": "What is a cross-encoder reranker?", "reference": "A cross-encoder reranker jointly encodes query and chunk text to score relevance more precisely.", "reference_contexts": ["Cross-encoders evaluate query-document pairs directly for relevance."]}
+{"user_input": "Why keep citations in RAG answers?", "reference": "Citations make answers auditable and help detect unsupported claims.", "reference_contexts": ["Citation tracking is used to verify whether claims are grounded in retrieved evidence."]}
+{"user_input": "What does faithfulness measure?", "reference": "Faithfulness measures whether answer statements are supported by retrieved evidence.", "reference_contexts": ["Faithfulness checks if generated claims are entailed by retrieval context."]}
+{"user_input": "What is NLI-based faithfulness?", "reference": "NLI-based faithfulness uses entailment models to estimate if answer statements are supported by context.", "reference_contexts": ["Natural Language Inference can score support versus contradiction for generated statements."]}
+{"user_input": "What is citation groundedness?", "reference": "Citation groundedness reflects how well cited evidence supports the associated answer statements.", "reference_contexts": ["Groundedness scores estimate support strength from cited chunks."]}
+{"user_input": "Why chunk documents before indexing?", "reference": "Chunking creates retrieval-sized units that improve recall and context precision.", "reference_contexts": ["Large documents are split into smaller chunks for better retrieval granularity."]}
+{"user_input": "What is token-based chunking?", "reference": "Token-based chunking splits text by model token counts instead of characters.", "reference_contexts": ["Token-aware chunking aligns chunk boundaries with LLM/embedding tokenization."]}
+{"user_input": "Why use chunk overlap?", "reference": "Overlap preserves context continuity across adjacent chunks and reduces boundary loss.", "reference_contexts": ["Chunk overlap keeps neighboring context that might otherwise be truncated."]}
+{"user_input": "What is a vector database?", "reference": "A vector database stores embeddings and supports nearest-neighbor similarity search.", "reference_contexts": ["Vector databases index high-dimensional vectors for semantic retrieval."]}
+{"user_input": "What is HNSW in vector search?", "reference": "HNSW is a graph-based approximate nearest-neighbor index optimized for fast high-recall search.", "reference_contexts": ["HNSW provides efficient ANN search for vector retrieval systems."]}

rag-spec-evaluation.md

@@ -0,0 +1,96 @@
+# RAG Spec Evaluation Report
+
+## Context
+
+This is a gap analysis of the Doc-Ingestion app against a production-grade RAG spec across three phases. The goal is to identify what is built, what partially meets the spec, and what is missing or misaligned.
+
+---
+
+## Spec vs. Implementation: Detailed Evaluation
+
+### Phase 1 — Fundamentals
+
+| Requirement | Status | Detail |
+|---|---|---|
+| Ingest documents | ✅ Built | PDF, DOCX, TXT, MD, HTML via `src/core/document_processor.py` |
+| 500–800 **token** chunks | ⚠️ Partial / Misaligned | Chunking uses **characters** (default: 1000 chars, 200 overlap), not tokens. Spec requires token-based chunking (500–800 tokens). No tokenizer is applied during chunking. |
+| 100-token overlap | ⚠️ Partial / Misaligned | Overlap is 200 characters, not 100 tokens. Same root issue: characters vs. tokens. |
+| Vector store (Chroma or Qdrant) | ✅ Built | ChromaDB for dev (`data/embeddings/chroma`), Qdrant optional for prod. Both present in `src/utils/database.py`. |
+
+**Gap to fix:**
+- `src/core/document_processor.py` `chunk_text()` method uses character sliding window. Needs to be replaced with a tokenizer-aware splitter (e.g., `tiktoken` or `transformers` tokenizer) targeting 500–800 tokens with 100-token overlap.
+- `config.yaml` `chunk_size: 1000` and `overlap: 200` need to change to token units.
+
+---
+
+### Phase 2 — Hybrid Retrieval + Re-ranking
+
+| Requirement | Status | Detail |
+|---|---|---|
+| BM25 keyword search | ✅ Built | `src/core/bm25_search.py`, `BM25Search` using `rank-bm25` |
+| Vector semantic search | ✅ Built | `src/core/vector_search.py`, `VectorSearch` with ChromaDB |
+| Hybrid retrieval combining both | ✅ Built | `src/core/hybrid_retriever.py`, `HybridRetriever` with Reciprocal Rank Fusion (RRF), parallel execution |
+| Cross-encoder re-ranker | ✅ Built | `src/core/reranker.py`, `CrossEncoderReranker` using `cross-encoder/ms-marco-MiniLM-L-6-v2` |
+
+**Phase 2 is fully built and exceeds the spec** (adds RRF fusion, LRU caching, confidence scoring, configurable weights).
+
+---
+
+### Phase 3 — Evaluation Dataset + CI/CD
+
+| Requirement | Status | Detail |
+|---|---|---|
+| Golden dataset of 50–200 Q&A pairs | ⚠️ Missing | Only `evals/datasets/smoke.jsonl` (~few entries, 1 KB) and `evals/datasets/sample.jsonl` (~6 KB, estimated ~10–15 pairs). Neither meets the 50–200 pair threshold. |
+| Offline evaluation script | ✅ Built | `evals/run_evals.py` (504 lines) with 8+ metrics: answer_relevancy, context_precision, context_recall, ROUGE-L, citation_rate, faithfulness |
+| CI/CD pipeline integration | ⚠️ Partial | `.github/workflows/ci.yml` has `evals-smoke` job, but it runs with `--mock` flag (MockPipeline). It does **not** measure real faithfulness — it tests the eval harness, not the RAG pipeline. |
+| Measure faithfulness | ⚠️ Partial | NLI faithfulness via `src/evaluation/truthfulness.py` exists. RAGAS integration exists in `evals/adapters/ragas_llm_adapter.py` but is optional (requires `ragas>=0.2`, `langchain-core` extra deps) and not wired into CI. |
+
+---
+
+### Recommended Tech Stack Alignment
+
+| Recommendation | Status | Detail |
+|---|---|---|
+| LangChain or LangGraph | ❌ Not used | Core pipeline uses direct HTTP API calls (`src/core/llm_provider.py`). LangChain only appears as a thin adapter in `evals/adapters/ragas_llm_adapter.py` to satisfy RAGAS interface — it is not the orchestration framework. |
+| ChromaDB or Qdrant | ✅ Built | Both present |
+| Ragas for evaluation | ⚠️ Optional / Incomplete | Present as optional adapter, not enforced. CI runs MockPipeline without RAGAS. |
+
+---
+
+## Summary: What Is Missing
+
+### Must-Fix (spec violations)
+
+1. **Token-based chunking** — `src/core/document_processor.py:chunk_text()` uses character counts, not tokens. Replace with tokenizer-aware chunking (e.g., `tiktoken`) targeting 500–800 tokens, 100-token overlap. Update `config.yaml` units accordingly.
+
+2. **Golden evaluation dataset (50–200 pairs)** — `evals/datasets/` only has smoke (~few entries) and sample (~10–15 pairs). Need to create a curated dataset of at least 50 ground-truth Q&A pairs with reference contexts, authored against real ingested documents.
+
+3. **CI/CD runs real faithfulness evaluation** — The `evals-smoke` GitHub Actions job uses `--mock`. A CI job that runs `LivePipeline` against the golden dataset (or a representative subset) and gates on a faithfulness threshold is required by the spec.
+
+### Should-Fix (partial alignment)
+
+4. **RAGAS made non-optional** — RAGAS faithfulness should be a hard dependency in `evals/`, not a conditional import behind `try/except`. The eval report should always include RAGAS faithfulness score.
+
+5. **LangChain / LangGraph adoption** — The spec recommends LangChain/LangGraph as the orchestration layer. Currently the pipeline is custom HTTP. This is a tech stack deviation, not a functional gap — worth noting but lower priority than items 1–3.
+
+---
+
+## What Is Already Production-Grade (exceeds spec)
+
+- Full hybrid retrieval with RRF fusion, parallel execution, and LRU caching
+- Cross-encoder reranking with batch scoring and threshold filtering
+- Multi-provider LLM routing (Ollama, OpenAI, Anthropic, Gemini) with streaming
+- NLI-based inline truthfulness scoring in the serving path
+- Citation tracking and verification
+- Response caching (Redis + in-memory fallback)
+- Rate limiting, API key auth, audit logging
+- Comprehensive IR metrics module (P@K, R@K, MRR, MAP, NDCG)
+- Unit + integration test coverage with CI
+
+---
+
+## Verification Steps (after fixes)
+
+1. After token-based chunking: ingest a known document, query it, verify chunk boundaries fall within 500–800 token range using `tiktoken` inspection script.
+2. After golden dataset: run `python -m evals.run_evals --dataset evals/datasets/golden.jsonl --live` and confirm dataset size ≥ 50.
+3. After CI wiring: confirm GitHub Actions `evals-golden` job fails when faithfulness drops below threshold (e.g., `nli_faithfulness < 0.7`).

requirements/base.txt

@@ -24,6 +24,7 @@ transformers>=4.30.0
 torch>=2.0.0
 accelerate>=0.20.0
 tokenizers>=0.13.0
+tiktoken>=0.5.0
 rouge-score>=0.1.2
 bert-score>=0.3.13
 sacrebleu>=2.3.0

src/core/document_processor.py

@@ -12,14 +12,40 @@ from datetime import datetime
 from typing import Dict, List, Optional
 
 import PyPDF2
+import tiktoken
 from bs4 import BeautifulSoup
 from docx import Document
 
 
+class _RegexTokenizer:
+    """Offline fallback tokenizer when tiktoken encoding cannot be loaded."""
+
+    _token_pattern = re.compile(r"\w+|[^\w\s]", re.UNICODE)
+
+    def encode(self, text: str) -> List[str]:
+        return self._token_pattern.findall(text)
+
+    def decode(self, token_ids: List[str]) -> str:
+        if not token_ids:
+            return ""
+        return " ".join(token_ids)
+
+
 class DocumentProcessor:
-    def __init__(self, chunk_size: int = 1000, overlap: int = 200):
+    def __init__(self, chunk_size: int = 600, overlap: int = 100, tokenizer_name: str = "gpt2"):
+        if chunk_size <= 0:
+            raise ValueError("chunk_size must be > 0")
+        if overlap < 0:
+            raise ValueError("overlap must be >= 0")
+        if overlap >= chunk_size:
+            raise ValueError("overlap must be smaller than chunk_size")
         self.chunk_size = chunk_size
         self.overlap = overlap
+        self.tokenizer_name = tokenizer_name
+        try:
+            self._tokenizer = tiktoken.get_encoding(tokenizer_name)
+        except Exception:
+            self._tokenizer = _RegexTokenizer()
         self._seen_hashes: set = set()
 
     def process_document(self, file_path: str) -> Optional[Dict]:
@@ -76,17 +102,24 @@ class DocumentProcessor:
         return text.strip()
 
     def chunk_text(self, text: str) -> List[str]:
-        chunks = []
+        token_ids = self._tokenizer.encode(text)
+        if not token_ids:
+            return []
+
+        chunks: List[str] = []
         step = self.chunk_size - self.overlap
         start = 0
-        while start < len(text):
-            end = min(start + self.chunk_size, len(text))
-            chunk = text[start:end]
-            if chunk:
+        while start < len(token_ids):
+            end = min(start + self.chunk_size, len(token_ids))
+            chunk = self._tokenizer.decode(token_ids[start:end])
+            if chunk.strip():
                 chunks.append(chunk)
             start += step
         return chunks
 
+    def count_tokens(self, text: str) -> int:
+        return len(self._tokenizer.encode(text))
+
     # --- private extractors ---
 
     def _extract_pdf_text(self, file_path: str) -> str:

src/ingest.py

@@ -37,10 +37,19 @@ def collect_files(path: str) -> list[str]:
 def ingest(docs_path: str) -> tuple[BM25Index, VectorDatabase]:
     # ── 1. Config ────────────────────────────────────────────────────────────
     cfg = load_config("config.yaml")
-    logger.info("Config loaded: chunk_size=%d overlap=%d", cfg.chunk_size, cfg.overlap)
+    logger.info(
+        "Config loaded: chunk_size=%d overlap=%d tokenizer=%s",
+        cfg.chunk_size,
+        cfg.overlap,
+        cfg.chunk_tokenizer,
+    )
 
     # ── 2. Components ─────────────────────────────────────────────────────────
-    processor = DocumentProcessor(chunk_size=cfg.chunk_size, overlap=cfg.overlap)
+    processor = DocumentProcessor(
+        chunk_size=cfg.chunk_size,
+        overlap=cfg.overlap,
+        tokenizer_name=cfg.chunk_tokenizer,
+    )
     index = BM25Index()
     db = VectorDatabase(mode="dev", chroma_path="data/embeddings/chroma")
     db.create_collection(COLLECTION_NAME)

src/utils/config.py

@@ -135,8 +135,9 @@ class APISettings(BaseModel):
 
 
 class Config(BaseModel):
-    chunk_size: int = Field(1000, description="Size of text chunks")
-    overlap: int = Field(200, description="Overlap between chunks")
+    chunk_size: int = Field(600, description="Chunk size in tokens")
+    overlap: int = Field(100, description="Chunk overlap in tokens")
+    chunk_tokenizer: str = Field("gpt2", description="Tokenizer encoding used for ingestion chunking")
     data_dir: str = Field("data", description="Directory for input files")
     output_dir: str = Field("output", description="Directory for processed output")
     log_level: str = Field("INFO", description="Logging level")

tests/unit/test_config.py

@@ -16,8 +16,9 @@ def _write_config(data: dict) -> str:
 class TestConfigDefaults:
     def test_default_values(self):
         cfg = Config()
-        assert cfg.chunk_size == 1000
-        assert cfg.overlap == 200
+        assert cfg.chunk_size == 600
+        assert cfg.overlap == 100
+        assert cfg.chunk_tokenizer == "gpt2"
         assert cfg.log_level == "INFO"
 
     def test_custom_values(self):
@@ -45,7 +46,7 @@ class TestLoadConfig:
         path = _write_config({"chunk_size": 256})
         try:
             cfg = load_config(path)
-            assert cfg.overlap == 200  # default
+            assert cfg.overlap == 100  # default
         finally:
             os.unlink(path)
 
@@ -53,7 +54,7 @@ class TestLoadConfig:
         path = _write_config({})
         try:
             cfg = load_config(path)
-            assert cfg.chunk_size == 1000
+            assert cfg.chunk_size == 600
         finally:
             os.unlink(path)
 

tests/unit/test_document_processor.py

@@ -7,7 +7,7 @@ from src.core.document_processor import DocumentProcessor
 
 @pytest.fixture
 def processor():
-    return DocumentProcessor(chunk_size=100, overlap=20)
+    return DocumentProcessor(chunk_size=50, overlap=10)
 
 
 def _write_temp_file(content: str, suffix: str) -> str:
@@ -69,22 +69,23 @@ class TestChunkText:
         assert chunks[0] == "short text"
 
     def test_long_text_produces_multiple_chunks(self, processor):
-        text = "a" * 300
+        text = "token " * 300
         chunks = processor.chunk_text(text)
         assert len(chunks) > 1
 
     def test_chunk_size_respected(self, processor):
-        text = "x" * 250
+        text = "word " * 250
         chunks = processor.chunk_text(text)
-        assert all(len(c) <= processor.chunk_size for c in chunks)
+        assert all(processor.count_tokens(c) <= processor.chunk_size for c in chunks)
 
     def test_overlap_creates_shared_content(self):
-        proc = DocumentProcessor(chunk_size=50, overlap=10)
-        text = "a" * 100
+        proc = DocumentProcessor(chunk_size=20, overlap=5)
+        text = "alpha beta gamma delta epsilon " * 40
         chunks = proc.chunk_text(text)
-        # With overlap, step = 40, so chunks[1] should start at index 40
-        # meaning the last 10 chars of chunk[0] equal the first 10 of the overlap region
-        assert len(chunks) >= 3
+        assert len(chunks) >= 2
+        first_tokens = proc._tokenizer.encode(chunks[0])
+        second_tokens = proc._tokenizer.encode(chunks[1])
+        assert first_tokens[-proc.overlap:] == second_tokens[:proc.overlap]
 
     def test_no_empty_chunks(self, processor):
         chunks = processor.chunk_text("hello world " * 20)